Purpose: Regularization of inverse treatment planning can improve dose uniformity and decrease delivery time of intensity‐modulated radiotherapy (IMRT) thus improving the radiobiological effect. We attempt to select an optimal value of the regularization parameter which improves the smoothness of beam intensity functions without deterioration of accuracy of fit in the IMRT optimization problem. Method and Materials: We apply to inverse treatment planning the L‐curve method which was developed to determine an optimal value of regularization parameter in the discrete ill‐posed problems. The method is based on finding the regularization parameter which minimizes the residual norm which is a measure of accuracy of fit and the solution norm which is a measure of smoothness of solution. The main idea of the L‐curve method is to plot the smoothing norm as a function of the residual norm for all values of the regularization parameter. Remarkably, this characteristic curve has an L‐shaped dependence. The optimal value of regularization parameter can be found at the “corner” of the L‐curve. Results: We apply the L‐curve criterion to the optimization problem of the prostate radiotherapy cancer treatment with intensity‐modulated beams. We show the distinctive L‐shaped dependence of the smoothing norm as a function of the residual norm. Of great practical importance is the vertical part of the L‐curve because the smoothing norm is decreasing and the residual norm remains approximately of the same value. The corner of the L‐curve is located using criteria of 1) maximum curvature and 2) minimum total of the residual and smoothing norm. Conclusion: The L‐curve method can be an important graphical tool for selection of the regularization parameter in inverse treatment planning. The regularization parameter determined using the L‐curve provides the optimal trade‐off between the smoothness of beam intensity functions and the accuracy of fit.