In this paper we propose a simple and intuitive method for constructing partially linear models and, in general, partially parametric models, using support vector machines for regression and, in particular, using regularization networks (splines). The results are more satisfactory than those for classical nonparametric approaches. The method is based on a suitable approach to selecting the kernel by relaying on the properties of positive definite functions. No modification is required of the standard SVM algorithms, and the approach is valid for the e-insensitive loss. The approach described here can be immediately applied to SVMs for classification and to other methods that use the kernel as the inner product.