Generalized linear models are widely used in data analysis and machine learning, especially in large-scale machine learning because of its simplicity and good performance. Generalized linear models include regression, like linear regression, lasso and classification, support vector machine and logistic regression. We have some popular optimization methods to solve them, including stochastic gradient descent, coordinate descent and alternating direction method of multipliers. Commonly used systems for generalized linear model use a single optimization algorithm to solve all kinds of models. However, experiments show that it is impossible to achieve a cost-effective solution for all kinds of generalized linear models due to the differences between different models. In order to resolve the problem, we propose a rule-based optimization algorithm selector to select the best optimization algorithm automatically. In this paper, we first broadly review the three commonly used optimization methods, stochastic gradient, coordinate descent, alternating direction method of multipliers, perform some experiments, and then propose a rule-based optimizer to guide the solving. We also design a new system, GLM+, based on the three rules we proposed with the two engineering techniques, parameter selection and optimization for sparse data. We conduct some experiments on real datasets and compare it with the commonly used machine learning systems, Shotgun and scikit-learn, to verify the effectiveness of our system GLM+. The experiments demonstrate that GLM+ can be 3-10×, sometimes 20×, faster than existing popular systems for generalized linear models.