Smoothing or geometrical optimization is one of basic procedures to improve the quality of mesh. This paper first introduces an error-based mesh quality metric based on the concept of optimal Delaunay triangulations, and then examines the smoothing scheme which minimizes the interpolation error among all triangulations with the same number of vertices. Facing to its deficiency, a modified smoothing scheme and corresponding optimization model for tetrahedral mesh that avoid illegal elements are proposed. The optimization model is solved by integrating chaos search and BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm efficiently. Quality improvement for tetrahedral mesh is realized through alternately applying the smoothing approach suggested and topological optimization technique. Testing results show that the proposed approach is effective to improve mesh quality and suitable for combining with topological technique.