The element-free Galerkin (EFG) method is one of the widely used meshfree methods for solving partial differential equations. In the EFG method, shape functions are derived from a moving least-squares (MLS) approximation, which involves the inversion of a small matrix for every point of interest. To avoid the calculation of matrix inversion in the formulation of the shape functions, an improved MLS approximation is presented, where an orthogonal function system with a weight function is used. However, it can also lead to ill-conditioned or even singular system of equations. In this paper, aspects of the IMLS approximation are analyzed in detail. The reason why singularity problem occurs is studied. A novel technique based on matrix triangular process is proposed to solve this problem. It is shown that the EFG method with present technique is very effective in constructing shape functions. Numerical examples are illustrated to show the efficiency and accuracy of the proposed method. Although our study relies on monomial basis functions, it is more general than existing methods and can be extended to any basis functions.