A new O(N) algorithm based on a recursion method, in which the computational effort is proportional to the number of atoms N, is presented for calculating the inverse of an overlap matrix which is needed in electronic structure calculations with the the non-orthogonal localized basis set. This efficient inverting method can be incorporated in several O(N) methods for diagonalization of a generalized secular equation. By studying convergence properties of the 1-norm of an error matrix for diamond and fcc Al, this method is compared to three other O(N) methods (the divide method, Taylor expansion method, and Hotelling's method) with regard to computational accuracy and efficiency within the density functional theory. The test calculations show that the new method is about one-hundred times faster than the divide method in computational time to achieve the same convergence for both diamond and fcc Al, while the Taylor expansion method and Hotelling's method suffer from numerical instabilities in most cases.

17 pages and 4 figures