AbstractIn this paper, a novel algorithm is presented for direction of arrival (DOA) estimation and array self-calibration in the presence of unknown mutual coupling. In order to highlight the relationship between the array output and mutual coupling coefficients, we present a novel model of the array output with the unknown mutual coupling coefficients. Based on this model, we use the space alternating generalized expectation-maximization (SAGE) algorithm to jointly estimate the DOA parameters and the mutual coupling coefficients. Unlike many existing counterparts, our method requires neither calibration sources nor initial calibration information. At the same time, our proposed method inherits the characteristics of good convergence and high estimation precision of the SAGE algorithm. By numerical experiments we demonstrate that our proposed method outperforms the existing method for DOA estimation and mutual coupling calibration.