An original algorithm for underdetermined mixing matrix estimation is proposed in this paper, which can estimate the mixing matrix effectively when the number of source signals is unknown. Firstly, a new single source point (SSP) detection algorithm based on transform matrix is proposed, which can effectively detect single source time-frequency (TF) points by using the characteristics of complex ratio and improve the sparsity of source signals. In view of the fact that the number of source signals is unknown, a novel estimation algorithm based on element sorting is proposed, which can significantly improve the estimation accuracy of the number of source signals. Finally, the mixing matrix is estimated by using the cluster center obtained by agglomerative hierarchical clustering (AHC) algorithm. The simulation results show that the proposed algorithm can improve the estimation accuracy of the number of source signals and underdetermined mixing matrix obviously, and the algorithm has higher robustness compared with other algorithms.