The aim of this paper is to estimate the mixing matrix of frequency hopping (FH) signals under the underdetermined blind source separation (UBSS) model. The novel mixing matrix estimation algorithm is based on time-frequency (TF) analysis and improved potential function clustering method. For FH signals with synchronous orthogonal network, it is easy to satisfy sparsity in TF domain. The ratios of the TF data at single source points (SSPs) contain the information of mixing matrix, and the angles of the ratios have obvious clustering feature. Therefore, we can utilize the clustering method to obtain the mixing matrix. First, short-time Fourier transform (STFT) is utilized to make received signals sparse in TF domain. Second, the noise energy threshold is set and low-energy points are removed to reduce the amount of calculation and avoid the effects of noise. Then, SSP detection is performed on residual TF points. Next, the improved potential function clustering method is utilized to cluster the angles of the TF ratios at SSPs, and the clustering centers are obtained. Finally, the mixing matrix estimation of each hop is obtained. Synchronous non-orthogonal network allows different FH signals to use the same frequency at certain times. The mixing matrix estimation obtained by the above algorithm is not accurate. Therefore, some conditions is discussed in synchronous non-orthogonal network. Simulation results indicate that the proposed algorithm is effective and has better performance than the compared algorithm. The improved potential function clustering algorithm provides a new option for clustering analysis in other models and applications.