In order to solve the problem of extracting ultrasonic signals from strong background noise, a novel method, which is termed APSO-SD algorithm and based on improved adaptive particle swarm optimization (APSO) and sparse decomposition (SD) theory, is proposed in this paper. This method can convert the ultrasonic signal denoising problem into optimizing the function on the infinite parameter set. First, based on the sparse decomposition theory and the structural characteristics of ultrasonic signal, the objective function of particle swarm optimization algorithm and the reconstruction algorithm of the denoised signal are constructed, so that particle swarm optimization and ultrasonic signal denoising can be combined. Second, in order to improve the robustness of the proposed approach, an APSO algorithm is proposed. What is more, because particle swarm optimization algorithm can be used to optimize in continuous parameter space, and according to the empirical characteristics of the ultrasonic signals used in practical engineering, a continuous super complete dictionary for matching ultrasonic signals is established. Since the super complete dictionary is continuous, there are an infinite number of atoms in the established dictionary. The redundancy of dictionaries is enhanced by the method in this paper. Based on the fact that the inner product of the optimal atom and the ultrasonic signal is one and the inner product of the noise and the optimal atom is zero in the established dictionary, the objective optimization function of APSO-SD algorithm is established. Finally, the optimal atom is determined based on the optimization result of the objective function. In this way, the denoising ultrasonic signal can be reconstructed by using the optimal atom according to the reconstruction algorithm. The processing results of simulated ultrasonic signals and measured ultrasonic signals show that the proposed method can effectively extract weak ultrasonic signals from strong background noise whose signal-to-noise ratio is lowest, as low as-4 dB. In addition, compared with the adaptive threshold based wavelet method, the proposed method in this paper shows the good denoising performance. In this paper, it is demonstrated that the problem of ultrasonic signal denoising can be transformed into the optimization of constraint functions. Furthermore, the ability of the proposed APSO-SD algorithm to accurately recover signals from noisy acoustic signals is better than that of the common wavelet method.