The multi-window time-frequency reassignment helps to improve the time-frequency resolution of bark-frequency spectral coefficient (BFSC) analysis of heart sounds. For this purpose, a new heart sound classification algorithm combining feature extraction based on multi-window time-frequency reassignment BFSC with deep learning was proposed in this paper. Firstly, the randomly intercepted heart sound segments are preprocessed with amplitude normalization, the heart sounds were framed and time-frequency rearrangement based on short-time Fourier transforms were computed using multiple orthogonal windows. A smooth spectrum estimate is calculated by arithmetic averaging each of the obtained independent spectra. Finally, the BFSC of reassignment spectrum is extracted as a feature by the Bark filter bank. In this paper, convolutional network and recurrent neural network are used as classifiers for model comparison and performance evaluation of the extracted features. Eventually, the multi-window time-frequency rearrangement improved BFSC method extracts more discriminative features, with a binary classification accuracy of 0.936, a sensitivity of 0.946, and a specificity of 0.922. These results present that the algorithm proposed in this paper does not need to segment the heart sounds and randomly intercepts the heart sound segments, which greatly simplifies the computational process and is expected to be used for screening of congenital heart disease.