Electrocardiogram (ECG) and Phonocardiogram (PCG) play important roles in early prevention and diagnosis of cardiovascular diseases. As the development of machine learning technique, detection of cardiovascular diseases from ECG and PCG has been attracted much attention. However, current available methods are mostly based on single data resource. It is desirable to develop efficient multi-modal machine learning methods to predict and diagnose cardiovascular diseases. In this study, we propose a novel multi-modal method for predicting cardiovascular diseases based on ECG and PCG features. By building up conventional neural networks, we extract ECG and PCG deep coding features respectively. The genetic algorithm is used to screen the combined features and obtain the best feature subset. Then support vector machine makes classification decision. Experimental results show that compared with using single-modal features ECG and PCG, the performance of this method reaches an AUC value of 0.936 when using multi-modal data resources. This dataset is developed from a real-world dataset which was assembled by PhysioNet/CinC Challenge in 2016. The original dataset can be downloaded from website (http://www.physionet.org/challenge/2016/).