Detecting cancer early greatly improves the chance of survival and decreases the amount of money required to treat it. Most current technologies used in routine cancer screening only use one source of data and therefore have a higher rate of misdiagnosis and longer time until diagnosed. To overcome this obstacle, we propose a new multi-modal artificial intelligence (AI) system that adds multiple sources of data together to form a consolidated platform for diagnosis, such as combining brain MRIs, lung CTs, electronic health records (EHR's), laboratory test results, genomic markers, and clinical notes. Our new system uses convolutional neural networks (CNN's) to classify brain and lung tumours; random forest classifiers to sensor symptom characteristics; and displays results using GradCAM heat maps to provide visual images in a manner that makes sense to the referring physician. Results from experiments conducted with the new system showed a 94.2% accuracy rate (plus or minus), 92.8% sensitivity, 95.1% specificity, and an area under the ROC curve (AUC) of 0.96. The results confirm that multi-modal datasets will improve the methods used for detecting cancer earlier.