Breast cancer is still a major cause of cancer death in women around the world, so it needs a precise diagnosis that often goes beyond one imaging method. While deep learning has shown great promise in automated diagnosis, current research often suffers from the separation of imaging modalities. This study offers a comprehensive, physics-aware deep learning framework that combines Ultrasound, MRI, and Mammography. We present custom preprocessing pipelines specific to each modality to tackle the different physical degradation models associated with each type of imaging. For MRI, we adopt a 2D slice-based methodology involving Key Slice Extraction to find the most informative tumor cross-section followed by N4 Bias Field Correction and Otsu’s thresholding for intensity normalization. This allows the effective use of 2D Convolutional Neural Networks without incurring the computational cost associated with 3D processing. Using transfer learning from ResNet50 (Ultrasound/Mammography) and DenseNet121 (MRI), our centralized models reached state-of-the-art accuracies of 92.50%, 90.63%, and 92.00%, proving that a centralized multi-modal approach can be effective in enhancing diagnostic precision.