Microwave signal-based binary classification for detecting the presence of stroke presents a promising avenue for cost-effective and portable diagnosis. However, implementing this technology in real-world settings remains challenging due to difficulties in accurately modeling wave scattering, which often leads to suboptimal performance of classification systems. Our proposed deep learning framework addresses this issue by bypassing complex modeling tasks and directly extracting stroke signatures from wavelet-filtered microwave signals, combined with clinical variables and antenna measurements. Experiments conducted with total of 431 real-world stroke patients demonstrate that our system significantly outperforms recent classification approaches achieving 76.19% accuracy. Furthermore, our approach shows comparable performance to magnetic resonance imaging (MRI) based deep learning classification, in scenarios with limited imaging data. These results not only highlight the potential of microwave imaging (MWI) as a critical diagnostic tool but also suggest that MWI devices could become a viable option for clinical stroke diagnosis in the near future.