Oral cancer remains a major global health concern due to its high mortality rates. To address this, a novel hybrid deep learning framework is proposed that combines a Convolutional Neural Network (CNN) with Sand Cat Swarm Optimization (SCSO) for the effective classification of oral squamous cell carcinoma (OSCC) from histopathological images. The CNN architecture is designed to automatically extract discriminative features from images, while the SCSO algorithm fine-tunes crucial hyperparameters such as learning rate, batch size, and dropout rate to enhance model performance. The system is evaluated using a publicly available oral histopathology dataset, containing 100x and 400x magnification images of normal and OSCC tissues. Experimental results demonstrate that the SCSO-CNN model achieves superior classification performance, with an accuracy of 99.25% at 400x magnification, outperforming conventional CNN models like DenseNet201, Xception, and NASNetLarge. This optimized approach significantly improves the SCSO-CNN system’s performances that indicates the system potential for reliable and early detection of oral cancer in clinical settings.