File fragment classification (FFC) identifies the type of binary fragments without metadata or headers, representing a core challenge in digital forensics. While recent FFC methods often achieve high accuracy within their training domains, they consistently struggle to generalize to unseen datasets. In this study, we systematically evaluate cross-dataset generalization capabilities using three representative machine learning paradigms: a shallow Random Forest, an explainable deep forest model (gcForest) relying on hand-crafted features, and a state-of-the-art convolutional neural network (FIFTY-CNN) utilizing automatic feature extraction. Experiments were conducted using three benchmarks, including a newly collected and structured dataset, ITC-MNP (categorized by visual content: Medical, Nature, and People), and the public FFT-75 dataset, to evaluate how variations in content diversity and encoding settings affect classifier transferability. To interpret the structural mechanics behind varying cross-domain performance, we apply Feature Space Analysis (FSA) with interpretable Decision Trees, revealing exactly how specific statistical and structural features influence classification outcomes across dataset boundaries.Our results consistently demonstrate that models trained on datasets with richer content diversity, such as ITC-MNP, generalize more effectively to external domains. Conversely, internal dataset biases related to specific encoding pipelines significantly hinder transferability. Notably, gcForest's hand-crafted statistical features handled category-specific content distributions more robustly than the automated feature extraction of FIFTY-CNN. This study highlights the systemic limitations of current FFC evaluation methodologies, underscores the critical importance of semantic dataset composition, and demonstrates that combining content-diverse training with explainable feature representations is vital for developing reliable forensic classifiers.