Accurate classification of file fragments is a critical challenge in digital forensics, particularly when formats exhibit similar structural or statistical properties. Recent deep learning approaches achieve high within-dataset accuracy but suffer from poor generalization, with performance degrading significantly on external datasets from different distributions. To address this, we develop format-specific discriminative features targeting frequently misclassified pairs identified via baseline models. Three datasets—ITC-MNP (training), ITC-Image, and FiFTy—are analyzed across five image formats: BMP, GIF, JPEG, PNG, and TIFF. Baselines using Random Forest (RF), XGBoost, and 1D convolutional neural network (1D-CNN) reveal confusion hotspots, particularly BMP vs. TIFF and BMP vs. JPEG. Specialized features, including Mean Absolute Byte Difference (MABD) capturing pixel smoothness, yield substantial gains of up to 12 percentage points (pp) on ITC-Image and 4 pp on FiFTy in our experimental setup. These results demonstrate targeted feature engineering as an effective, practical strategy to enhance model robustness in file fragment classification, opening avenues for forensics applications.