Abstract: Small-cohort clinical classification studies in neuroscience and healthcare routinely report accuracy figures that cannot be reproduced under rigorous evaluation. EEG disease detection is an exemplar of this broader problem, and what happens here is likely happening across neuroimaging and clinical machine learning more broadly. Across five resting-state cohorts spanning Alzheimer's disease, frontotemporal dementia, major depressive disorder, and schizophrenia, we identify three structural evaluation traps. Trap~1 (identity leakage): when epochs (fixed-length segments cut from a continuous EEG recording) from the same subject appear in both train and test sets, accuracy is inflated. Trap~2 (lucky folds): even with subject-disjoint splits, results can still vary strongly depending on which subjects are held out, with some folds reaching perfect accuracy. Trap~3 (objective mismatch): epoch accuracy and subject accuracy can rank models differently. All three traps replicate across every cohort regardless of biomarker strength, confirming they are structural properties of evaluation design. We propose a four-point reporting standard and release all prepared data and code in an accessible and extensible format.

Code and data availability All split manifests, YAML pipeline configurations, evaluation source code, and run-level results are available at: https://github.com/anon333science2026-afk/eeg-evaluation-traps Code and data availability All split manifests, YAML pipeline configurations, evaluation source code, and run-level results are available at: https://github.com/[your real repo URL or the anonymous one if you want to keep that for now]Datasets used (public)* OpenNeuro ds004504 (Miltiadous et al., 2023) — Alzheimer's & frontotemporal dementia EEG* OpenNeuro ds004904 (Miltiadous et al.) — frontotemporal dementia EEG* Mumtaz et al. (2016/2017) — major depressive disorder EEG (eyes-closed, eyes-open)* Olejarczyk & Jernajczyk (2017) — schizophrenia EEG (RepOD) Citation If you use this work, please cite this Zenodo deposit (DOI assigned on publication) and the underlying datasets.

Preprint, currently under peer review.