
Phase 1 methodological-pilot snapshot for the EMSE Methodological Articles submission, release tag v1.6.1-emse-option-b. Supersedes v0.3.0; see the Data Integrity narrative in the manuscript (§1.4, §3.5, §6.1) for the silent-fabrication discovery and correction that distinguishes v1.6 from v0.3.0 headline numbers. Pre-registration: OSF DOI 10.17605/OSF.IO/NKD6J, registered 2026-06-06 (Phase 2 design). Phase 1 Amendment 1 filed 2026-06-11. Contains: 600-pair specificity-inclusive corpus (manifest v2: 400 defects + 200 controls); patched judge wrappers with malformed propagation; dispatcher fail-fast gate; analyzer with the _flag_silent_fabrications retroactive filter; paired-test script; v1.6 manuscript and cover letter; the contaminated original parquets retained for audit reproducibility. Headline numbers (v1.6, integrity-audited subset): OpenAI gpt-5-codex 87.5% accuracy / 73.1% recall / 100% specificity (n=375); Claude Sonnet 4.5 78.4% accuracy / 45.8% recall / 100% specificity (n=208, attrition-confined to two of eight profiles); pairwise Cohen's κ = 0.679 [0.565, 0.788] on the n=208 mixed intersection, 0.423 [0.277, 0.583] on the defect-only n=83 intersection; McNemar p ≈ 6.0×10−8. Llama 3.2 Vision 11B is reported in Appendix A only owing to a documented multi-image input confound.
LLM-as-Judge, empirical software engineering, Cohen's kappa, visual regression testing, test oracles, pre-registration, Phase 1 pilot, specificity, silent fabrication, data integrity
LLM-as-Judge, empirical software engineering, Cohen's kappa, visual regression testing, test oracles, pre-registration, Phase 1 pilot, specificity, silent fabrication, data integrity
| selected citations These citations are derived from selected sources. This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | 0 | |
| popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network. | Average | |
| influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | Average | |
| impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network. | Average |
