Powered by OpenAIRE graph
Found an issue? Give us feedback
addClaim

Visual Oracle Bench (Phase 1, v1.6) — Two-Judge Synthetic-HTML Pilot for LLM-as-Judge Visual Regression Detection with Specificity Reporting

Authors: Malhotra, Suneet;

Visual Oracle Bench (Phase 1, v1.6) — Two-Judge Synthetic-HTML Pilot for LLM-as-Judge Visual Regression Detection with Specificity Reporting

Abstract

Phase 1 methodological-pilot snapshot for the EMSE Methodological Articles submission, release tag v1.6.1-emse-option-b. Supersedes v0.3.0; see the Data Integrity narrative in the manuscript (§1.4, §3.5, §6.1) for the silent-fabrication discovery and correction that distinguishes v1.6 from v0.3.0 headline numbers. Pre-registration: OSF DOI 10.17605/OSF.IO/NKD6J, registered 2026-06-06 (Phase 2 design). Phase 1 Amendment 1 filed 2026-06-11. Contains: 600-pair specificity-inclusive corpus (manifest v2: 400 defects + 200 controls); patched judge wrappers with malformed propagation; dispatcher fail-fast gate; analyzer with the _flag_silent_fabrications retroactive filter; paired-test script; v1.6 manuscript and cover letter; the contaminated original parquets retained for audit reproducibility. Headline numbers (v1.6, integrity-audited subset): OpenAI gpt-5-codex 87.5% accuracy / 73.1% recall / 100% specificity (n=375); Claude Sonnet 4.5 78.4% accuracy / 45.8% recall / 100% specificity (n=208, attrition-confined to two of eight profiles); pairwise Cohen's κ = 0.679 [0.565, 0.788] on the n=208 mixed intersection, 0.423 [0.277, 0.583] on the defect-only n=83 intersection; McNemar p ≈ 6.0×10−8. Llama 3.2 Vision 11B is reported in Appendix A only owing to a documented multi-image input confound.

Keywords

LLM-as-Judge, empirical software engineering, Cohen's kappa, visual regression testing, test oracles, pre-registration, Phase 1 pilot, specificity, silent fabrication, data integrity

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average