Powered by OpenAIRE graph
Found an issue? Give us feedback
ZENODOarrow_drop_down
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

A Validation and Governance Framework for Multi-Agent LLM Scientific Software Development

Authors: Bass, Tim;

A Validation and Governance Framework for Multi-Agent LLM Scientific Software Development

Abstract

Large language models are non-deterministic systems whose outputs vary across runs, model versions, and context configurations. Existing benchmarks for LLM code generation evaluate correctness against synthetic test suites or competitive programming problems, not against peer-reviewed scientific data. This paper presents quantum_bench, a controlled multi-agent experiment in which generated code is validated against analytical values from Griffiths and Schroeter (2018), a standard graduate-level quantum mechanics reference. The experiment implements exact analytical solutions to five Tier 2 applied quantum mechanics problems in pure Ruby, using a two-agent LLM architecture: Claude as architect (prompt designer) and Codex as coder (Ruby implementer), with a human principal investigator as the non-delegable evaluator at each of 13 development gates. The primary finding is not about quantum mechanics. It is about the multi-agent workflow itself: Claude, acting as architect, repeatedly hallucinated experiment goals that were never stated, substituted its own interpretations despite explicit correction, and directed Codex down architecturally wrong paths. Codex performed correctly throughout, implementing what each prompt specified. In this Claude-as-architect, Codex-as-coder configuration, the architect role was the dominant source of failures, not the coder role. Claude errors are documented in five groups ordered by severity: goal substitution, incomplete refactors, context loss, prompt design gaps, and process violations, totaling 21 architect-level errors across 13 gates against zero architectural errors from Codex. All five quantum mechanics problems ultimately pass validation against Griffiths and Schroeter values. Governance and control methods based on experimental lessons learned are also summarized.

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Upload OA version
Are you the author of this publication? Upload your Open Access version to Zenodo!
It’s fast and easy, just two clicks!