Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Software
Data sources: ZENODO
addClaim

Dissociating Prompt Quality from Response Compliance in Automated Prompt-Engineering Assessment: evaluator, results and figure code

Authors: Hassen, Shaho Ismael;

Dissociating Prompt Quality from Response Compliance in Automated Prompt-Engineering Assessment: evaluator, results and figure code

Abstract

MATLAB implementation, per-item numeric results and figure code for the study "Dissociating Prompt Quality from Response Compliance in Automated Prompt-Engineering Assessment". A four-agent evaluator running on a locally hosted Qwen 2.5-7B-Instruct judge scores each prompt as an artifact before any response exists, then scores the resulting response twice on identical items: once against criteria extracted from that prompt, and once against a fixed rubric that never sees it. Verifiable constraints are checked with the IFEval reference implementation rather than model-generated labels. The whole pipeline runs on consumer CPU hardware with no external API calls. Contents: src/ the MATLAB agents and run scripts; scripts/ the IFEval verifier and the figure code; data/ a SHA-256 manifest of all 4,275 benchmark prompts and the filtered IFEval subset; results/ per-item results for the main run (n = 500) and the ablation (n = 120), the construct-validation output and the IFEval verification output. Corpus not redistributed. No prompt or response text from LMSYS-Chat-1M or WildChat-1M is included; the LMSYS agreement prohibits transfer to third parties and WildChat-1M carries flow-down obligations under the AI2 ImpACT licence. Prompts appear only as SHA-256 hashes, so an independently rebuilt corpus can be verified as identical to the one used here. The IFEval subset is included in full under Apache-2.0.

Powered by OpenAIRE graph
Found an issue? Give us feedback