Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Software
Data sources: ZENODO
addClaim

AI-as-Infrastructure/aiinfra-atlas: Inter-rater upgrade

Authors: James Smithies;

AI-as-Infrastructure/aiinfra-atlas: Inter-rater upgrade

Abstract

This release prepares ATLAS for the inter-rater reliability focus group study (15 paid reviewers × 20 ratings each, 100 prompts). Highlights since v0.2.5: Inter-rater session seeding (make seed / seed-dry / seed-reset) with 100 curated 1901 parliamentary prompts across AU/NZ/UK/all corpora. Count-aware allocator (sort by inter_rater_count ASC, sha256 tiebreak) 60 s per-user cache: delivers the gold-standard 1:1 design of exactly 3 independent ratings per prompt under full attendance. INTER_RATER_DEFAULT_UI mode: reviewers land directly on the inter-rater workflow with the New Session button and privacy toggle hidden. Per-scale comment fields on all 6 Likert scales, mandatory for extreme ratings (1, 2, 5). INTER_RATER_SESSIONS_PER_USER bumped 10 -> 20 for the focus-group workload. Security: python-jose -> PyJWT migration, FastAPI security headers, prod-gated debug endpoint, dependency bumps for python-multipart, python-dotenv, langchain-text-splitters, langchain-openai, pytest. Documentation: canonical 'Default Focus-Group Study Configuration' section with a 10-item researcher checklist; seed commands in README.

Powered by OpenAIRE graph
Found an issue? Give us feedback