Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Preprint
Data sources: ZENODO
addClaim

Consensus Collapse in Language Model Pretraining

Authors: Charlton, Madison Matti;

Consensus Collapse in Language Model Pretraining

Abstract

This manuscript investigates source disagreement in language-model pretraining. It argues that standard next-token cross-entropy collapses source-conditioned disagreement structure into a source-frequency-weighted marginal, making source reliability invisible to the training objective. The paper develops a formal framework for this phenomenon ("consensus collapse"), proves a collapse theorem and a non-identifiability result for source-conditioned families under marginalization, derives consequences for attribution, calibration, and alignment, and evaluates the framework on controlled synthetic corpora. Included are the primary manuscript, a technical note documenting derivations and discarded hypotheses, experimental code, and supporting materials.

Powered by OpenAIRE graph
Found an issue? Give us feedback