Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Preprint . 2026
License: CC BY
Data sources: ZENODO
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
versions View all 3 versions
addClaim

Emergent Depthwise Activation Structure in Decoder-Only Transformer Language Models: The Hau Curve and Its Early-Training Convergence Pattern

Authors: Hau, Adrian;

Emergent Depthwise Activation Structure in Decoder-Only Transformer Language Models: The Hau Curve and Its Early-Training Convergence Pattern

Abstract

Decoder-only transformer language models (LLMs) exhibit a highly regular internal organization in how computation is allocated across depth, across the model families, checkpoints, and scales examined. In this study, we identify a robust, tri-phasic depthwise activation geometry summarized by three operational landmarks: the Early Layer Extremum (𝐸𝑥), Mid-layer Plateau (𝑀𝑝), and Late Layer Surge (𝐿𝑠). By analyzing layer-wise ℓ1 activations across a 60-model core cohort (augmented with 17 additional models for boundary mapping, total 𝑁𝑐𝑜ℎ𝑜𝑟 𝑡 = 77), we show that these landmarks emerge early in pretraining (within roughly 7–15% of steps in the examined runs) and then remain stable. Crucially, the available evidence is consistent with an emergence zone centered around 𝑁𝑙𝑎𝑦𝑒𝑟 𝑠 ≈ 8–12 layers rather than a sharp universal threshold. Below this approximate range, the proximity of the 𝐸𝑥 and the 𝐿𝑠 leads to phase congestion, where limited depth resolution causes early- and late-phase activation regimes to overlap and suppress clear expression of the 𝑀𝑝. Above this range, the 𝐸𝑥 and the 𝐿𝑠 decouple sufficiently to provide the computational real estate for a stabilized 𝑀𝑝. Together, these results suggest that increasing decoder depth is not merely a quantitative increase in parameters, but also an expansion of the geometric room available for a recurring activation structure. These structural regularities offer a new lens for model analysis, bridging the gap between low-level mechanistic interpretability and high-level behavioral scaling laws. Notably, quantities such as layerwise activation magnitude—often treated as secondary or unstable—are shown here to track a recurring architecture-level geometric structure that emerges early and persists across sufficiently deep decoder-only models.

Keywords

Depthwise activations, Large language models, Mechanistic Interpretability

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Green