Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Preprint
Data sources: ZENODO
addClaim

The Origami Structure of Transformer Representations: Crease-Plateau Geometry and Its Applications

Authors: Yang, Haolin;

The Origami Structure of Transformer Representations: Crease-Plateau Geometry and Its Applications

Abstract

We uncover a geometric structure universally present in transformer language models: the representation change ∆ℓ = Hℓ − Hℓ−1 between adjacent layers undergoes a characteristic dimensionality collapse at a small number of crease layers, followed by an extended plateau of nearly isometric processing. We analyze this structure through three complementary metrics—effective rank of ∆, pairwise token distance correlation across layers, and principal direction alignment of representation change vectors. On GPT-2 Small (12 layers), a single crease at layers 1–3 collapses ∆ to effective rank 1 (98% variance explained by the top singular vector), while the subsequent 7 plateau layers operate with near-perfect pairwise distance preservation (ρ = 0.962). Extending to GPT-2 Medium (24 layers), the crease consolidates to a single sharper fold at layer 3, and the plateau expands to 18 layers with improved isometry (ρ = 0.988). We demonstrate two practical applications of this geometric understanding: (1) crease-aware fine-tuning, where freezing the crease zone and training only plateau layers achieves identical domain adaptation performance to full-model fine-tuning at 77% parameter count; and (2) trainable v-rotation, a control primitive that rotates the dominant direction of the crease transformation via an SVD-constrained orthogonal matrix, achieving +62% target domain perplexity improvement while preserving base capabilities (+16% on held-out evaluation). Our findings establish a structural principle of transformer representations—that information gain is non-uniform across layers and concentrated at creases—and provide both diagnostic and interventional tools for exploiting this principle.

Powered by OpenAIRE graph
Found an issue? Give us feedback