An O(1) Modular Fingerprint of Input Norm Predicts Spectral Concentration in Transformer Hidden States — and the Correlation Sign Flips with Rotary Embeddings
An O(1) Modular Fingerprint of Input Norm Predicts Spectral Concentration in Transformer Hidden States — and the Correlation Sign Flips with Rotary Embeddings
We introduce an O(1)-time, dimension-independent diagnostic computed solely from raw token IDs that predicts spectral concentration in transformer final-layer hidden states. The diagnostic exhibits a reproducible sign flip from positive correlation under absolute positional embeddings to negative correlation under rotary embeddings (RoPE), with the negative correlation strengthening with model scale. Partial regression controlling for sequence length shows the signal is an independent geometric feature in large (>2B) RoPE models, while acting as a high-speed length proxy in smaller ones. A proof-of-concept routing gate using the diagnostic yields 57.2% fewer tokens processed and 53.0% lower energy consumption on Apple Silicon. Isolated tests confirm the inference bottleneck is memory-bound. # Modular LLM Diagnostic Reproducibility Package Paper: An O(1) Modular Fingerprint... (Lynch, 2025) ## Setup pip install torch transformers scipy numpy pandas statsmodels tqdm ## Reproduce Key Results - env_test.py: Verify Torch/MPS. - ROPEmodelTest.py: TinyLlama correlation (r=-0.287). - ROPEmodelTest2.py: Mistral-7B (r=-0.420). - test_models.py: GPT2/Phi-2. - extractor.py: Extract CSVs from logs. - CSV.py: Generate Table 1 + confounders. - measure_mac_01.py: Gated routing (57% savings; sudo needed for powermetrics). - measure_mac_isolated_test.py: Latency/energy tests. - proof3.py: Standalone λ₂ calc. prompts.txt/simple_prompts.txt: Test datasets. Raw logs/JSONs: Gating outputs.
transformer geometry, inference optimization, O(1) diagnostic, algebraic connectivity, large language models, rotary positional embeddings, mixture-of-experts routing, memory-bound inference, spectral concentration, energy efficiency, number theory in deep learning
transformer geometry, inference optimization, O(1) diagnostic, algebraic connectivity, large language models, rotary positional embeddings, mixture-of-experts routing, memory-bound inference, spectral concentration, energy efficiency, number theory in deep learning
selected citations These citations are derived from selected sources.This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).0 popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.Average influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).Average impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.Average
Citations provided by BIP!Popularity provided by BIP!