Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Preprint . 2026
License: CC BY
Data sources: ZENODO
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
versions View all 2 versions
addClaim

PalQuant: Calibration-Free 7× KV Cache Compression via Adaptive Per-Block Quantization

Authors: Sharratt, Johnathan Paul;

PalQuant: Calibration-Free 7× KV Cache Compression via Adaptive Per-Block Quantization

Abstract

Existing online KV cache quantization methods optimise the compression primitive — uniform 2-bit (KIVI), outlier-aware fixed precision (KVQuant), random orthogonal rotation (TurboQuant) — and assign one format to all blocks at thepopulation level. We present PalQuant, an adaptive system that optimises the compression architecture: at inference time, each (layer, head, chunk) selects 4 active formats from a fixed 16-format codebook spanning 0.25–16 bits per element, with each block dynamically routed to one of the 4 based on its measured reconstruction error. Selection uses structurally distinct K and V error metrics matching their attention paths: mean-of-top-4 weighted absolute error for K (capturing softmax-amplified outliers), warp-mean squared error for V (the L2 budget for V’s attention-weighted contribution), with parameter-free sink protection via tanh-weighted Q·K alignment statistics. Deployed identically across three models with no per-model calibration data, PalQuant achieves 7.42× cache compression on Qwen3-8B, 7.04× on Qwen3 30B-A3B (MoE), and 5.02× on Llama-3.2-3B, validated end-to-end by a multi session story rewrite test where each concurrent session must preserve a distinct assigned character identity (name and gender) under aggressive compression. PalQuant passes the multi-session test at all 11 compression levels (C0C10) up to 7.42× CR, while uniform Q4 0 fails at 3.56× CR on Llama-3.2-3B and Qwen3-30B-A3B — even though uniform Q4 0 holds competitive bulk PPL, demonstrating a structural dissociation between perplexity and the multi-session quality metric (§4.5). The selection mechanism’s decisions transfer cleanly across MoE/dense, model families, and model sizes because pre-RoPE K/V activation structure is universal at the per-block level. PalQuant approaches Kitty’s offline calibrated compression band (7.42× vs ˜8×) without calibration data and substantially exceeds TurboQuant (4.6× at FP16-equivalent quality). Native CUDA kernels on a single RTX 4090 Mobile (16 GB) support up to 256/120/64 concurrent Llama-3.2-3B/Qwen3-30B-A3B/Qwen3-8B sessions at 168K/75K/41K totalKVcache tokens.

Keywords

CUDA kernel, KV cache compression, KV cache quantization, LLM inference

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Green