Powered by OpenAIRE graph
Found an issue? Give us feedback
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/ ZENODOarrow_drop_down
image/svg+xml art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos Open Access logo, converted into svg, designed by PLoS. This version with transparent background. http://commons.wikimedia.org/wiki/File:Open_Access_logo_PLoS_white.svg art designer at PLoS, modified by Wikipedia users Nina, Beao, JakobVoss, and AnonMoos http://www.plos.org/
ZENODO
Preprint . 2026
License: CC BY
Data sources: ZENODO
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
ZENODO
Preprint . 2026
License: CC BY
Data sources: Datacite
versions View all 4 versions
addClaim

Runtime Fisher Spectral Sensitivity for Early Hallucination Detection in Edge-Deployed Language Models

Authors: Alekberli, Rahid; Karimov, Hikmat;

Runtime Fisher Spectral Sensitivity for Early Hallucination Detection in Edge-Deployed Language Models

Abstract

We study whether the spectral sensitivity of the per-token empirical Fisher Information Matrix (FIM), λmax(Ft), can serve as a lightweight, model-agnostic runtime signal for anticipating hallucination during autoregressive decoding on consumer edge hardware. Across four pre-planned experiments on twelve open-weight models run locally via Ollama and MLX on an Apple M5 (32GB unified memory), we establish, in turn: cross-run stability of the signal, its predictive correlation with hallucination onset, its early-warning lead time, and the effectiveness of an intervention triggered by the resulting alarm. Concretely, ten of twelve models satisfy a stability gate (coefficient of variation < 0.20); the signal correlates strongly with an entropy-based hallucination-onset label for non-thinking-mode decoding, most strongly for qwen3:32b (r= 0.962, p= 1.4 ×10−28, median r= 0.479 across nine non-thinking models) – a correlation we report as an upper bound given a noted oracle-circularity caveat between the label and the signal itself; a first-passage-time alarm flags onset at a median lead time of 27 tokens (IQR 12–39) with an 85.6% detection rate (t= 33.2, p= 2.2 ×10−114); and truncating generation once the alarm fires reduces post-onset token count by 66% (Cohen’s d= 1.95) without measurable quality degradation. Separately, we report and analyze a 244×gap between the analytically derived and the empirically calibrated KL-divergence alarm threshold under 4-bit (Q4 K M) quantization on this hardware, corroborated by a multi-model robustness check (range 64–287× across eight models) – a finding we frame not as a universal constant but as evidence for a general Calibration Necessity Protocol: safety-relevant inference-time thresholds must be empirically recalibrated per hardware/quantization configuration rather than transferred analytically. We report this drift, a full experimental and hardware configuration, and a set of explicit limitations – including a top-K = 20 logprob constraint, single-hardware-platform scope, and an open correction-mechanism gap – to support independent replication and extension.

Replaced the term 'pre-registered' throughout with 'pre-planned' and added an explanatory note clarifying that no formal, timestamped registration was lodged prior to running the experiments. This follows feedback received during peer review, per TMLR's guidance that authors should adjust claims to match available evidence rather than remove the underlying finding.

Related Organizations
Keywords

Fisher information, runtime monitoring, large language models, quantization, hallucination detection, on-device inference

  • BIP!
    Impact byBIP!
    selected citations
    These citations are derived from selected sources.
    This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    0
    popularity
    This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
    Average
    influence
    This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
    Average
    impulse
    This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
    Average
Powered by OpenAIRE graph
Found an issue? Give us feedback
selected citations
These citations are derived from selected sources.
This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Citations provided by BIP!
popularity
This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network.
BIP!Popularity provided by BIP!
influence
This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically).
BIP!Influence provided by BIP!
impulse
This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network.
BIP!Impulse provided by BIP!
0
Average
Average
Average
Green