Methods & Protocol · Validation of scientific ML Preprint · 5 June 2026 Mechanism Audit for Learned Surrogates Mechanism-Faithful Validation of Differentiable Simulators and Learned Surrogates A tiered audit protocol that separates a model which has learned the mechanism from one that only reproduces the outputs — with a worked network-dynamics benchmark János Gábor Melegh Independent computational study Companion to P02_TOPOLOGY_PHASE_MAP·Manuscript ID: P02_SURROGATE_VALIDATION · v0.1·5 June 2026 Relationship to the preliminary study. This protocol generalises the three-run audit architecture of the network-voter-dynamics study (V04 phenomenology → P02_05 mechanism audit → P02_06b matched-degree residual audit). That study is used here not merely as a citation but as a fully worked validation benchmark: its four mechanism signatures — the matched-degree residual, the non-monotone collapse window, the sharp degree–collapse boundary, and the asymmetric bistability — are reinterpreted as discriminating probes that a faithful surrogate must reproduce and a hollow one cannot. All mathematics carried over from that study is restated in Section V. Abstract Learned emulators — graph neural networks, neural operators, and differentiable solvers — increasingly displace explicit numerical simulation across the physical and design sciences. The central risk of this substitution is rarely a large forward error; it is that a surrogate attains excellent agreement on the training manifold while having learned a shortcut rather than the generative mechanism. Such a model passes every output-matching test yet fails silently the moment it is used for the purpose that motivated it: extrapolation, intervention, or gradient-based inverse design. We formalise the distinction between output fidelity and mechanism fidelity by treating a surrogate not as an approximation to a point map but as an approximation to a response operator: the object of interest is how the model's output changes under controlled interventions, not the output itself. We define a three-tier audit — forward fidelity (Tier 1), interventional and invariance fidelity (Tier 2), and extrapolation, matched-control, and boundary fidelity (Tier 3) — together with a fourth, transversal requirement for differentiable surrogates: gradient and adjoint fidelity, which forward accuracy does not imply and which is exactly the property exploited by inverse-design and PDE-constrained optimisation. We then instantiate the protocol concretely. The preliminary network-dynamics study supplies four quantitative mechanism signatures that become reference probes: a residual structural spread of $\approx 0.195$ at matched mean degree; a non-monotone collapse window peaking near $f_{\mathrm{neg}}\!\approx\!0.35$; a steep degree–collapse boundary progressing through $\{0,\,0.11,\,0.79,\,0.97,\,1.00\}$; and a $\sim\!12\times$ asymmetric bistability under degree-targeted perturbation. Each maps to a probe, a metric, a reference value, and a pass criterion. We give the validator's own validation — positive (hollow) and negative (faithful) controls and a discriminative-power requirement — a failure-mode taxonomy, the statistical-power and compute-budget accounting needed for deployment, and a broad survey of application contexts and what is at stake when blind trust in an empty model is scaled across a discipline. Keywords — surrogate validation · differentiable simulation · neural operators · graph neural networks · mechanism audit · interventional testing · adjoint/Jacobian consistency · out-of-distribution generalisation · shortcut learning · scientific machine learning · digital twins IIntroduction A growing fraction of scientific and engineering computation is no longer executed; it is predicted. Where a partial differential equation was once integrated, a neural operator now maps boundary data to a solution field in a single forward pass; where a costly molecular or fluid simulation once ran, a graph neural network emulates the trajectory. The economic and scientific case is overwhelming — orders-of-magnitude speed-ups, end-to-end differentiability, deployment inside optimisation loops — and adoption is accordingly rapid. The risk introduced by this substitution is subtle precisely because the obvious failure mode — a large prediction error — is the one practitioners already guard against. A surrogate is normally accepted on the basis of a held-out test error that is small. But a small test error on data drawn from the same distribution as the training set certifies only that the model interpolates well on that manifold. It says nothing about whether the model has internalised the rule that generated the data, or has instead latched onto a correlate that happens to track the answer within the sampled region. The literature on shortcut learning has documented this gap repeatedly in perception tasks [1]; in scientific surrogates the consequences are sharper, because the surrogate is typically deployed for exactly the things its training distribution did not cover. The matter is most acute for differentiable surrogates. The reason one builds a differentiable emulator is to use its gradients — for inverse design, for parameter estimation, for PDE-constrained optimisation. Yet forward accuracy and gradient accuracy are distinct properties: a model can predict the forward map well while producing a Jacobian that is wrong in magnitude, or even in sign, along the directions that matter. A validation procedure that inspects only the forward pass is structurally blind to the failure that most directly defeats the surrogate's purpose. Central thesis A surrogate does not approximate a point; it approximates a response. The validation question is therefore never whether $\mathcal{S}(x)\approx\mathcal{T}(x)$ on the training manifold — a sufficiently large lookup table achieves that — but whether the surrogate's derivative with respect to interventions matches the true system's. The "empty" model is precisely the one whose forward error is small while its interventional response, off-manifold behaviour, and gradient field diverge from the reference. This paper turns that thesis into an operational protocol. Section II formalises the three notions of fidelity and explains why output matching underdetermines mechanism. Section III defines the tiered audit and its verdict logic. Section IV treats the differentiable-specific requirement of gradient and adjoint fidelity. Section V draws the worked benchmark from the preliminary network-dynamics study, restating its mathematics and converting each of its mechanism signatures into a concrete probe. Section VI assembles the probe battery for practice, including the controls that validate the validator, and the statistical-power and compute accounting required to run it on a budget. Section VII gives a failure-mode taxonomy. Section VIII surveys the application landscape and what is at stake at scale. Section IX states the scope and the honest limits of what such a battery can and cannot certify. IIThe validation problem, formalised II.AThe surrogate as a response operator Let the trusted reference simulator be a map $\mathcal{T}:\mathcal{X}\to\mathcal{Y}$ from a parameter/configuration space $\mathcal{X}$ to an observable space $\mathcal{Y}$, and let the surrogate be $\mathcal{S}_\theta:\mathcal{X}\to\mathcal{Y}$, trained on samples drawn from a measure $\mathcal{D}$ supported on a sub-manifold $\mathcal{M}=\operatorname{supp}\mathcal{D}\subset\mathcal{X}$. The reference need not be cheap — it is invoked sparingly, and only by the validator — but it must be trustworthy in the regime probed. The naive acceptance criterion is forward fidelity on $\mathcal{M}$: $$\varepsilon_1 \;=\; \mathbb{E}_{x\sim\mathcal{D}}\!\big[\,d_{\mathcal{Y}}\!\big(\mathcal{S}_\theta(x),\,\mathcal{T}(x)\big)\big]\;+\;\lambda\, W\!\big(\mathcal{S}_{\theta\#}\mathcal{D},\;\mathcal{T}_{\#}\mathcal{D}\big)\;\le\;\tau_1, \tag{1}$$ where $d_{\mathcal{Y}}$ is a pointwise metric, $W$ is a distributional distance (e.g. a Wasserstein or maximum-mean-discrepancy term comparing the pushforwards, so that tails and not only means are matched), and $\lambda$ weights the two. Condition (1) is necessary. It is also weak: it constrains $\mathcal{S}_\theta$ only on $\mathcal{M}$, and an uncountable family of maps agreeing on $\mathcal{M}$ disagree arbitrarily elsewhere and under intervention. The mechanism the surrogate is meant to carry is not a value but a response. Let $I_\delta:\mathcal{X}\to\mathcal{X}$ denote an intervention — a controlled, semantically meaningful modification of the configuration parametrised by $\delta$ (adding edges to a graph, perturbing a boundary condition, removing the highest-degree nodes, shifting a material parameter). Define the response operators $$\mathcal{R}_{\mathcal{T}}(x,\delta)\;=\;\mathcal{T}\!\big(I_\delta x\big)-\mathcal{T}(x),\qquad \mathcal{R}_{\mathcal{S}}(x,\delta)\;=\;\mathcal{S}_\theta\!\big(I_\delta x\big)-\mathcal{S}_\theta(x). \tag{2}$$ Mechanism fidelity is agreement of responses, not of outputs. A model that reproduces $\mathcal{T}$ on $\mathcal{M}$ but predicts the wrong sign or magnitude of $\mathcal{R}_{\mathcal{T}}$ under a relevant $I_\delta$ has reproduced the outputs without the mechanism — it is the empty model of the thesis. II.BWhy output matching underdetermines mechanism The geometric picture is simple and worth stating explicitly. Two surrogates can coincide to within $\tau_1$ everywhere on the training manifold and yet have Jacobians that differ along directions transverse to it. Forward fidelity constrains the function's values on $\mathcal{M}$; mechanism fidelity constrains its derivatives, including derivatives that point off $\mathcal{M}$. The two are independent whenever the interventions of interest move configurations off the training support — which, for a surrogate used in design or extrapolation, is the entire point. Figure 1 makes the underdetermination concrete. intervention amplitude δ (off the training manifold →) observable 𝒴 training manifold ℳ reference 𝒯 faithful 𝒮• hollow 𝒮∘ divergence appears onlyunder intervention Fig. 1. Output matching underdetermines mechanism. Inside the training manifold (shaded) the reference $\mathcal{T}$, a faithful surrogate $\mathcal{S}^{\bullet}$, and a hollow surrogate $\mathcal{S}^{\circ}$ are indistinguishable; all satisfy the forward criterion (1). They diverge only once an intervention drives the configuration off $\mathcal{M}$: $\mathcal{S}^{\bullet}$ tracks the reference's non-monotone response, while $\mathcal{S}^{\circ}$ extrapolates the simplest correlate it learned. A Tier-1-only audit cannot tell them apart; the response operator (2) can. II.CInvariances as a second mechanism constraint Many reference systems carry exact invariances — conservation laws, symmetries, or quantities provably independent of a structural variable. If a group $G$ acts on $\mathcal{X}$ with $\mathcal{T}(g\!\cdot\!x)=\mathcal{T}(x)$ for all $g\in G$, then a mechanism-faithful surrogate must inherit the invariance to tolerance: $$\sup_{g\in G}\,\big\|\mathcal{S}_\theta(g\!\cdot\!x)-\mathcal{S}_\theta(x)\big\|\;\le\;\tau_{\mathrm{inv}}. \tag{3}$$ Invariance violations are diagnostic: a surrogate that lets an observable drift under a transformation the reference holds fixed has manufactured a dependence that does not exist in the mechanism. The preliminary study supplies exactly such an invariant — the topology-independence of the mean local stress, established at $p=1.00$ on a Kruskal–Wallis test — which Section V converts into an invariance probe. IIIThe tiered audit protocol The protocol stratifies validation into three tiers of increasing stringency, mirroring the three-run escalation of the preliminary study — phenomenological replication, mechanism audit, matched-degree residual audit — generalised to arbitrary surrogates. A surrogate is assigned the highest tier it clears; passing a tier is necessary but not sufficient for the next. TIER 1 · Forward fidelity held-out error · distribution match · calibration → necessary, not sufficient TIER 2 · Mechanism fidelity interventional response · invariances · directional asymmetry TIER 3 · Extrapolation & control off-manifold · matched-confounder residual · boundary sharpness GRADIENT · ADJOINT TRACK differentiable only increasing stringency Fig. 2. The tiered protocol. Three nested tiers escalate from forward fidelity to mechanism fidelity to extrapolation-and-control fidelity. For differentiable surrogates a transversal track audits the gradient/adjoint field at every tier, because gradient quality is independent of forward quality (Section IV). A model's verdict is the highest tier it clears. III.ATier 1 — forward fidelity Condition (1), with both a pointwise and a distributional term, plus probabilistic calibration where the surrogate emits uncertainties. This is the standard test; the protocol's stance is only that clearing it earns no conclusion beyond interpolation on $\mathcal{M}$. III.BTier 2 — mechanism fidelity Over a curated probe set $\mathcal{P}=\{(x_i,\delta_i)\}$ of (state, intervention) pairs chosen to exercise the system's known mechanisms, require interventional consistency, $$\varepsilon_2\;=\;\max_{(x,\delta)\in\mathcal{P}}\;\big\|\mathcal{R}_{\mathcal{S}}(x,\delta)-\mathcal{R}_{\mathcal{T}}(x,\delta)\big\|\;\le\;\tau_2, \tag{4}$$ together with the invariance condition (3). Crucially, (4) compares differences, so a surrogate cannot pass by being uniformly close in absolute value while getting every response wrong in the same direction. Directional and asymmetric responses — where $\mathcal{R}_{\mathcal{T}}(x,+\delta)\neq-\mathcal{R}_{\mathcal{T}}(x,-\delta)$ — are the most discriminating members of $\mathcal{P}$, because a surrogate that assumes a smooth symmetric response cannot reproduce them at any forward accuracy. III.CTier 3 — extrapolation, matched control, and boundary sharpness Three independent stresses, each clearable or not: Off-manifold generalisation. Evaluate at $x$ with $\operatorname{dist}(x,\mathcal{M})\ge\rho$ along controlled directions, and require (1) to hold with a relaxed but stated tolerance $\tau_3^{\mathrm{ood}}$ as a function of $\rho$. A monotonically and predictably degrading error is a pass; a sudden collapse is a fail. Matched-confounder residual. When a dominant predictor $c$ explains most of the variance, hold $c$ fixed and require the surrogate to reproduce the residual structure that survives the control. With $\Sigma_{\mathcal{T}}(c)=\operatorname{Var}_{\text{family}}[\mathcal{T}\mid c]$ the reference residual spread at fixed $c$, require $\big|\Sigma_{\mathcal{S}}(c)-\Sigma_{\mathcal{T}}(c)\big|\le\tau_3^{\mathrm{res}}$. This catches the most common scientific shortcut: regressing on the dominant variable and discarding the rest. Boundary sharpness. At a transition surface, compare the location and the width of the boundary, not merely the values on either side. A surrogate that smears a sharp transition incurs little forward penalty but destroys the physics; the width-ratio metric $w_{\mathcal{S}}/w_{\mathcal{T}}$ exposes it. III.DVerdict logic The protocol returns a tier verdict and, on failure, the offending probe — not a single scalar. A useful report reads, for example: Tier-1 PASS · Tier-2 PASS · Tier-3 FAIL (boundary width 2.3× reference). This is what makes the audit actionable rather than merely evaluative: the failing probe localises the defect to a mechanism, and the tier locates it on the interpolation–intervention–extrapolation axis. IVDifferentiable surrogates: gradient and adjoint fidelity For a differentiable surrogate the gradient is not a by-product; it is the deliverable. Neural operators and differentiable solvers are built so that their outputs can be differentiated with respect to inputs and parameters, and that derivative is then driven through an optimiser for inverse design, data assimilation, or control. Forward accuracy says nothing about whether that derivative is trustworthy. IV.AWhy forward accuracy does not imply gradient accuracy Two functions can be uniformly close while their derivatives are far apart: adding a small, high-frequency component leaves the values nearly unchanged but corrupts the slope. Learned surrogates are particularly prone to this — the training loss penalises output error, not Jacobian error, so the gradient field is unconstrained except indirectly. The result is a model with small $\varepsilon_1$ whose Jacobian is wrong in magnitude or, in the worst case, in sign along the search directions that an optimiser will follow. IV.BJacobian consistency Let $J_{\mathcal{S}}(x)=\partial\mathcal{S}_\theta/\partial x$ be obtained by automatic differentiation and $J_{\mathcal{T}}(x)$ by the reference adjoint (or, where unavailable, by a finite-difference probe of the reference). Require both directional and magnitude agreement, $$\cos\angle\!\big(J_{\mathcal{S}},J_{\mathcal{T}}\big)=\frac{\langle J_{\mathcal{S}},\,J_{\mathcal{T}}\rangle_F}{\|J_{\mathcal{S}}\|_F\,\|J_{\mathcal{T}}\|_F}\;\ge\;1-\tau_g, \qquad \frac{\|J_{\mathcal{S}}-J_{\mathcal{T}}\|_F}{\|J_{\mathcal{T}}\|_F}\;\le\;\tau_g'. \tag{5}$$ The cosine term is the decisive one for optimisation: a gradient that points roughly the right way still descends, whereas a misaligned gradient sends the optimiser astray regardless of forward accuracy. Where the full Jacobian is too large to materialise, random-projection (Hutchinson-style) sketches of $J_{\mathcal{S}}v$ versus $J_{\mathcal{T}}v$ for probe vectors $v$ estimate (5) at a fraction of the cost. IV.CThe optimisation surrogate-gap The end-to-end, user-facing test is whether an optimisation performed through the surrogate yields a design that is good under the reference. Let $L$ be a design objective and $$x^\star_{\mathcal{S}}=\arg\min_{x} L\!\big(\mathcal{S}_\theta(x)\big),\qquad \Gamma\;=\;L\!\big(\mathcal{T}(x^\star_{\mathcal{S}})\big)\;-\;\min_{x}L\!\big(\mathcal{T}(x)\big)\;\ge\;0. \tag{6}$$ The surrogate-gap $\Gamma$ is the regret incurred by trusting the surrogate's gradient field for design: it is zero when the surrogate-optimal design is reference-optimal, and large when the surrogate's smooth landscape hides the reference's true optimum. A surrogate may clear (1) and even (5) on the training manifold and still exhibit a large $\Gamma$ if the optimiser is driven into a region where (5) fails — which is why the gradient track runs through all three tiers in Figure 2 rather than sitting beside Tier 1. Practical primacy If a differentiable surrogate is validated for one property only, it should be (6): the optimisation surrogate-gap is the quantity the deployment actually depends on, and it is the one a forward-error report never measures. VA worked benchmark from network voter dynamics The preliminary study provides a complete, quantitatively characterised dynamical system together with four mechanism signatures already measured to high precision. It is an ideal validation benchmark precisely because the mechanisms are non-obvious and a naive surrogate will get them wrong in characteristic ways. We first restate the relevant mathematics, then convert each signature into a probe. V.AThe reference system and its observables On a connected undirected network $G=(V,E)$ with $|V|=n$, each node $i$ carries a discrete state $s_i\in\{0,\dots,k-1\}$ with $k=4$, and each edge a sign $\sigma_{ij}\in\{+1,-1\}$; the fraction of negative edges $f_{\mathrm{neg}}$ is the external control. After relaxation each node yields a local stress $T_i\in[0,1]$ and a local admissibility $\psi_i\in[0,1]$. The principal observable is the across-node Spearman rank correlation of the two, $$H_3\;=\;\rho_S\!\big(T_\cdot,\psi_\cdot\big)\;=\;1-\frac{6\sum_i d_i^{\,2}}{n\,(n^2-1)}, \tag{7}$$ with $d_i$ the rank difference of $T_i$ and $\psi_i$. Auxiliary observables include the mean admissibility $\langle\psi\rangle$, the zero-admissibility fraction $\pi_0$, and a binary collapse flag, true when $\langle\psi\rangle0.95$. The reference simulator for the benchmark is the map $\mathcal{T}:(G,f_{\mathrm{neg}})\mapsto(H_3,\langle\psi\rangle,\text{collapse})$; a surrogate would be, e.g., a graph neural network ingesting $(G,f_{\mathrm{neg}})$ and predicting these observables. V.BThe decomposed mechanism the surrogate must reproduce A random-forest variance attribution on the mechanism-audit dataset finds mean degree $\langle k\rangle$ to be the dominant predictor of $H_3$ (importance $0.94$, cross-validated $R^2=0.99$) and of $\langle\psi\rangle$ (importance $0.83$). The mechanism, however, does not reduce to that single predictor; the study's causal decomposition reads $$H_3\;=\;\underbrace{g(\langle k\rangle)}_{\text{dominant variance}}\;+\;\underbrace{h\!\big(\mathrm{CV}(k),\,\langle C\rangle,\,\langle k\rangle\!\times\!\langle C\rangle\big)}_{\text{structural residual}}, \tag{8}$$ $$P(\text{collapse})\;\approx\;F\!\big(\langle k\rangle,\,f_{\mathrm{neg}},\,\langle C\rangle,\,\mathrm{CV}(k)\big), \tag{9}$$ where $F$ is non-monotone in $f_{\mathrm{neg}}$ at fixed $\langle k\rangle$, steep and monotone in $\langle k\rangle$, and asymmetric under degree-distribution perturbation. Equation (8) is the crux: a surrogate that learns only $g(\langle k\rangle)$ — the high-importance shortcut — will pass Tier 1 because $g$ carries most of the variance, yet fail the moment the residual $h$ is probed under matched $\langle k\rangle$. This is the empty model in concrete form, and the random-forest importance of $0.94$ is precisely the bait. The exact regular-graph limit $H_3=-1$ (to machine precision on lattice and random-regular ensembles) is consistent with a mean-field locking of $T_i$ and $\psi_i$: at constant degree each node's stress and admissibility are strict monotone-decreasing functions of the same local frustrated-edge count, driving the Spearman statistic to its bound. A surrogate that smooths this exact value away — a near-universal tendency of regressors — has lost a mechanism that the reference carries exactly. V.CThe four signatures as probes Each of the study's four headline findings becomes a probe. We display them as the small-multiple panel of Figure 3, echoing the source figures, and tabulate the metrics in Table II. P1 · Matched-degree residual ≈0.195 at fixed ⟨k⟩=8, vary f_neg P2 · Non-monotone window peak ≈0.35 collapse rate vs f_neg P3 · Sharp degree boundary 6 8 12 16 24 collapse rate vs ⟨k⟩ order-of-magjump 8→12 P4 · Asymmetric bistability 0 −60 pp degree-targeted node-thinning +5 pp edge addition asymmetry ratio ≈ 12× Fig. 3. Four mechanism signatures, four probes. Reproduced schematically from the preliminary study. P1: at matched $\langle k\rangle=8$, seven topology families fan out to a residual $H_3$ spread of $\approx0.195$ — the structure that survives the dominant-predictor control. P2: the collapse rate is non-monotone in $f_{\mathrm{neg}}$, peaking near $0.35$. P3: the degree–collapse boundary is steep, progressing $\{0,0.11,0.79,0.97,1.00\}$ with an order-of-magnitude jump across $\langle k\rangle\in[8,12]$. P4: $10\%$ degree-targeted node-thinning reduces collapse by $\sim60$ pp while equal edge addition deepens it by only $\sim5$ pp — a $\sim12\times$ directional asymmetry. A hollow surrogate fails all four in characteristic ways; a faithful one reproduces them within tolerance. Table I. The mechanism signatures of the preliminary study mapped to probes, tiers, metrics, reference values, and the failure each exposes. Tolerances $\tau$ are deployment-specific and stated here as illustrative starting points. Probe Tier Reference signature Metric Failure it exposes P1 · matched-degree residual 3 residual spread $\Sigma_{\mathcal{T}}\!\approx\!0.195$ at $\langle k\rangle\!=\!8$ $|\Sigma_{\mathcal{S}}\!-\!\Sigma_{\mathcal{T}}|\le\tau_3^{\mathrm{res}}$ shortcut on dominant predictor $\langle k\rangle$ (eq. 8) P2 · non-monotone window 2 collapse peak at $f_{\mathrm{neg}}\!\approx\!0.35$ extremum location & amplitude monotone hallucination of a non-monotone response P3 · sharp boundary 3 $\{0,0.11,0.79,0.97,1.00\}$ across $\langle k\rangle$ boundary location & width ratio $w_{\mathcal{S}}/w_{\mathcal{T}}$ smearing of a steep transition P4 · interventional asymmetry 2 $\sim\!12\times$ uncollapse/collapse ratio $A_{\mathcal{S}}=\Delta P_{\downarrow}/\Delta P_{\uparrow}$ vs $A_{\mathcal{T}}$ symmetric-response assumption (eq. 4) INV · stress invariance 2 $\langle T\rangle$ topology-independent, $p\!=\!1.00$ $\sup_g\|\mathcal{S}(g\!\cdot\!x)\!-\!\mathcal{S}(x)\|\le\tau_{\mathrm{inv}}$ manufactured dependence on an invariant EXACT · regular-graph limit 2 $H_3\!=\!-1$ to machine precision $|\mathcal{S}\!-\!(-1)|$ on regular ensembles smoothing-away of an exact mean-field locking VIThe probe battery in practice VI.AValidating the validator: positive and negative controls A battery is only credible if it is shown to discriminate. The protocol therefore requires two controls. The positive (hollow) control $\mathcal{S}^{\circ}$ is a surrogate constructed to be empty by design — for the benchmark of Section V, a model regressing $H_3$ on $\langle k\rangle$ alone, i.e. learning only the $g(\langle k\rangle)$ term of (8). By construction it clears Tier 1, and it must fail the residual probe P1 and the asymmetry probe P4. The negative (faithful) control $\mathcal{S}^{\bullet}$ is a mechanism-respecting model — physics-informed, or the reference itself subsampled — which must clear all tiers. The battery score $V$ is admissible only if it separates them, $$V(\mathcal{S}^{\bullet})-V(\mathcal{S}^{\circ})\;\ge\;\Delta_{\min}. \tag{10}$$ If (10) fails, the battery — not the surrogate — is at fault, and the probes must be sharpened before any candidate model is judged. This step is what prevents a validation suite from being a rubber stamp. VI.BStatistical power and compute budget Several probes — collapse rates, asymmetry ratios — are Bernoulli proportions estimated from finite samples, so the battery needs a power calculation, not merely a tolerance. To resolve a proportion difference $\Delta$ against a baseline $\bar p$ at confidence $z$, the per-condition sample obeys $$N\;\gtrsim\;\frac{z^2\,\bar p\,(1-\bar p)}{\Delta^{2}}. \tag{11}$$ The preliminary study is instructive here: its bistability module ran at $\sim\!340$ cells per condition, and the headline asymmetry estimate was stable as coverage grew from $\sim\!22\%$ to $\sim\!73\%$ (the random-regular effect moving only from $-54$ to $-60$ pp). This is the kind of stability-under-growing-coverage evidence a deployed battery should record, since the reference simulator is the expensive ingredient and the audit must be runnable on a partial budget with a stated, monotonically tightening verdict. A practical battery therefore reports coverage alongside each verdict and refuses to upgrade a tier verdict until the supporting probe has reached its (11)-implied sample. VI.CA reference-simulator interface For the battery to be reusable across systems it needs a minimal, system-agnostic contract with the reference. Three operations suffice: a forward evaluation $\mathcal{T}(x)$; an intervention $\mathcal{T}(I_\delta x)$ for the response operator (2); and, where the reference admits it, an adjoint/gradient $J_{\mathcal{T}}(x)$ for (5)–(6). A probe library then expresses P1–P4, the invariance and exact-limit probes, and the gradient probes against this contract, and an automated report emits the per-tier verdict and the offending probe. Anything that implements the three-operation contract — a PDE solver, an agent-based model, a Monte-Carlo pipeline, the voter dynamics of Section V — can be audited by the same battery. Integration into a training loop follows naturally: run Tier 1 after every fit and the heavier Tier 2/3 probes at milestones, so the audit becomes continuous rather than a post-hoc ceremony. VIIFailure-mode taxonomy Naming the failure classes makes the audit diagnostic: each mode is caught by a specific tier and presents a specific symptom, with a direct analogue in the worked benchmark. Table II. Failure modes, the tier that exposes each, the symptom, and the benchmark analogue from Section V. Failure mode Caught by Symptom Benchmark analogue Dominant-predictor shortcut Tier 3 · P1 residual spread collapses to ~0 at matched confounder learning only $g(\langle k\rangle)$ in eq. (8) Manifold memorisation Tier 3 · OOD error jumps discontinuously off $\mathcal{M}$ fit only where $f_{\mathrm{neg}}\in$ training range Transition smearing Tier 3 · P3 boundary width ratio $\gg 1$ degree–collapse boundary flattened Monotonicity hallucination Tier 2 · P2 extremum missing or displaced collapse window replaced by a monotone trend Symmetric-response error Tier 2 · P4 asymmetry ratio $A_{\mathcal{S}}\approx 1$ $12\times$ bistability reproduced as $\sim1\times$ Broken invariant Tier 2 · INV observable drifts under a fixed-quantity transform mean stress made topology-dependent Gradient corruption Gradient track good forward, misaligned/sign-wrong $J$, large $\Gamma$ inverse-design optimum poor under reference VIIIApplication landscape and what is at stake The protocol is system-agnostic; its value scales with the consequence of trusting an empty model. The contexts below span the families where learned surrogates are displacing explicit computation fastest, and in each the meaning of "mechanism" — and the dominant failure risk — is specific. Table III. Application contexts, the surrogate types in use, what mechanism fidelity means in each, and the dominant failure risk a forward-only audit misses. Domain Typical surrogate What "mechanism" means here Dominant failure risk PDE / fluid & climate neural operators (FNO, DeepONet) [2][3] conservation laws; correct response to forcing energy/mass drift; smeared shocks & fronts Molecular / materials GNN potentials, force fields energy conservation; correct forces (= gradients) good energies, wrong forces → unstable dynamics Inverse design / control differentiable solvers trustworthy Jacobian along the search path large optimisation surrogate-gap $\Gamma$ (eq. 6) Epidemic / network dynamics graph emulators interventional response; phase boundaries shortcut on a dominant predictor (the eq. 8 case) Digital twins real-time DeepONet inference [3] off-nominal extrapolation; correct sensitivities silent failure outside the calibrated envelope Cosmology / astrophysics simulation-based emulators sharp features; parameter sensitivities washed-out features bias downstream inference The unifying lesson is that the failure missed by a forward-only audit is the same in every domain: the surrogate reproduces the bulk of the variance — captured by a dominant, easily-learned correlate — and discards the structured residual, the sharp feature, the directional response, or the gradient, that the surrogate was deployed to exploit. As emulators move from research prototypes into design pipelines and operational digital twins, this residual is no longer a curiosity; it is the part of the answer on which the decision turns. A discipline that scales blind trust in empty models propagates a systematic, correlated bias across every result downstream of them — a failure mode whose prevention is cheap relative to its consequences, and most cheaply done before deployment, by a battery that probes mechanism rather than output. IXScope and limitations The protocol is a falsification instrument, not a certificate. It can exclude specific, named classes of failure — those of Table II — by exhibiting a probe on which a candidate diverges from the reference. It cannot prove that a surrogate "has learned the physics," and it should never be presented as doing so. Three limits bound it honestly. Probe coverage. The battery tests the mechanisms encoded in $\mathcal{P}$. A failure mode no probe exercises is invisible; the battery's reach is exactly the union of its probes, and constructing $\mathcal{P}$ requires domain knowledge of which interventions and invariants matter. The worked benchmark is valuable precisely because its mechanisms were independently characterised first. Reference trust. The audit is only as sound as the reference $\mathcal{T}$ in the regime probed. Where the reference is itself uncertain off-manifold, the OOD probe degrades to a consistency check rather than a correctness check, and the report must say so. Tolerances are choices. The thresholds $\tau_1,\tau_2,\tau_3,\tau_g,\Delta_{\min}$ encode the deployment's risk appetite and must be set, and disclosed, before judging a candidate — otherwise the verdict can be tuned to a foregone conclusion. The controls of Section VI.A and the power calculation of (11) constrain these choices but do not eliminate the judgement. Framed this way, the protocol's claim is modest and defensible: it is an inexpensive, automatable screen that catches the empty-model failure before scale, in the same spirit, and with the same falsificationist logic, as the three-run mechanism audit of the preliminary study. XConclusion Learned surrogates are accepted, almost universally, on forward error — a quantity that certifies interpolation on the training manifold and nothing more. We have argued that the property that matters is mechanism fidelity: agreement of the surrogate's response to intervention, its invariances, its behaviour off-manifold and under matched controls, and — for differentiable models — its gradient field. We formalised these as a three-tier audit with a transversal gradient track, gave each tier a precise condition, and instantiated the whole on a worked network-dynamics benchmark whose four mechanism signatures become reference probes with measured tolerances. We specified the controls that validate the validator, the power and budget accounting that make it deployable, a failure-mode taxonomy, and the application contexts where the stakes are highest. The construction is general: any reference implementing a three-operation contract can be audited by the same battery. Its purpose is singular — to make checkable, before a surrogate is trusted at scale, whether it learned the mechanism or merely the outputs. References Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M. & Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence 2, 665–673 (2020). Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A. & Anandkumar, A. Fourier Neural Operator for Parametric Partial Differential Equations. In Proc. International Conference on Learning Representations (ICLR, 2021); arXiv:2010.08895. Lu, L., Jin, P., Pang, G., Zhang, Z. & Karniadakis, G. E. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence 3, 218–229 (2021). Karniadakis, G. E., Kevrekidis, I. G., Lu, L., Perdikaris, P., Wang, S. & Yang, L. Physics-informed machine learning. Nature Reviews Physics 3, 422–440 (2021). Pearl, J. Causality: Models, Reasoning and Inference, 2nd ed. (Cambridge University Press, 2009). — interventional ("do") semantics underlying the response operator (2). Maslov, S. & Sneppen, K. Specificity and stability in topology of protein networks. Science 296, 910–913 (2002). — degree-preserving rewiring null model used as an invariance probe. Melegh, J. G. Topology, Mean Degree, and Admissibility Degeneracy in the Local Stress–Admissibility Coupling of $k$-State Voter Dynamics on Networks (preliminary study). Preprint, Manuscript ID P02_TOPOLOGY_PHASE_MAP, v0.5 (2026). — source of the worked benchmark and equations (7)–(9). Companion methods paper to P02_TOPOLOGY_PHASE_MAP · Manuscript ID P02_SURROGATE_VALIDATION · v0.1 draft · 5 June 2026.Equations (7)–(9) and the four mechanism signatures (P1–P4) are restated from the preliminary study; all other content is original to this protocol. Tolerances are illustrative and must be fixed per deployment.