In the pursuit of primordial non-Gaussianities, we hope to access smaller scales across larger comoving volumes. At low redshift, the search for primordial non-Gaussianities is hindered by gravitational collapse, to which we often associate a scale $k_{\rm NL}$. Beyond these scales, it will be hard to reconstruct the modes sensitive to the primordial distribution. When forecasting future constraints on the amplitude of primordial non-Gaussianity, $f_{\rm NL}$, off-diagonal components are usually neglected in the covariance because these are small compared to the diagonal. We show that the induced non-Gaussian off-diagonal components in the covariance degrade forecast constraints on primordial non-Gaussianity, even when all modes are well within what is usually considered the linear regime. As a testing ground, we examine the effects of these off-diagonal components on the constraining power of the matter bispectrum on $f_{\rm NL}$ as a function of $k_{\rm max}$ and redshift, confirming our results against N-body simulations out to redshift $z=10$. We then consider these effects on the hydrogen bispectrum as observed from a PUMA-like 21-cm intensity mapping survey at redshifts $2

6+6 pages, 4 figures, code available at https://github.com/tsfloss/pyNG