Correlations between the energies of a system's spectrum are one of the defining features of quantum chaos. They can be probed using the Spectral Form Factor (SFF). We investigate how each spectral distance contributes in building this two-point correlation function. Specifically, starting from the spectral distribution of $k$-th neighbor level spacing ($k$nLS), we provide analytical expressions for the $k$-th neighbor Spectral Form Factor ($k$nSFF). We do so for the three Gaussian Random Matrix ensembles and the `Poissonian' ensemble of uncorrelated energy levels. We study the properties of the $k$nSFF, namely its minimum value and the time at which this minimum is reached, as well as the energy spacing with the deepest $k$nSFF. This allows us to quantify the contribution of each individual $k$nLS to the SFF ramp, which is a characteristic feature of quantum chaos. In particular, we show how the onset of the ramp, characterized either by the dip or the Thouless time, shifts to shorter times as contributions from longer-range spectral distance are included. Interestingly, the even and odd neighbors contribute quite distinctively, the first being the most important to built the ramp. They respectively yield a resonance or antiresonance in the ramp. All of our analytical results are tested against numerical realizations of random matrices. We complete our analysis and show how the introduced tools help characterize the spectral properties of a physical many-body system by looking at the interacting XXZ Heisenberg model with local on-site disorder that allows transitioning between the chaotic and integrable regimes.

13+6 pages, 10+6 figures. v3: Major revision, Accepted in Phys. Rev. B