The gravitational coupling of nearby massive bodies to test masses in a gravitational wave (GW) detector cannot be shielded, and gives rise to 'gravity gradient noise' (GGN) in the detector. In this paper we show that for any GW detector using local test masses in the Inner Solar System, the GGN from the motion of the field of $\sim 10^5$ Inner Solar System asteroids presents an irreducible noise floor for the detection of GW that rises exponentially at low frequencies. This severely limits prospects for GW detection using local test masses for frequencies $f_{\text{GW}} \lesssim (\text{few})\times 10^{-7}\,$Hz. At higher frequencies, we find that the asteroid GGN falls rapidly enough that detection may be possible; however, the incompleteness of existing asteroid catalogs with regard to small bodies makes this an open question around $f_{\text{GW}}\sim ��$Hz, and further study is warranted. We show that a detector network placed in the Outer Solar System would not be overwhelmed by this noise above $\sim 10\,$nHz, and make comments on alternative approaches that could overcome the limitations of local test masses for GW detection in the $\sim 10\,$nHz-$��$Hz band.

26 pages, 7 figures. Published version