ABSTRACT We investigate the clustering properties of 3 < z < 5 candidate passive galaxies from the Merlin et al. (2019) sample residing in the GOODS-North (35 sources) and GOODS-South (33 sources) fields. Within the large uncertainties due to the paucity of sources we do not detect clustering signal in GOODS-North, while this is present in GOODS-South, highlighting the importance of the effects of cosmic variance. The estimated correlation length in GOODS-South is $r_0=12^{+4}_{-5}$ Mpc, while the estimated minimum mass for a halo capable to host one of such high-redshift quenched galaxies is log10(Mmin/M⊙) = 13.0 ± 0.3, once also the constraints from their space density are taken into account. Both values are compatible with the results from GOODS-North. Putting the above findings in a cosmological context, these suggest no evolution of the dark matter content of the hosts of passive galaxies during the past 12.5 Gyr, i.e. during more than 90 per cent of the age of the Universe. We discuss possible scenarios for the observed trend.