Stellar-mass black holes (BHs) can be retained in globular clusters (GCs) until the present. Simulations of GC evolution find that the relaxation driven mass-loss rate is elevated if BHs are present, especially near dissolution. We capture this behaviour in a parametrized mass-loss rate, bench marked by results from N-body simulations, and use it to evolve an initial GC mass function (GCMF), similar to that of young massive clusters in the Local Universe, to an age of 12 Gyr. Low-metallicity GCs ([Fe/H] ≲ -1.5) have the highest mass-loss rates, because of their relatively high BH masses, which combined with their more radial orbits and stronger tidal field in the past explains the high turnover mass of the GCMF (~10^5 Msun) at large Galactic radii (>10 kpc). The turnover mass at smaller Galactic radii is similar because of the upper mass truncation of the initial GCMF and the lower mass-loss rate due to the higher metallicities. The density profile in the Galaxy of mass lost from massive GCs (>10^5 Msun) resembles that of nitrogen-rich stars in the halo, confirming that these stars originated from GCs. We conclude that two-body relaxation is the dominant effect in shaping the GCMF from a universal initial GCMF, because including the effect of BHs reduces the need for additional disruption mechanisms.