Cosmological simulations almost invariably estimate the accretion of gas on to supermassive black holes using a Bondi-Hoyle-like prescription. Doing so ignores the effects of the angular momentum of the gas, which may prevent or significantly delay accreting material falling directly on to the black hole. We outline a black hole accretion rate prescription using a modified Bondi-Hoyle formulation that takes into account the angular momentum of the surrounding gas. Meaningful implementation of this modified Bondi-Hoyle formulation is only possible when the inner vorticity distribution is well resolved, which we achieve through the use of a super-Lagrangian refinement technique around black holes within our simulations. We then investigate the effects on black hole growth by performing simulations of isolated as well as merging disc galaxies using the moving-mesh code AREPO. We find that the gas angular momentum barrier can play an important role in limiting the growth of black holes, leading also to a several Gyr delay between the starburst and the quasar phase in major merger remnants. We stress, however, that the magnitude of this effect is highly sensitive to the thermodynamical state of the accreting gas and to the nature of the black hole feedback present.