Several studies investigating the nature of the stellar initial mass function (IMF) of massive early-type galaxies (ETGs) provide strong evidence for a non-universal IMF. Recent results from stellar population synthesis models of ETGs suggest a radially varying IMF, which becomes a salpeter-like in the innermost region. A radially varying IMF, which is also related to the stellar mass-to-light(M/L) ratio will affect the dynamical black hole (BH) mass estimates. Most previous works on the dynamical mass estimates of ETG BHs have assumed a constant M/L in their mass modelling. In this talk I will present the results of an ongoing work where we have performed Schwarzschild orbit superposition modelling using MUSE and SINFONI IFU observations to measure the BH mass of four massive galaxies. In our dynamical modelling we have carefully included the variable IMF obtained from stellar population models to investigate the effect of the variable IMF on the BH mass measurement, in comparison with dynamical models with constant IMF. For NGC 3923 one of the four galaxies in our sample, we obtain a BH mass of log 9.1 Msun, which is about 20 % higher than one obtained with a constant IMF. In this talk I will discuss how and how much a variable IMF can effect the BH mass estimates, and why it is therefore needs to be carefully considered in future mass modelling work.