We study Rayleigh B��nard convection based on the Boussinesq approximation. We are interested in upper bounds on the Nusselt number $\mathrm{Nu}$, the upwards heat transport, in terms of the Rayleigh number $\mathrm{Ra}$, that characterizes the relative strength of the driving mechanism and the Prandtl number $\mathrm{Pr}$, that characterizes the strength of the inertial effects. We show that, up to logarithmic corrections, the upper bound $\mathrm{Nu}\lesssim \mathrm{Ra}^{\frac{1}{3}}$ of Constantin and Doering in 1999 persists as long as $\mathrm{Pr}\gtrsim \mathrm{Ra}^{\frac{1}{3}}$ and then crosses over to $\mathrm{Nu}\lesssim\mathrm{Pr}^{-\frac{1}{2}}\mathrm{Ra}^{\frac{1}{2}}$. This result improves the one of Wang by going beyond the perturbative regime $\mathrm{Pr} \gg \mathrm{Ra}$. The proof uses a new way to estimate the transport nonlinearity in the Navier-Stokes equations capitalizing on the no-slip boundary condition. It relies on a new Calder��n-Zygmund estimate for the non-stationary Stokes equations in $L^1$ with a borderline Muckenhoupt weight.