This is the fourth paper in a series of four in which we use space adiabatic methods in order to incorporate backreactions among the homogeneous and between the homogeneous and inhomogeneous degrees of freedom in quantum cosmological perturbation theory. In this paper, we finally consider the gauge invariant scalar (Mukhanov-Sasaki) and tensor (primordial gravitational wave) inhomogeneous perturbations of General Relativity coupled to an inflaton field which arise from a careful constraint analysis of this system up to second order in the perturbations. The simultaneous quantisation of the homogeneous and inhomogeneous degrees of freedom suggests the space adiabatic perturbation theory as an approximation scheme in order to capture the backreaction effects between these two sets of degrees of freedom. We are confronted with all the challenges at once that we found in the simpler models treated in this series of papers. We are able to compute these effects up to second order in the adiabatic parameter and find significant modifications as compared to earlier derivations of the effective quantum dynamics of the homogeneous sector.