Within a genuinely gauge invariant approach recently developed for the computation of the cosmological backreaction, we study, in a cosmological inflationary context and with respect to various observers, the impact of scalar fluctuations on the space-time dynamics in the long wavelength limit. We stress that such a quantum backreaction effect is evaluated in a truly gauge independent way using a set of effective equations which describe the dynamics of the averaged geometry. In particular we show under what conditions the free falling (geodetic) observers do not experience any scalar-induced backreaction in the effective Hubble rate and fluid equation of state.

4 pages. Typos corrected, comment added, version accepted for publication in Phys. Rev. Lett