Experiments at the interface of quantum-optics and chemistry have revealed that strong coupling between light and matter can substantially modify chemical and physical properties of molecules and solids. While the theoretical description of such situations is usually based on non-relativistic quantum electrodynamics, which contains quadratic light-matter coupling terms, it is commonplace to disregard these terms and restrict to purely bilinear couplings. In this work we clarify the physical origin and the substantial impact of the most common quadratic terms, the diamagnetic and self-polarization terms, and highlight why neglecting them can lead to rather unphysical results. Specifically we demonstrate its relevance by showing that neglecting it leads to the loss of gauge invariance, basis-set dependence, disintegration (loss of bound states) of any system in the basis set-limit, unphysical radiation of the ground state and an artificial dependence on the static dipole. Besides providing important guidance for modeling strongly coupled light-matter systems, the presented results do also indicate under which conditions those effects might become accessible.