The ability to protect quantum information from the effect of noise is one of the major goals of quantum information processing. In this article, we study limitations on the asymptotic stability of quantum information stored in passive $N$-qubit systems. We consider the effect of small imperfections in the implementation of the protecting Hamiltonian in the form of perturbations or weak coupling to a ground state environment. We thus depart from the usual Markovian approximation for a thermal bath by conce ntrating on models for which part of the evolution can be calculated exactly. We prove that, regardless of the protecting Hamiltonian, there exists a perturbed evolution that necessitates a final error correcting step for the state of the memory to be read. Such an error correction step is shown to require a finite error threshold, the lack thereof being exemplified by the 3D XZ-compass model \cite{bacon-2006}. We go on to present explicit weak Hamiltonian perturbations which destroy the logical information stored in the 2D toric code in a time $O(\log(N))$.