Gene expression in eukaryotic cells can be turned off by chemical modification of the chromatin fiber on which the DNA sequence of the gene resides. These epigenetic changes to the chromatin can be retained despite DNA replication leading to loss of the modifications, and the epigenetic state can be transmitted to daughter cells. Experiments have probed the longevity of this epigenetic memory and have found that under certain conditions it can persist over many cell divisions. Here we address the question of the origins of this long-term memory using a simple, coarse-grained model of chromatin as a one-dimensional lattice of histones that can be in two states, modified or not. The state of individual histones can be modified by the action of enzymes that read and write chemical modifications and enzymes that remove the modification, as well as DNA replication which on average removes half of the modifications. We show that the time scales over which the epigenetic state of the chromatin decays due to the combined action of these enzymes, can be orders of magnitude longer than the time set by the enzymatic reactions and cell division. We link the emergence of this long-time scale to the presence of a long-lived metastable state in the system. Our results suggest how experiments that tune the enzyme abundance in cells, or the cell division time can affect the decay of epigenetic memory.