
We present a two-phase sleep architecture for memory consolidation in language models, with slow-wave sleep (SWS) for individual fact consolidation via per-fact LoRA training, and REM sleep for knowledge integration via synthetic multi-fact conversations. We introduce per-fact staged consolidation where each fact independently advances through stages (0-3) based on individual chat recall testing, replacing all-or-nothing per-edit gating. Key findings: MEMIT achieves near-zero perplexity cost for fact injection; REM reduces SWS-induced perplexity damage by 88% at 3B; per-fact gating achieves 95% consolidation success at 8B; and we discover pathway separation where MEMIT edits the raw completion pathway while LoRA edits the chat pathway. We validate across 3B, 8B, and 70B models, demonstrating that the graduated MEMIT dissolution schedule (scale 1.0 -> 0.5 -> 0.1 -> 0.0) successfully transfers knowledge from MEMIT to LoRA.
Part of the Sleeping LLM research series on sleep-wake memory consolidation for lifelong learning in language models.
memory consolidation, language models, two-phase sleep, lifelong learning, MEMIT, LoRA, sleep-wake cycle, continual learning, per-fact gating
memory consolidation, language models, two-phase sleep, lifelong learning, MEMIT, LoRA, sleep-wake cycle, continual learning, per-fact gating
| selected citations These citations are derived from selected sources. This is an alternative to the "Influence" indicator, which also reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | 0 | |
| popularity This indicator reflects the "current" impact/attention (the "hype") of an article in the research community at large, based on the underlying citation network. | Average | |
| influence This indicator reflects the overall/total impact of an article in the research community at large, based on the underlying citation network (diachronically). | Average | |
| impulse This indicator reflects the initial momentum of an article directly after its publication, based on the underlying citation network. | Average |
