Recurrent Latent Memory

Atomic capabilities for continual learning

I've been thinking about the most simplistic architecture for continual learning. For the agent capabilities, the atomic ones are read/write strings and grep, and then everything else can be built on top of those. Similarly, I think for continual learning, the atomic capabilities should be read/write latent memories plus the current agent capabilities.

The reasons are first-principled:

  • We humans don't memorize everything verbatim in our memories, because our memory is limited. Instead, we store different levels of "indices". For example, to use a recipe, we may just need to remember where it is and read the content when needed. The information of where the recipe is stored is much smaller than the recipe itself. If we see similar patterns a lot of times, we form intuitions and gain experience, which can be considered as high-level indices. For example, after we have done a lot of math practices, when facing a new math problem, we know which possible techniques to use to solve it.
  • These compressed indices sometimes cannot be expressed in text, so they need to be stored in latent vectors.

Therefore, the way to enable continual learning agents is to:

  • train it to compress information of minimally useful amount into limited latent vectors
  • train it to update such latent memories properly
  • train it to retrieve precise information with the guidance of the abstract indices in its latent memories

Below is the illustration of the memory cycle of a continual learning agent.

Cycle t01 / 16
ToolsReady
Persistent Memory StorageEmpty
Agent Output Windowidle
No output yet
10KNext latent memory
Backbone
System Prompt
Latent memory10K
Working context0 shown · up to 80K
Empty · waiting for user message
<summarize> Tokens10K
Phase 01READY · awaiting input

The system prompt and recurrent latent state are already loaded. The exact working context starts empty.

Comparison with Recurrent and Weight-Updating Architectures

Recurrent latent memory sits between token-level recurrence and inference-time weight adaptation. It updates a bounded but potentially large continuous state without modifying the Backbone on every memory cycle.

Comparison dimensionConventional RNNRecurrent textual/discrete memoryRecurrent latent memoryInference-time weight adaptation
Mutable objectHidden state hth_tText tokens or discrete codes St∈VMS_t \in V^MLatent memory MtM_tModel weights θt\theta_t
State / update scaleddMM bounded token IDs; at most Mlog⁡2∣V∣M \log_2 |V| bitsM×dM \times d (MM can be 10K or even higher)∣θ∣|\theta| (potentially all model weights)
Update clockOnce per tokenAt each turn or consolidation boundaryAt a consolidation boundary or learned triggerDetermined by an online learning schedule
Information directly available to one updateCurrent token xtx_t + previous hidden state ht−1h_{t-1}Previous textual/discrete memory StS_t + up to CC exact tokensPrevious latent memory MtM_t + up to CC exact tokens (CC can be very large, such as 100K)Accumulated experience, feedback, and optimization signals
Learning mechanismA fixed recurrent transition trained offlineAutoregressive rewriting or discrete-code prediction; SFT, discrete estimators, and/or RLA learned memory read, write, and update policy; current agent-training tools such as RL can be usedLearning how to update model weights safely during inference

Compared to conventional RNNs: Because consolidation can jointly use a large previous state and a large exact working context, it is better informed and may produce higher-quality compression than a token-by-token update.

Compared to recurrent textual/discrete memory: Recurrent textual/discrete memory compresses M×dM \times d floating-point values into MM bounded token IDs, giving a raw state channel of at most Mlog⁡2∣V∣M \log_2 |V| bits. Writing those IDs normally requires MM sequential decoding steps; hard discrete choices block direct gradient flow and introduce irreversible quantization errors that can compound across recurrent rewrites. Recurrent latent memory preserves continuous states, writes them in one parallel prefill pass, and supports RNN-style pretraining; long recursive training is still costly, and latent states can also drift.

Compared to the holy-grail inference-time weight adaptation: Updating a bounded latent state can reuse today's agent-training tools, including RL. Reliable inference-time weight updates require a harder and substantially new online-learning paradigm, but the two are fully compatible: latent memory forms a fast learning loop, while weight adaptation forms a slow learning loop.

Extensibility: One Memory Interface, Many Memory Policies

The framework is extensible: update timing, consolidation evidence, writer model, memory budget, and retention horizon can all become learned or scheduled policies.

General update interface
Mt+1=Uϕt(Mt,Et;Kt)M_{t+1}=U_{\phi_t}(M_t,E_t;K_t)

UϕtU_{\phi_t} may be the serving Backbone or a larger specialist writer; EtE_t may include working context, archives, outcomes, and corrections; KtK_t is a fixed or adaptive latent-memory budget.

Policy axisCurrent examplePossible extensions
Update timingWhen the working context reaches its thresholdSemantic boundaries, learned triggers, or scheduled deep reflection
Consolidation evidencePrevious latent memory + current exact working contextArchived episodes, tool outcomes, corrections, task success, and rewards
Writer modelThe serving Backbone both reads and writesA small decoding model with a larger writer, or a hierarchy of updater models
Memory budgetA fixed MM latent-memory tokensBudget-conditioned KtK_t, or a controller/RL policy that selects the slot count
Retention horizonOne recurrent latent stateLearned signal-specific half-lives, optional fast/slow banks, and deep consolidation

Learned memory management

The updater can learn to copy, weaken, overwrite, or erase different latent signals. Explicit gates are therefore an optional inductive bias, not a prerequisite for selective retention and forgetting. Structural mechanisms add value when exact preservation, verifiable deletion, or sparse updates are required.

Heterogeneous readers and writers

Autoregressive decoding is often more memory-bandwidth constrained and can run frequently on a small model, while parallel consolidation can periodically use a larger writer. The writer must emit into the reader's latent input space, so different hidden dimensions require a shared interface or learned bridge.

Adaptive scale and clock

The model can be pretrained across different KtK_t values, while a controller or RL policy selects the required slot count. Because `<summarize>` positions are processed in parallel, KtK_t is normally chosen before consolidation or implemented with a maximum allocation and active mask; extremely large memories also require more efficient attention.

Outcome-aware deep consolidation

Frequent updates can preserve immediate experience, while an infrequent larger model revisits archived episodes, tool outcomes, and task success to perform deeper reflection or “dreaming.” The complete new state should be staged and then atomically, versionably swapped with the state still serving the agent.

Explanation of Recurrent Latent Memory to Agent

This mechanism manages exactly retrievable working context, persistent external records, and recurrent latent state as separate forms of memory. One cycle proceeds through four phases.

1. Interact
The system prompt and latent memory from the previous cycle enter the input first. A user message joins the empty working context as one parallel prefill chunk. The agent then decodes a tool call one token at a time, and the tool result returns as another prefill chunk. Facts that must remain exactly retrievable are written to persistent memory through a storage call before the agent decodes its reply.

2. Reach threshold
When the 80K exact working context reaches capacity, it combines with the 10K latent memory to form the 90K pre-consolidation input. The framework then inserts all 10K <summarize> tokens at once, bringing the illustrative allocation to 100K. The flexible system prompt remains outside that budget.

3. Consolidate
The Backbone processes the complete sequence as one parallel prefill operation. Final-layer hidden states at the 10K <summarize> positions directly become the next 10K latent memory tokens, with no additional projection and no retained KV cache. The old latent memory stays in place until the new state is complete and staged.

4. Turn over
Once the new latent memory is ready, the old working context and <summarize> positions clear, and the new latent memory replaces the old one. The system prompt stays fixed, exact records in persistent memory continue to exist independently, and an empty working context begins the next cycle.

Latent memory carries fuzzy state that can guide later reasoning, persistent memory preserves facts that must be recovered exactly, and the working context holds the explicit interaction for the current cycle.

  1. Recurrent Memory Transformer

    arXiv · 2207.06881

    AbstractRMT adds special memory tokens to an otherwise unchanged Transformer so information can be processed and passed recurrently between sequence segments. Its experiments show competitive language-modeling performance and gains on tasks that require longer-range processing, positioning the architecture as a general mechanism for long-term dependencies and memory-based reasoning.

  2. Adapting Language Models to Compress Contexts

    arXiv · 2305.14788

    AbstractAutoCompressors adapt pretrained language models to compress preceding text segments into compact summary vectors that can be reused as soft prompts. Trained with an unsupervised language-modeling objective, they extend usable context, substitute compressed vectors for demonstrations, and reduce inference cost in long-context, retrieval, and reranking settings.

  3. Learning to Compress Prompts with Gist Tokens

    arXiv · 2304.08467

    AbstractGisting trains language models to compress reusable prompts into a small set of gist tokens by changing the attention mask. On LLaMA and FLAN-T5, the method achieves substantial prompt compression and compute and storage savings with limited loss in output quality.

  4. In-context Autoencoder for Context Compression in a Large Language Model

    arXiv · 2307.06945

    AbstractICAE uses a lightweight learned module around a language model to turn long context into compact memory slots, trained through autoencoding and language-modeling objectives before instruction tuning. It reports fourfold compression with about one percent additional parameters, lowering inference latency and GPU memory cost while preserving useful context representations.

  5. A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts

    arXiv · 2402.09727

    AbstractReadAgent divides long documents into episodes, compresses them into textual gist memories, and decides when to look up original passages for details. Across three long-document question-answering tasks, this human-inspired read, remember, and retrieve loop extends effective context and outperforms retrieval and direct long-context baselines.

  6. MEMORYLLM: Towards Self-Updatable Large Language Models

    arXiv · 2402.04624

    AbstractMEMORYLLM couples a Transformer with a fixed-size latent memory pool that can absorb new textual knowledge after deployment. It retains previously injected information, performs well on model-editing and long-context evaluations, and remains operationally stable across nearly one million memory updates.

  7. G-MemLLM: Gated Latent Memory Augmentation for Long-Context Reasoning in Large Language Models

    arXiv · 2602.00015

    AbstractG-MemLLM combines a frozen language model with a trainable latent memory bank updated through GRU-style gates. Selective preserve, update, and overwrite operations aim to reduce context rot and knowledge dilution, improving multi-hop reasoning and relational precision across model scales.

  8. R³Mem: Bridging Memory Retention and Retrieval via Reversible Compression

    ACL Anthology · 2025.findings-acl.235

    AbstractR³Mem compresses long histories into virtual memory tokens at multiple semantic granularities and uses a reversible architecture to reconstruct source content. Its bidirectional, cycle-consistent training targets both retention and retrieval, with strong results in long-context modeling, retrieval-augmented generation, and conversational agents.

  9. MemGen: Weaving Generative Latent Memory for Self-Evolving Agents

    arXiv · 2509.24704

    AbstractMemGen uses a memory trigger to decide when memory is needed and a memory weaver to generate machine-native latent token sequences from the current reasoning state. It interleaves latent recall with reasoning and reports emergent planning, procedural, and working-memory behaviors without explicit supervision.

  10. FlashMem: Distilling Intrinsic Latent Memory via Computation Reuse

    ACL Anthology · 2026.findings-acl.230

    AbstractFlashMem distills memory directly from transient reasoning states by reusing the backbone's internal computation instead of introducing a separate encoder. A shared-KV consolidator synthesizes memory from cached states, while an attention-entropy monitor triggers consolidation adaptively; the paper reports comparable performance with substantially lower latency.

  11. LiveMem: Maintaining Memory State Continuity in Long-Running LLM Inference

    arXiv · 2608.02515

    AbstractLiveMem formulates long-running inference as maintaining state continuity while bounded working context turns over. It equips a pretrained full-attention language model with a fixed-capacity recurrent memory state plus post-training and serving support that keep the state load-bearing after supporting tokens leave the active context.

  12. MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents

    arXiv · 2506.15841

    AbstractMEM1 trains agents with reinforcement learning to maintain a compact shared state while reasoning across long multi-turn tasks. At each turn it merges previous memory with new observations and discards irrelevant or redundant detail, enabling constant-memory operation with improved efficiency and performance over longer horizons.

  13. Titans: Learning to Memorize at Test Time

    arXiv · 2501.00663

    AbstractTitans introduces a neural long-term memory module that learns from historical context at test time while attention handles precise short-term dependencies. Three architectures combine these components, with reported gains over Transformers and modern recurrent models and scaling to contexts beyond two million tokens.