Layer 4 — Model Architecture · Chapter 5
Memory, Context, and Continual Learning
Intelligence requires more than processing the present input. A useful system must preserve relevant information, recover it later, update what it knows, and distinguish temporary context from durable memory.
Context is working memory
The context window contains the information directly available during a model run: the prompt, recent conversation, retrieved documents, tool results, and generated tokens. A larger context can hold more material, but size alone does not guarantee that the model will use every detail correctly.
Several forms of memory
| Memory type | Purpose | Typical mechanism |
|---|---|---|
| Parametric memory | General patterns learned during training. | Knowledge stored across model parameters. |
| Contextual memory | Information needed for the current task. | Tokens or representations inside the active context. |
| Retrieved memory | Facts and records fetched when needed. | Search, databases, vector retrieval, or knowledge graphs. |
| Episodic memory | Past events, interactions, and completed work. | External logs, summaries, and memory stores. |
| Working state | Progress inside a multi-step process. | Scratchpads, state machines, recurrent state, or agent records. |
Retrieval is part of architecture
Retrieval-augmented systems separate reasoning from storage. Instead of forcing every fact into model parameters, the system searches an external source and places relevant material into context. The architectural challenge is deciding what to retrieve, when to retrieve it, how much to include, and how to judge source quality.
Compression and forgetting
Unlimited memory is neither practical nor desirable. Old information may be irrelevant, contradictory, private, or outdated. Memory systems therefore need summarization, prioritization, expiration, correction, and deletion. Forgetting is an architectural capability, not merely a failure.
Continual learning
A continually learning model updates from new experience after deployment. This sounds natural but is difficult. New learning can erase old capabilities, amplify errors, absorb malicious inputs, or make behavior unstable. Safer designs may update a memory store or a small adapter rather than changing the entire model.
The unresolved problem of identity over time
A persistent assistant or robot must maintain goals, preferences, permissions, relationships, and a history of its own actions. That requires memory structures that are auditable and editable—not merely a long transcript hidden inside a prompt.
A useful memory system must know not only how to remember, but what to trust, what to revise, and what to forget.