Layer 6 — AI Hardware · Chapter 3

Memory, Storage, and Data Movement

Book chapterLayer 6

AI systems are frequently limited not by the number of arithmetic units they contain, but by how quickly data can reach those units. A processor that waits for model weights or intermediate results is not computing, regardless of its theoretical speed. Memory and data movement therefore sit at the center of AI hardware design.

The memory hierarchy

Fast memory is expensive and physically close to the processor. Large memory is cheaper but slower and farther away. AI systems use a hierarchy:

  1. Registers and local processor memory: extremely fast storage for values in immediate use.
  2. On-chip cache or scratchpad memory: keeps frequently reused tensors close to compute units.
  3. High-bandwidth memory or external accelerator memory: holds model weights, activations, and working data.
  4. System memory: supports processors, preprocessing, and models that exceed accelerator capacity.
  5. Local and networked storage: contains datasets, checkpoints, model files, logs, and archives.

Capacity and bandwidth are different

Capacity is how much data can be stored. Bandwidth is how much data can be transferred per unit of time. A large model may fit into memory but still run slowly if its weights cannot be delivered fast enough. Conversely, extremely fast memory is of limited use if it cannot hold the working set required by the model.

Capacity pressure

Model weights, optimizer state, activations, long contexts, large batches, and multiple models compete for space.

Bandwidth pressure

Each generated token or training step may require repeated movement of weights and temporary state.

Latency pressure

Interactive applications need the first result quickly, even when total throughput is adequate.

Energy pressure

Moving data across a chip, package, board, or network can consume more energy than the arithmetic operation itself.

What occupies AI memory?

DataPurposeWhy it matters
Model weightsStore learned parametersDetermine minimum memory capacity and recurring transfer demand.
ActivationsRepresent intermediate valuesCan dominate memory during training and large-batch inference.
Gradients and optimizer stateUpdate parameters during trainingMultiply the memory required beyond the model weights alone.
Attention or context cachePreserve prior sequence information during generationGrows with users, sequence length, and model structure.
Temporary buffersSupport kernels, communication, and data transformationReduce usable capacity and affect scheduling.

Techniques for reducing memory pressure

Packaging and proximity

Physical distance matters. Placing memory close to processors can increase bandwidth and reduce energy, but advanced packaging raises manufacturing complexity, thermal density, and cost. Designers increasingly treat the package as a miniature system containing compute dies, memory stacks, interconnects, and specialized I/O.

Storage and the model lifecycle

Training systems also depend on high-throughput storage. Large datasets must be read repeatedly, checkpoints written without pausing entire clusters, and model versions retained for reproducibility. As model fleets grow, storage architecture becomes a governance problem as well as a performance problem: organizations must know which data and model version produced a particular result.

In AI hardware, the expensive operation is often not calculating a number. It is placing the right number in the right location at the right time.