Capacity pressure
Model weights, optimizer state, activations, long contexts, large batches, and multiple models compete for space.
Layer 6 — AI Hardware · Chapter 3
AI systems are frequently limited not by the number of arithmetic units they contain, but by how quickly data can reach those units. A processor that waits for model weights or intermediate results is not computing, regardless of its theoretical speed. Memory and data movement therefore sit at the center of AI hardware design.
Fast memory is expensive and physically close to the processor. Large memory is cheaper but slower and farther away. AI systems use a hierarchy:
Capacity is how much data can be stored. Bandwidth is how much data can be transferred per unit of time. A large model may fit into memory but still run slowly if its weights cannot be delivered fast enough. Conversely, extremely fast memory is of limited use if it cannot hold the working set required by the model.
Model weights, optimizer state, activations, long contexts, large batches, and multiple models compete for space.
Each generated token or training step may require repeated movement of weights and temporary state.
Interactive applications need the first result quickly, even when total throughput is adequate.
Moving data across a chip, package, board, or network can consume more energy than the arithmetic operation itself.
| Data | Purpose | Why it matters |
|---|---|---|
| Model weights | Store learned parameters | Determine minimum memory capacity and recurring transfer demand. |
| Activations | Represent intermediate values | Can dominate memory during training and large-batch inference. |
| Gradients and optimizer state | Update parameters during training | Multiply the memory required beyond the model weights alone. |
| Attention or context cache | Preserve prior sequence information during generation | Grows with users, sequence length, and model structure. |
| Temporary buffers | Support kernels, communication, and data transformation | Reduce usable capacity and affect scheduling. |
Physical distance matters. Placing memory close to processors can increase bandwidth and reduce energy, but advanced packaging raises manufacturing complexity, thermal density, and cost. Designers increasingly treat the package as a miniature system containing compute dies, memory stacks, interconnects, and specialized I/O.
Training systems also depend on high-throughput storage. Large datasets must be read repeatedly, checkpoints written without pausing entire clusters, and model versions retained for reproducibility. As model fleets grow, storage architecture becomes a governance problem as well as a performance problem: organizations must know which data and model version produced a particular result.
In AI hardware, the expensive operation is often not calculating a number. It is placing the right number in the right location at the right time.