Scale-up
Connects processors within a package, server, or tightly integrated rack. It prioritizes very high bandwidth, low latency, and shared operation.
Layer 6 — AI Hardware · Chapter 4
A large AI model rarely fits comfortably on one processor. Training and increasingly inference distribute work across many devices. Networking turns those devices into a single usable system. When communication cannot keep pace with computation, adding more accelerators produces diminishing returns.
AI systems use both scale-up and scale-out communication.
Connects processors within a package, server, or tightly integrated rack. It prioritizes very high bandwidth, low latency, and shared operation.
Connects servers and racks across a cluster or data center. It prioritizes reach, routing, resilience, and expansion to very large systems.
The boundary can shift as packaging and rack design evolve, but the underlying problem remains the same: processors must exchange model state, activations, gradients, cache, and control information without spending most of their time waiting.
Bandwidth determines how much information can move. Latency determines how long a transfer or coordination step takes. AI workloads may require both: large transfers benefit from bandwidth, while repeated synchronization and small messages can be dominated by latency.
Effective utilization depends on overlap. Well-designed systems communicate while other computation continues. Poorly balanced systems pause entire processor groups while waiting for the slowest transfer or worker.
Topology describes how devices and switches are connected. Different layouts trade cable count, cost, path length, fault tolerance, and ease of expansion. The best topology depends on communication patterns generated by the model and parallelization strategy.
| Design concern | Hardware question | Software consequence |
|---|---|---|
| Path capacity | Can many processors communicate simultaneously without congestion? | Determines scaling efficiency and scheduling choices. |
| Locality | Which processors have the fastest direct links? | Models and tasks should be placed to minimize expensive transfers. |
| Failure domains | What happens when a link, switch, or server fails? | Jobs need recovery, rerouting, or checkpointing. |
| Expansion | Can the cluster grow without redesigning the entire fabric? | Influences procurement and long-term utilization. |
| Standards | Are links open, proprietary, or interoperable? | Affects supplier choice and software portability. |
Electrical links are common over short distances and integrate naturally with semiconductor systems. As bandwidth and distance increase, signal integrity and energy become harder problems. Optical links can carry large quantities of data over longer distances with different energy and packaging trade-offs. The long-term architecture may place optical components progressively closer to processors.
Distributed AI frequently performs coordinated operations such as broadcast, gather, reduce, and all-reduce. Specialized libraries and network support can execute these patterns more efficiently than treating each transfer independently. Hardware, topology, and parallelization strategy must be designed together.
When a system contains thousands of components, individual failures become routine rather than exceptional. Networks need error detection, rerouting, monitoring, and replacement procedures. Training software needs checkpoints and the ability to resume. Inference services need redundancy so a failed link does not interrupt users.
A cluster is not a pile of fast processors. It is a communication machine whose arithmetic happens to be distributed across many chips.