Layer 6 — AI Hardware · Chapter 4

Networking and Interconnects

Book chapterLayer 6

A large AI model rarely fits comfortably on one processor. Training and increasingly inference distribute work across many devices. Networking turns those devices into a single usable system. When communication cannot keep pace with computation, adding more accelerators produces diminishing returns.

Two communication domains

AI systems use both scale-up and scale-out communication.

Scale-up

Connects processors within a package, server, or tightly integrated rack. It prioritizes very high bandwidth, low latency, and shared operation.

Scale-out

Connects servers and racks across a cluster or data center. It prioritizes reach, routing, resilience, and expansion to very large systems.

The boundary can shift as packaging and rack design evolve, but the underlying problem remains the same: processors must exchange model state, activations, gradients, cache, and control information without spending most of their time waiting.

What AI clusters communicate

Bandwidth, latency, and utilization

Bandwidth determines how much information can move. Latency determines how long a transfer or coordination step takes. AI workloads may require both: large transfers benefit from bandwidth, while repeated synchronization and small messages can be dominated by latency.

Effective utilization depends on overlap. Well-designed systems communicate while other computation continues. Poorly balanced systems pause entire processor groups while waiting for the slowest transfer or worker.

Topology

Topology describes how devices and switches are connected. Different layouts trade cable count, cost, path length, fault tolerance, and ease of expansion. The best topology depends on communication patterns generated by the model and parallelization strategy.

Design concernHardware questionSoftware consequence
Path capacityCan many processors communicate simultaneously without congestion?Determines scaling efficiency and scheduling choices.
LocalityWhich processors have the fastest direct links?Models and tasks should be placed to minimize expensive transfers.
Failure domainsWhat happens when a link, switch, or server fails?Jobs need recovery, rerouting, or checkpointing.
ExpansionCan the cluster grow without redesigning the entire fabric?Influences procurement and long-term utilization.
StandardsAre links open, proprietary, or interoperable?Affects supplier choice and software portability.

Electrical and optical communication

Electrical links are common over short distances and integrate naturally with semiconductor systems. As bandwidth and distance increase, signal integrity and energy become harder problems. Optical links can carry large quantities of data over longer distances with different energy and packaging trade-offs. The long-term architecture may place optical components progressively closer to processors.

Collective communication

Distributed AI frequently performs coordinated operations such as broadcast, gather, reduce, and all-reduce. Specialized libraries and network support can execute these patterns more efficiently than treating each transfer independently. Hardware, topology, and parallelization strategy must be designed together.

Reliability at scale

When a system contains thousands of components, individual failures become routine rather than exceptional. Networks need error detection, rerouting, monitoring, and replacement procedures. Training software needs checkpoints and the ability to resume. Inference services need redundancy so a failed link does not interrupt users.

A cluster is not a pile of fast processors. It is a communication machine whose arithmetic happens to be distributed across many chips.