Training
Large batches, heavy communication, long-running jobs, high memory use, and sensitivity to system reliability.
Layer 6 — AI Hardware · Chapter 1
AI hardware is the collection of physical components and engineered systems that perform, store, move, sense, and support the computations required by artificial intelligence. At the smallest scale it includes transistors and memory cells. At larger scales it includes processors, circuit boards, racks, networks, power distribution, cooling plants, cameras, microphones, motors, and complete robots.
Traditional computing often emphasizes sequential instructions, business transactions, databases, and general-purpose flexibility. Modern AI relies heavily on repeated operations over large arrays of numbers. Training requires processing vast datasets while adjusting billions of parameters. Inference repeatedly loads model weights, maintains temporary state, and generates outputs under latency and cost constraints.
These workloads create four dominant physical demands:
| Level | Examples | Function |
|---|---|---|
| Semiconductor device | Logic transistors, memory cells, optical components | Represent and manipulate electrical or optical signals. |
| Chip | CPU, GPU, AI accelerator, memory device, network processor | Provide specialized compute, storage, or communication. |
| Package and board | Advanced packages, modules, accelerator boards | Connect chips to memory, power, and high-speed signals. |
| Server and rack | AI server, storage system, network switch | Combine components into serviceable computing systems. |
| Data center | Compute hall, electrical systems, cooling plant, network fabric | Operate large numbers of systems continuously. |
| Edge machine | Phone, vehicle, camera, factory controller, robot | Run AI near the source of data and action. |
Training hardware is optimized for sustained, large-scale computation across many processors. Inference hardware must serve completed models and may prioritize low latency, high utilization, predictable cost, or limited power consumption. Edge and robotics systems add real-time sensing, strict energy budgets, environmental durability, and direct control of physical actions.
Large batches, heavy communication, long-running jobs, high memory use, and sensitivity to system reliability.
Many user requests, variable workloads, batching, model loading, context memory, and cost per output.
Limited power and memory, privacy, intermittent connectivity, and immediate response.
Sensor fusion, control loops, safety, actuators, batteries, and operation in an unpredictable physical world.
AI hardware cannot be evaluated independently of software. A processor may have impressive theoretical throughput but perform poorly if the model cannot use its data types, memory layout, compiler, communication system, or programming tools. Conversely, model designers increasingly adapt architectures to the strengths and limitations of available hardware.
The useful unit is not the chip. It is the complete system: model, compiler, runtime, memory, network, power, cooling, and application.
Layer 6 supplies physical capability. Layer 5 coordinates that capability through scheduling, serving, data systems, and operations. Layer 4 defines the model structures executed on the hardware. Layer 7 uses the full stack to accelerate scientific discovery.