Layer 6 — AI Hardware · Chapter 1

What Is AI Hardware?

Book chapterLayer 6

AI hardware is the collection of physical components and engineered systems that perform, store, move, sense, and support the computations required by artificial intelligence. At the smallest scale it includes transistors and memory cells. At larger scales it includes processors, circuit boards, racks, networks, power distribution, cooling plants, cameras, microphones, motors, and complete robots.

Why AI creates unusual hardware demands

Traditional computing often emphasizes sequential instructions, business transactions, databases, and general-purpose flexibility. Modern AI relies heavily on repeated operations over large arrays of numbers. Training requires processing vast datasets while adjusting billions of parameters. Inference repeatedly loads model weights, maintains temporary state, and generates outputs under latency and cost constraints.

These workloads create four dominant physical demands:

  1. Computation: perform matrix, vector, and tensor operations at very high throughput.
  2. Memory: hold model weights, activations, optimizer state, context, and intermediate results.
  3. Communication: move data between processor cores, chips, servers, and data centers.
  4. Energy and heat management: deliver electrical power and remove heat without damaging equipment or making the economics impractical.

The AI hardware stack

LevelExamplesFunction
Semiconductor deviceLogic transistors, memory cells, optical componentsRepresent and manipulate electrical or optical signals.
ChipCPU, GPU, AI accelerator, memory device, network processorProvide specialized compute, storage, or communication.
Package and boardAdvanced packages, modules, accelerator boardsConnect chips to memory, power, and high-speed signals.
Server and rackAI server, storage system, network switchCombine components into serviceable computing systems.
Data centerCompute hall, electrical systems, cooling plant, network fabricOperate large numbers of systems continuously.
Edge machinePhone, vehicle, camera, factory controller, robotRun AI near the source of data and action.

Training, inference, and embodied computing

Training hardware is optimized for sustained, large-scale computation across many processors. Inference hardware must serve completed models and may prioritize low latency, high utilization, predictable cost, or limited power consumption. Edge and robotics systems add real-time sensing, strict energy budgets, environmental durability, and direct control of physical actions.

Training

Large batches, heavy communication, long-running jobs, high memory use, and sensitivity to system reliability.

Cloud inference

Many user requests, variable workloads, batching, model loading, context memory, and cost per output.

Edge inference

Limited power and memory, privacy, intermittent connectivity, and immediate response.

Robotics

Sensor fusion, control loops, safety, actuators, batteries, and operation in an unpredictable physical world.

Hardware–software co-design

AI hardware cannot be evaluated independently of software. A processor may have impressive theoretical throughput but perform poorly if the model cannot use its data types, memory layout, compiler, communication system, or programming tools. Conversely, model designers increasingly adapt architectures to the strengths and limitations of available hardware.

The useful unit is not the chip. It is the complete system: model, compiler, runtime, memory, network, power, cooling, and application.

Where Layer 6 ends

Layer 6 supplies physical capability. Layer 5 coordinates that capability through scheduling, serving, data systems, and operations. Layer 4 defines the model structures executed on the hardware. Layer 7 uses the full stack to accelerate scientific discovery.