Monolithic design
Simple logical integration and fast on-die communication, but large dies can be expensive and yield-sensitive.
Layer 6 — AI Hardware · Chapter 5
AI hardware begins as a design but becomes economically meaningful only when it can be manufactured, packaged, powered, cooled, tested, and operated. These physical systems impose long lead times and capital requirements that are very different from those of ordinary software.
A modern AI chip passes through a network of specialized activities:
No single company needs to perform every step, but every step must work. The chain depends on highly specialized equipment, materials, intellectual property, manufacturing knowledge, and quality control.
Not every manufactured die functions perfectly. Yield is the share of usable chips obtained from a wafer or packaging process. Large, complex dies are more exposed to defects. Chiplets can improve modularity and reuse, but they add packaging and communication complexity. Product economics therefore depend on architecture, die size, process maturity, packaging yield, testing, and the ability to sell partially functional components into lower tiers.
As it becomes harder to improve every function on one monolithic die, packaging becomes a central technology. A package may combine compute chiplets, memory stacks, I/O dies, and high-density interconnects. This can increase bandwidth and design flexibility, but it concentrates heat and creates difficult mechanical, electrical, and manufacturing problems.
Simple logical integration and fast on-die communication, but large dies can be expensive and yield-sensitive.
Allows modular components and process specialization, but requires advanced links, standards, and packaging.
Places high-bandwidth memory close to compute, increasing performance while raising thermal and supply complexity.
May bring high-speed communication closer to the package as electrical links face distance and energy limits.
AI systems require power at multiple levels: utility connection, substations, backup systems, facility distribution, rack-level delivery, voltage conversion, and finally stable power to chips. Losses at each stage become heat. Rapid changes in workload can also create electrical challenges that ordinary facilities were not designed to manage.
Power availability influences where data centers can be built and how quickly they can expand. Hardware efficiency therefore has strategic value: an efficient system can produce more useful computation from the same electrical and facility capacity.
Cooling removes the heat generated by processors, memory, networking, and power conversion. Air cooling is simple and familiar but becomes difficult at high rack densities. Liquid cooling can move more heat and place cooling closer to components, but introduces pumps, plumbing, maintenance, leakage management, and facility redesign.
| Cooling approach | Strength | Challenge |
|---|---|---|
| Air cooling | Mature, serviceable, widely deployed | Limited heat-removal capacity at very high density |
| Direct-to-chip liquid cooling | Efficient transfer from hot components | Requires liquid distribution and compatible server design |
| Immersion cooling | Can cool dense systems directly | Changes service procedures, materials, and facility operations |
| Facility heat reuse | Can recover a portion of waste heat | Depends on location, temperature, and nearby demand |
Advanced AI systems may depend on a small number of suppliers for fabrication, memory, packaging, networking components, manufacturing equipment, or design tools. A bottleneck in any one category can limit the output of the entire system. Companies therefore pursue second sources, long-term contracts, inventory, geographic diversification, and redesigns that tolerate substitute components.
AI hardware capacity is created by a chain. The rarest link—not the most visible chip—often determines how much computing can actually be delivered.