Layer 5 — AI Infrastructure · Chapter 4

Serving, Orchestration, and AI Platforms

Book chapterLayer 5

Serving infrastructure receives requests and turns them into model computation. Orchestration infrastructure coordinates the surrounding steps: retrieval, tools, memory, validation, queues, human approval, and follow-up actions. Together, these systems form the runtime of an AI application.

The path of a production request

  1. Authenticate: identify the user, application, organization, and permissions.
  2. Prepare input: validate, moderate, transform, and enrich the request.
  3. Route: choose a model, region, provider, or specialized endpoint.
  4. Assemble context: add instructions, retrieved evidence, memory, and tool definitions.
  5. Schedule inference: place the work on suitable compute, often batching compatible requests.
  6. Generate: stream or return the model’s output.
  7. Verify and act: check the output, call tools, request approval, or trigger another model step.
  8. Record: log operational and quality data under appropriate privacy rules.

Model servers

A model server loads model weights into memory, accepts requests, manages queues, runs computation, and returns results. Its design affects latency, throughput, utilization, and cost. Important techniques include dynamic batching, quantization, parallel execution, prompt caching, key-value cache management, speculative generation, and autoscaling.

Gateways and routers

A model gateway presents one controlled interface to multiple models or providers. It can enforce authentication, rate limits, budgets, logging, fallback policies, and consistent request formats. A router can choose among models according to task difficulty, modality, privacy, latency, cost, or regional requirements.

Routing policyExample decisionTrade-off
Quality-firstSend difficult requests to the strongest available model.Higher cost and possibly higher latency
Cost-firstUse a smaller model unless confidence is low.Requires reliable escalation and quality measurement
Privacy-firstKeep sensitive work inside a private environment.May limit model choice or increase operational burden
Latency-firstChoose a nearby or already-warm endpoint.May sacrifice some quality
Specialization-firstRoute medical images, code, or speech to specialized systems.More models and policies to maintain

Orchestration and agent runtimes

An agent runtime manages multi-step execution. It may store state, call tools, enforce permissions, retry failed actions, pause for approval, and resume later. It must distinguish between model-generated suggestions and authorized actions. This makes orchestration partly a software-engineering problem and partly a control and governance problem.

Queues, workflows, and event systems

Not all AI work should occur inside a live web request. Long document analysis, video processing, large batches, and agent tasks may run asynchronously through queues and durable workflows. These systems preserve state, handle retries, prevent duplicate actions, and make long-running work observable.

Managed platforms and composable stacks

Organizations can buy an integrated platform or assemble specialized components. Integrated platforms reduce setup and operational complexity. Composable stacks offer flexibility, provider independence, and the ability to optimize individual parts. The right choice depends on scale, expertise, regulation, and strategic importance.

A strong AI platform hides unnecessary complexity without hiding accountability.