Layer 5 — AI Infrastructure · Chapter 5
Evaluation, Observability, Security, and Governance
AI systems can run without crashing and still fail their users. They may produce plausible errors, expose private data, follow malicious instructions, call the wrong tool, drift after a model update, or cost far more than expected. Layer 5 therefore needs measurement and control systems that go beyond ordinary uptime monitoring.
Evaluation before deployment
Evaluation asks whether a system is good enough for its intended use. A useful evaluation program combines several forms of evidence:
- Task tests: representative cases with expected outcomes or scoring rules.
- Expert review: assessment by people who understand the domain and consequences.
- Adversarial tests: attempts to trigger unsafe, misleading, or policy-violating behavior.
- Regression tests: ensure that changes do not break previously working behavior.
- Comparative tests: compare models, prompts, retrieval systems, and workflows under the same conditions.
- Operational tests: latency, capacity, failure recovery, and cost under realistic load.
Production observability
Observability helps operators understand what happened inside a real request. A trace may include model version, prompts, retrieved documents, tool calls, intermediate decisions, token use, latency, policy checks, and final outcome. Sensitive content should be minimized, redacted, access-controlled, and retained only as long as necessary.
| Signal | What it reveals |
|---|---|
| Latency and queue time | Whether users are waiting because of model computation, retrieval, tools, or capacity limits. |
| Token and compute usage | Where cost is accumulating and whether prompts or contexts are unnecessarily large. |
| Groundedness and citation quality | Whether claims are supported by retrieved or approved evidence. |
| Tool success and action errors | Whether agents are executing the correct operations and recovering from failure. |
| User corrections and abandonment | Where the system fails to satisfy real needs despite passing laboratory tests. |
| Safety and policy events | Where misuse, prompt injection, leakage, or prohibited content appears. |
AI-specific security risks
- Prompt injection: untrusted text attempts to override instructions or manipulate tool use.
- Data leakage: sensitive context, training examples, logs, or retrieved documents reach an unauthorized party.
- Excessive agency: an agent receives broader permissions than the task requires.
- Model or supply-chain compromise: weights, packages, adapters, datasets, or endpoints are altered or untrusted.
- Extraction and abuse: automated users copy model behavior, evade limits, or consume resources maliciously.
- Indirect attacks: websites, emails, documents, or tool outputs contain instructions designed for the model rather than the human.
Defense in depth
No single classifier or system prompt is enough. Protection may include input isolation, least-privilege credentials, tool allowlists, schema validation, separate approval for high-impact actions, content filtering, sandboxing, rate limits, anomaly detection, and auditable logs.
Governance
Governance assigns responsibility. It defines approved use cases, risk levels, data rules, evaluation standards, release authority, incident procedures, human oversight, vendor requirements, and documentation. Good governance is not paperwork added after development; it is the mechanism that tells infrastructure what policies to enforce.
Reliable AI requires a closed loop: evaluate before release, observe after release, investigate failures, and feed the lessons back into the system.