Error Budgets for AI Features: A Concrete Policy

A 99%-accurate LLM feature can still burn the error budget in a week. Your SLO will not catch it.

Classical SLOs assume binary output. The request either succeeded or it did not. An LLM feature produces a probability surface: a confidence distribution, a hallucination rate, a calibration curve, a per-token cost. A "success" today might be a hallucination tomorrow. A 99.5% availability target, applied naively, hides every drift and confidence collapse that actually matters to the user.

This article translates the standard SLO and error-budget vocabulary into a workable budget policy for an LLM-backed feature. It covers four budgets a TPM should track, the burn-rate thresholds that justify a kill-switch, and a worked example with a customer-support agent that almost shipped a hallucination budget burn in week two.

Why classical SLOs fail for LLM features

Three structural reasons.

Output is not binary. A request that returns a valid response can still be wrong, uncalibrated, or unsafe. A 200 OK response tells you nothing about whether the user got what they needed.

Confidence and correctness decouple over time. The model does not change; the world does. The same prompt that returned a confident, accurate answer in March can return a confident, hallucinated answer in August when the underlying data has shifted. A traditional SLO that measures "requests served" does not see this drift.

Cost is part of the reliability story. A model that handles every request correctly but costs ten cents per call is not a reliable feature; it is a budget bomb. SRE error budgets traditionally only consider availability and latency. For AI features, cost must be in the same conversation.

The implication: a TPM running an AI feature launch needs a multi-dimensional budget policy. Four budgets. Not one SLO. Not three.

The four budgets

Confidence budget

Track the fraction of model responses where the model's stated confidence exceeds a calibrated threshold. The Anthropic Claude Sonnet 4.5 system card and OpenAI evals documentation both surface calibration as a first-class metric.

A practical policy: confidence below 0.7 should trigger an automatic "low confidence" response path. The user gets a hedged answer, an offer to escalate to a human, or a refusal to answer. Above 0.7, the response goes through. The budget is the fraction of allowed high-confidence responses that turn out to be wrong.

Drift budget

Track the rate at which the model's output distribution shifts away from a baseline captured at launch. NIST AI RMF 1.0 calls this "performance drift" and lists it as a tracked risk control.

A practical policy: weekly drift score (KL divergence, PSI, or embedding-distance mean) should stay below a configured threshold. Burn a budget point for every week the score exceeds the threshold.

Hallucination budget

Track the rate of factually wrong responses in a sampled evaluation set. The Google SRE Workbook approach applies cleanly here: a sampled review of 100 responses per week, with a hard error budget for the fraction that are factually wrong.

A practical policy: hallucination rate below 2% for customer-facing flows, below 0.5% for compliance-sensitive flows (legal, medical, financial). Burn rate = (current rate - target rate) / target rate, applied per week.

Cost budget

Track the per-request cost against a per-feature cap. This is the budget the CFO cares about.

A practical policy: per-request cost should stay below a configured ceiling. A 50% overrun in any 24-hour window burns one budget point. A 200% overrun burns the entire budget and triggers a kill-switch.

Burn-rate thresholds and a kill-switch protocol

The Google SRE Workbook defines burn-rate thresholds as the rate at which the budget is consumed relative to the calendar. For a 30-day budget, a burn rate of 1.0 means you exhaust the budget exactly on day 30.

For AI features, three thresholds:

| Burn rate | Action | |---|---| | Below 1.0 | Normal operation. Log drift and cost, weekly review. | | 1.0 to 3.0 | Yellow alert. Notify the on-call TPM. Increase eval sample rate to 200/week. | | Above 3.0 | Orange alert. Notify engineering leadership. Pause non-critical new rollouts. | | Above 10.0 | Red alert. Kill-switch. Route to fallback flow (cached responses, human escalation, or hard refusal). |

The kill-switch is the load-bearing piece. A budget policy without an executable kill-switch is a postmortem waiting to happen. The kill-switch should be:

1. Documented in the launch PRD. Not a wiki page; the PRD that the engineering manager signed off on.
2. Owned by a named on-call. A specific engineer or TPM, not "the team."
3. Triggerable in under 15 minutes. From detection to traffic routed to the fallback path.
4. Reversible without a deploy. A config flag, not a code change.

Worked example: a customer-support agent

Consider a customer-support agent that handles 40,000 tickets per week. The launch PRD commits to:

• 95% availability (the standard SLO)
• 2% hallucination budget (customer-facing, factual answers matter)
• $0.08 per-request cost ceiling
• Drift score under 0.15 (weekly PSI)

Week one. Burn rates are normal. Confidence holds above 0.85 average. Hallucination rate sits at 1.2%. Cost per request averages $0.06.

Week two. Something changes. The support team has been A/B testing a new knowledge base format, and the retrieval layer now returns documents in a different chunking scheme. The model's confidence is still high, but the answers start referencing internal documents the customer cannot see. Hallucination rate on the sampled review jumps to 3.4%. Confidence budget burn rate: 1.7 (yellow alert). Cost is fine; cost budget is not burning.

Week two, day three. The on-call TPM gets the yellow alert, increases eval sample rate, and the manual review confirms the issue: 3.4% of responses are now factually wrong in a way that exposes internal documentation. Burn rate: 1.7. Action: notify engineering leadership, pause the A/B test, roll back the chunking change.

Week two, day four. Rollback completes. Hallucination rate returns to 1.1%. The hallucination budget has burned 0.8 of its 1.0 monthly allowance in week two. The remaining 0.2 budget is enough for the rest of the month, with weekly monitoring.

What the kill-switch would have done. If the on-call had been unavailable or if the drift had been faster than the rollback window, the kill-switch would have routed all traffic to the previous model checkpoint, with a customer-visible "we are updating our answers" message. The 40,000 tickets per week would have continued to flow, just with the older, less-helpful model, until the rollback completed.

What to do when the budget burns in week two

The first time the budget burns early, do three things.

Write the postmortem the same week. Not at the end of the quarter. The postmortem should be a one-page artifact with: what burned, what detected it, how long detection took, what the response was, and what changes to the budget policy the burn-rate suggested.

Recalibrate, do not just reset. A budget that burns in week two every month is a budget that is sized wrong. Either the budget is too small (raise the target) or the feature is too unreliable (lower the launch surface). A reset without recalibration is a delayed failure.

Surface the budget burn to leadership in the launch PRD. The PRD should have a "budget burn trigger" section that names what happens when the budget burns early: a feature flag rollback, a customer communication, an executive escalation. The TPM's job is to write this section before launch, not after.

The four-budget template

A TPM can adopt the four-budget policy by adding four lines to the launch PRD's reliability section:

- Confidence budget:  confidence floor;  weekly sample.
- Drift budget:  score ceiling;  weekly eval.
- Hallucination budget:  target;  weekly eval.
- Cost budget:  per-request;  burn alert.

Plus a kill-switch row:

- Kill-switch: owner=; trigger=; reversal-time=.

What this does not solve

This policy does not solve alignment, adversarial prompting, or systemic bias. Those need separate risk controls (NIST AI RMF 1.0 has a wider governance layer that covers them). What the four-budget policy does is give the TPM a measurement framework for the questions that come up in week two, week four, and week twelve. Without it, you find out about reliability problems in the customer escalation queue.

Adoption checklist

• [ ] Write the four budgets into the launch PRD before the launch review.
• [ ] Name the on-call TPM and the kill-switch trigger.
• [ ] Configure the weekly drift eval and the confidence-floor routing.
• [ ] Stamp the kill-switch reversal-time target into the on-call runbook.
• [ ] Schedule the first budget-burn postmortem review at the end of month one.

Adopt the four-budget template. The budget that burns in week two is the budget you should have written down in week zero.

---

Sources: Google SRE Workbook (error budget policy template, burn-rate thresholds); NIST AI RMF 1.0 (risk controls, drift metrics); Anthropic Claude Sonnet 4.5 system card (calibration and confidence reporting); OpenAI evals documentation (hallucination measurement patterns).