Production reliability for AI query systems.

Course 12: Production Reliability

One-hour objective: design the reliability envelope around a production AI query engine, then explain how overload controls, provider-limit handling, rollout gates, and incident loops keep agentic behavior bounded.

Course 12 Reading List

Use exactly three required readings: one provider-limit source, one overload-control source, and one rollout-safety source.

Readiness Checklist

You are ready for the interview version of this topic when you can turn reliability from a slogan into concrete controls and tradeoffs.

• You can define admission control before expensive work: tenant quotas, per-user caps, token budgets, BigQuery bytes limits, tool fanout limits, and provider rate-limit headers.
• You can explain when to retry, queue, shed, or fail fast, including retry caps, jitter, timeout budgets, and the risk of multiplying load through nested agent retries.
• You can design graceful degradation paths: cached metadata, narrower retrieval, smaller model, dry-run-only SQL, delayed execution, partial summaries, or human approval for risky actions.
• You can separate interactive analyst tasks from scheduled agent tasks with different criticality, latency SLOs, queue priorities, and cancellation behavior.
• You can name model-drift signals for a query copilot: eval regression, user corrections, SQL dry-run errors, table-selection shifts, policy denials, cost spikes, and unsupported-answer rates.
• You can canary model, prompt, retriever, and tool changes with control comparisons, representative traffic, release gates, rollback triggers, and post-release monitoring.
• You can close the incident loop by turning traces, failed canaries, user escalations, and outage notes into runbooks, regression evals, quota changes, and safer defaults.

Interview Drill: AI Infra System Design

Prompt: design production reliability for a BigQuery GenAI query engine that calls frontier models, retrieval services, policy checks, dry-run validation, and BigQuery execution tools.

• Start with SLOs: availability, p95 latency, time to first token, successful task completion, cost per task, dry-run success, correction rate, and unsupported-answer rate.
• Define demand controls: tenant quotas, provider-limit adapters, per-user caps, token and BigQuery byte budgets, queue classes, concurrency limits, and burst protection.
• Design overload behavior: classify requests by criticality, shed scheduled work first, cap retries, return honest degraded answers, and prevent tool loops with circuit breakers.
• Design provider-failure handling: read rate-limit headers, use jittered backoff, switch to smaller or alternate models only when quality risk is acceptable, and expose retry-after state to callers.
• Design rollout safety: canary model, prompt, retriever, SQL tool, and policy changes with control populations, eval gates, SLO gates, cost gates, safety gates, and automatic rollback.
• Design drift detection: compare live traces against golden tasks, monitor table-selection and dry-run error shifts, sample high-risk domains, and require review for sudden behavioral changes.
• Close with operations: dashboards, alert routes, runbooks, incident commander handoff, customer-safe status language, replay bundles, postmortems, and new regression tests from every serious incident.