Observability for agentic query systems.

Course 11: Observability

One-hour objective: design the telemetry and debugging workflow for a production AI query engine, then explain how traces, receipts, metrics, and replay shorten incident response.

Course 11 Reading List

Use exactly three required readings: one open telemetry standard, one OpenAI agent tracing source, and one production observability workflow source.

Readiness Checklist

You are ready for the interview version of this topic when you can defend observability as a product reliability system, not just log collection.

• You can define the core spans in a BigQuery copilot run: user request, retrieval, planning, model generation, SQL draft, dry run, approval, execution, summary, and policy checks.
• You can describe a tool receipt schema with tool name, arguments, caller span, auth scope, BigQuery job ID, result summary, error class, latency, cost, and redaction status.
• You can separate high-cardinality debug attributes from low-cardinality metrics that should drive dashboards and alerts.
• You can choose latency and cost metrics for interactive query assistance: time to first token, end-to-end latency, tool latency, queue time, token counts, cache hits, and BigQuery bytes processed.
• You can explain how to capture state diffs between agent steps without logging raw sensitive data unnecessarily.
• You can design a failure replay package with prompt hash, model version, retrieval IDs, schema snapshot, tool receipts, policy decisions, sampled artifacts, and expected access controls.
• You can connect observability back to evals by turning production incidents, user corrections, and high-risk traces into new regression cases.

Interview Drill: AI Infra System Design

Prompt: design observability for a BigQuery GenAI query engine that uses a planner, retrieval, SQL generation, dry runs, approval gates, execution tools, and final-answer summarization.

• Start with requirements: debug wrong-table answers, SQL dry-run failures, tool loops, policy denials, high latency, token and BigQuery cost spikes, and user-visible unsupported claims.
• Define trace boundaries: one trace per user task, spans for planner decisions, retrieval, model calls, tool calls, approvals, retries, execution, summarization, and user feedback.
• Define telemetry fields: model ID, prompt hash, schema snapshot ID, retrieval corpus version, table IDs, BigQuery job IDs, token counts, bytes processed, tool status, error class, and redaction policy.
• Pick metrics and SLOs: p95 end-to-end latency, time to first token, tool error rate, loop count, dry-run failure rate, approval rate, cost per successful task, and correction rate.
• Design storage and privacy: raw traces with access control and retention, redacted trace views for broad debugging, low-cardinality metrics for dashboards, and sampled artifacts for replay.
• Explain failure replay: reconstruct the prompt context, retrieval candidates, schema state, model version, tool receipts, and policy decisions in a sandbox before creating a regression eval.
• Close with operations: on-call dashboards, alert routing, incident runbooks, canary comparison traces, eval backfills, customer-safe exports, and named owners for telemetry schema changes.