Durable agent execution for long-running analytics work.

Course 9: Durable Agent Execution

One-hour objective: design a durable execution layer for an AI analytics agent and explain how state machines, queues, checkpoints, human approval, replay, and cancellation fit together in production.

Course 9 Reading List

Use three required sources: one durable OpenAI-agent recipe, one checkpointing reference, and one containment engineering post for approval-boundary realism. Keep the optional refresher for core durable-execution vocabulary only.

Readiness Checklist

You are ready for the interview version of this topic when you can defend durable execution as a distributed-systems design, not an SDK feature flag.

• You can explain the difference between a best-effort request loop, a background job, and a durable workflow with persisted state transitions.
• You can define the state record for a BigQuery agent: intent, plan, selected datasets, approvals, current step, tool receipts, retry counters, cancellation status, and final answer.
• You can choose checkpoint boundaries that support resume, replay, user-visible status, human approval, and safe incident debugging.
• You can keep side effects outside replay-sensitive logic by using idempotency keys, recorded tool results, BigQuery job IDs, and compensation steps where rollback is impossible.
• You can design human approval as an interrupt and resume point with immutable context, policy checks, timeout behavior, and a clear record of what was approved.
• You can specify cancellation semantics for queued work, in-flight tool calls, long-running BigQuery jobs, downstream notifications, and partially completed analyses.
• You can propose evals and operations checks for stuck workflows, retry storms, approval bypasses, duplicate tool calls, replay correctness, queue backpressure, and trace completeness.

Interview Drill: Agentic AI System Design

Prompt: design durable execution for a BigQuery analytics agent that may spend minutes or hours clarifying a request, inspecting metadata, waiting for approval, running approved SQL, and recovering from model or worker failures.

• Start with requirements: user-visible status, resumability after crash, approval before expensive or sensitive queries, cancellation, auditability, tenant isolation, latency targets, and cost caps.
• Draw the control plane: API request creates a workflow instance, state transitions persist to a store, queue workers run activities, and the UI subscribes to status and approval interrupts.
• Define state transitions: received, clarifying, planning, waiting-for-approval, queued-for-execution, running-query, summarizing, complete, failed, cancelled, and needs-human-follow-up.
• Separate deterministic orchestration from side effects: tool calls, model calls, BigQuery dry runs, BigQuery jobs, notifications, and ticket creation run as activities with timeouts and receipts.
• Handle replay and retries: replay workflow decisions from persisted history, do not repeat completed external actions, retry transient failures with caps, and escalate non-idempotent uncertainty to a human.
• Handle approval and cancellation: freeze the plan shown to the approver, record approval scope, expire stale approvals, propagate cancellation to queued and in-flight work, and mark completed side effects explicitly.
• Close with operations: traces across model and tool steps, queue-depth alerts, stuck-workflow sweeps, replay debugging, duplicate-job detection, approval-bypass tests, and golden long-running analytics scenarios.