AI Learning Ramp
Course 3 is a one-hour systems session on TTFT, ITL, throughput, prompt caching, and unit economics, built to help you defend batching, routing, and admission-control choices in a frontier AI interview.
Assume your BigQuery-connected GenAI analyst serves three request classes: interactive dashboard Q&A, longer schema-heavy SQL generation, and tool-using agentic investigations. Your job is to keep interactive p95 latency tight while preventing long-running work from wrecking GPU cost and queue health.
One-hour objective: defend a latency-and-cost operating model for mixed interactive and agentic workloads using explicit metrics, queue policy, and prompt-caching strategy.
Write the few numbers you will manage to: p95 TTFT, p95 end-to-end latency, user TPS, and cost per successful request.
Use the seven-principles framework to separate token-speed issues from request-count, prompt-shape, and UX choices.
Focus on TTFT, model choice, output-length control, and streaming so you can explain real versus perceived responsiveness.
Map exact-prefix reuse, retention, and cache-routing behavior to repeated agent workflows and shared system prompts.
Use the optional metrics guide only if you want sharper language for TTFT, ITL, TPS, RPS, and goodput before the drill.
State your routing lanes, admission policy, cache strategy, and one explicit tradeoff you are making to keep cost bounded.
Keep this session tight: exactly three required readings (about 38 minutes total) and one optional refresher.
Official OpenAI guide that frames latency work as seven levers, not just faster inference. It is the cleanest source for thinking about token speed, request count, parallelism, and product-side responsiveness together.
Extract: which levers change TTFT, which change total latency, and which remove work from the model entirely.
Official Anthropic guidance on measuring baseline latency and TTFT, then reducing it through model choice, shorter prompts and outputs, and streaming.
Extract: which changes improve actual generation speed versus which mostly improve perceived responsiveness.
Official mechanics for cache-friendly prompt design, including exact-prefix matching, retention policies, and the latency/cost impact of repeated long prefixes.
Extract: how prompt assembly order changes both p95 latency and input-token cost for repeated agent or analytics workflows.
Use this only if you want a tighter metric glossary before the drill. It cleanly distinguishes TTFT, ITL, TPOT, TPS, RPS, goodput, and p95/p99 behavior.
Extract: which two metrics you will use for user experience and which two you will use for fleet efficiency.
Prompt: "Design the serving policy for a BigQuery-native AI analyst that supports interactive Q&A, scheduled batch enrichment, and agentic investigations without blowing p95 latency or GPU spend."