AI Learning Ramp
Course 2 is a one-hour systems session on when to use vLLM, TGI, TensorRT-LLM, or SGLang, and how to defend routing, SLO, and capacity choices in a frontier AI interview.
Assume you run a BigQuery-connected GenAI analyst that serves mixed workloads: short dashboard questions, long schema-heavy SQL generation, and occasional agentic multi-step investigations. Your job is to pick the serving stack per request class, keep p95 latency inside SLO, and control GPU cost while preserving reliability.
One-hour objective: defend an engine-selection strategy for vLLM, TGI, TensorRT-LLM, and SGLang under explicit latency, throughput, reliability, and operability constraints.
Write your scorecard: p95 TTFT, tokens/s/GPU, memory efficiency, failover model, and ops complexity.
Review PagedAttention and why memory fragmentation directly affects usable batching and practical throughput.
Use the official Hugging Face engine guidance to place TGI vs newer engine options in platform strategy.
Focus on disaggregated serving and where specialized performance optimization justifies operational complexity.
Skim the optional runtime docs to understand when SGLang should be tested as a candidate in your routing layer.
Deliver a 90-second recommendation with one primary engine, one fallback path, and one risk mitigation plan.
Keep this session tight: exactly three required readings (about 38-42 minutes total) and one optional refresher.
Primary research source for vLLM's core design. It gives interview-safe language for explaining memory fragmentation, throughput limits, and why paging-style KV management matters.
Extract: how KV cache allocation strategy becomes a first-order performance constraint.
Official engine guidance from Hugging Face, including current positioning of TGI and migration context to newer engines such as vLLM and SGLang.
Extract: when TGI is operationally acceptable versus when modern engine alternatives are better long-term choices.
Official NVIDIA technical docs on splitting prefill and decode into separate serving groups. High-signal for advanced latency and utilization tradeoffs at scale.
Extract: when disaggregated prefill/decode architecture improves SLO adherence and fleet efficiency.
Use only if you need a quick refresh on SGLang runtime capabilities before the drill. Treat it as context, not mandatory reading for this hour.
Extract: where SGLang can fit as an alternative serving/runtime layer in a multi-engine strategy.
Prompt: "Design the serving layer for a BigQuery-native AI analyst with strict p95 latency SLOs and variable query complexity." Keep answers explicit about engine routing and failure handling.