Serving, Gateways and Cost
The platform half: run models like production software.
This is where platform engineering and AI meet, and where most AI engineers stop being useful. Routing, budgets, fallbacks, GPU economics, and cost attribution are the difference between a demo and a service your company can afford to run.
What you learn
- Gateways: routing, budgets, fallbacks, per-tenant virtual keys
- Self-hosted serving (vLLM, KServe-class) and when it actually beats hosted APIs
- GPU basics for platform people: VRAM math, KV cache, continuous batching, cold starts
- Autoscaling model workloads without paying for idle accelerators
- Cost observability: per request, per tenant, per feature
- Delivered on AWS, GCP, Azure, on-prem, and cloud-agnostic Kubernetes
Tools you will actually touch
The assignment pack
Three product teams share one AI budget and keep blaming each other for the bill. Stand up a model plane that routes their traffic, enforces per-team budgets, fails over when a provider degrades, and attributes every cent to a team and a feature.
- Gateway in front of all model traffic with per-tenant virtual keys
- Enforced budget that actually stops spend, not just alerts on it
- Automatic fallback proven by degrading the primary provider
- Cost dashboard broken down by tenant and feature
- A documented decision on hosted versus self-hosted with the numbers behind it
A shared API key and a monthly invoice surprise.
Gateway with per-tenant keys, enforced budgets, working fallback, cost dashboard.
Self-hosted path benchmarked against hosted with real throughput and cost figures, and autoscaling that does not pay for idle GPUs.
The tutorial pack
Stand up a model plane
- 1Put a gateway in front of one provider and route all traffic through it
- 2Issue per-tenant virtual keys and set hard budgets
- 3Add a second provider and configure fallback rules
- 4Kill the primary provider and prove failover works
- 5Deploy a self-hosted model and benchmark tokens per second against the hosted path
- 6Export cost and token metrics; build the per-tenant dashboard
Size a model so its KV cache exceeds available VRAM at your target concurrency, then watch it OOM under load. Do the VRAM math afterwards and resize. GPU economics stop being abstract at that point.
- Cost per 1,000 requests, hosted versus self-hosted
- p95 latency under target concurrency
- Budget enforcement: spend stops at the ceiling
- Failover time when the primary provider degrades
Per-target lab variants
Same outcome, five stacks. You train on the one you will actually run.
You must be able to answer
AI use is mandatory on every assignment. The integrity mechanism is defend-your-work, so these are the questions that decide whether the work counts.
- “At what request volume does self-hosting beat the API, and show your math.”
- “A tenant hits their budget mid-request. What does the user see?”
- “How much VRAM does your deployment need at 20 concurrent requests?”
- “Your provider has a partial outage, 30 percent of calls fail. What happens?”
Want to be walked through this phase?
The packs are delivered with live teaching, reviewed assignments, and a defence.
Train with me