AI Observability: Traces, Sessions and Cost
See every model call and agent step: who it served, what it cost, and why it went wrong.
An LLM request returns 200 while the answer is wrong, the bill doubles or the agent loops, so the APM dashboard stays green through all three. This phase makes each model call and tool step a span with tokens, cost and the conversation it belongs to, keeps personal data inside your own collector, and makes the dashboard dollars match the invoice.
What you learn
- The OpenTelemetry GenAI conventions as they stand: model, agent and tool spans, token metrics, content capture levels, and why everything is still at Development status
- Instrumentation choices: the official GenAI instrumentations, OpenLLMetry, OpenInference, LiteLLM callbacks and proxy capture with Helicone, and what each records by default
- Langfuse, Laminar, Arize Phoenix and the Traceloop platform in depth: sessions, users, scores, prompts, datasets, browser replay and self-hosting
- The OpenTelemetry Collector as the spine: redaction with the transform processor, tail sampling that keeps errors, slow and expensive traces, and fan-out to several backends
- Cost attribution from token usage and a versioned rate card, reconciled with the bill, plus SLOs and alerts on time to first token, latency, errors, tokens, cost and quality
Tools you will actually touch
The assignment pack
A shopping assistant handling 60,000 conversations a month had a quarter of invisible incidents: the model bill rose by half with no traffic change, a customer found her phone number in a vendor trace viewer, users called answers slow while APM showed green, and an agent looped eleven tool calls in one turn. Build the telemetry pipeline that turns each of those into a dashboard, an alert or a query.
- Every model call and tool step is a span with tokens, cost, session and user, from one pinned instrumentation
- A pinned collector masks personal data before any backend sees it, proven with a planted probe
- Tail sampling keeps every error, slow, expensive and thumbs-down trace and thins the rest
- Token totals from metrics match the provider bill, and cost is priced from a versioned rate card
- A written capture policy with retention per store, and SLOs with burn-rate and spend alerts
Traces exist but content is captured by default, errors are sampled away, or cost totals come from sampled spans.
Pinned instrumentation, context on every span, masking proven by a probe, a tail policy that keeps every incident trace, and metrics that reconcile with the bill.
Sampling arithmetic against a storage budget, a cost metric computed before the sampler, an unbiased evaluation sample, and a backend swap done as one collector change.
The tutorial pack
Trace the assistant, then leak a phone number on purpose
- 1Write a deterministic model stub that keeps a bill you can read back to the token
- 2Instrument the assistant with one pinned GenAI instrumentation and export OTLP to a pinned collector
- 3Carry session and user ids onto every span through baggage with an allowlist
- 4Plant an email address and a phone number, then prove the collector masks both
- 5Price every call from a rate card and add a tail-sampling policy that keeps every incident
- 6Replay a fixed traffic mix and reconcile the metrics with the bill
Write the redaction as one statement list for span and span event attributes. The collector validates it, logs nothing, and masks nothing, because the list runs once per span event and GenAI spans have none. Then watch an over-broad phone pattern eat a provider response id.
- Error, slow and expensive traces kept against the counts the provider served
- Planted personal data found in any export file (target: zero)
- Metric token totals against billed tokens, per model
- Sum of sampled span costs against the bill, and why it is lower
You must be able to answer
AI use is mandatory on every assignment. The integrity mechanism is defend-your-work, so these are the questions that decide whether the work counts.
- “A customer says the assistant promised her a refund last Tuesday. Walk from her complaint to the exact prompt, model snapshot and tool results, using only what your pipeline keeps.”
- “Your cost dashboard and the invoice disagree by 20 percent. What are the likely causes, in order, and what do you open first?”
- “Which trace would your sampling policy drop that you would want in an incident, and what does keeping it cost?”
Want to be walked through this phase?
The packs are delivered with live teaching, reviewed assignments, and a defence.
Train with me