Agent Operations: See, Control, and Optimize Agents

What you learn

• Trajectories as traces: OpenTelemetry GenAI agent and tool spans, Laminar session replay, LangSmith Studio, Langfuse and Phoenix agent views, and Claude Code telemetry as a worked example
• Runtime control outside the model: allowlists and permission order, hooks as policy, approvals that park instead of block, budgets, kill switches, circuit breakers and sandboxes
• Typed decisions with the TypeSafe Jev model: the System One API, noul, choice and score questions, probabilities and confidence, the SDKs, and the vendor list of weaknesses
• Confidence-gated routing, shadow mode beside an existing classifier, fail-open inside one timeout, and calibration with reliability bins, ECE and threshold tuning
• Optimizing agents with trajectory evals, cost per task, prompt caching, model routing, context pruning and parallel tool calls, plus incident response from detection to postmortem

Tools you will actually touch

The assignment pack

• One OpenTelemetry trace per run with GenAI-convention spans, and tokens and cost on the root that reconcile with the children
• Five budgets checked before the spend, with the exhausted dimension on the trace and a handoff that keeps the findings
• Refunds park in an approval queue with their arguments and pay exactly once when a human approves; non-allowlisted tools are denied
• An out-of-band kill switch that halts a running agent within one step
• A router that runs the decision model in shadow beside the rules, gates live mode on a measured threshold, and fails open inside one timeout
• A postmortem whose trace id, dimension and cost a reviewer can find in the spans

The tutorial pack

Trace an agent, stop it, and route beside it

1. 1Export one trace per run with invoke_agent, chat and execute_tool spans, and read it with the five trajectory questions
2. 2Watch a planner loop with the ceilings pushed back, then stop it with a ledger checked before the spend
3. 3Deny a tool that is not allowlisted, park a refund in an approval queue, and approve it twice to prove it pays once
4. 4Pull the kill switch before a run and in the middle of one
5. 5Run a System One decision stub in shadow beside the rules classifier and label the disagreements
6. 6Calibrate from the shadow log, set the measured threshold, and compare it with the vendor example floor

• Cost of a normal ticket against a runaway at its ceiling and with the ceilings removed
• Refund lines after two approvals of one parked call, and tool calls before a mid-run kill took effect
• Shadow agreement, expected calibration error, the measured threshold, its coverage, and live accuracy against rules alone

You must be able to answer

AI use is mandatory on every assignment. The integrity mechanism is defend-your-work, so these are the questions that decide whether the work counts.

• “Here is a run that cost forty times the median. Using only its spans, where did the money go and where did it first diverge from a good run?”
• “Your decision service starts answering in four seconds. What does every in-flight ticket experience, and which number on which dashboard moves first?”
• “Defend your router threshold: where did the number come from, what coverage and accuracy does it buy, and what would make you change it?”