Inspect: the UK AI Security Institute's framework for LLM and agent evals

Inspect is the evaluation framework built by the UK AI Security Institute, the government body that tests frontier models. It is free, MIT licensed, and written in Python. An eval is a task: a dataset of samples, a solver that produces answers (one model call or a whole agent with tools), and a scorer that grades them. Agents run their tools inside Docker or Kubernetes sandboxes, every step lands in a log you can replay in a browser, and more than 200 published benchmarks are ready to run against any model.

Inspect started at the UK AI Safety Institute, which the government renamed the UK AI Security Institute on 14 February 2025. Its documentation now credits the institute and Meridian Labs as developers, and Meridian Labs maintains companion packages such as inspect_swe. The repository lives at UKGovernmentBEIS/inspect_ai; releases are frequent (inspect-ai 0.3.279 shipped on 13 October 2026), and it needs Python 3.10 or newer. There is nothing to buy.

The building blocks:

• Tasks tie everything together and carry limits, epochs, sandboxes and model roles.
• Datasets are lists of Sample objects (input, target, optional choices, metadata, files to copy into a sandbox), loaded from code, CSV, JSON or Hugging Face.
• Solvers produce the answer: system_message(), generate(), chain_of_thought(), multiple_choice(), or an agent such as the built-in react().
• Scorers grade it: match(), includes(), pattern(), exact(), f1(), choice(), model-graded model_graded_qa() and model_graded_fact(), or your own @scorer with metrics like accuracy() and stderr().
• Tools and agents. bash(), python(), text editing, web search, web browsing and computer use; MCP tools; react() and a deep agent with subagents; an agent bridge that runs agents written for the OpenAI, Anthropic or Google APIs, including external CLIs such as Claude Code and Codex CLI.
• Sandboxes. docker and local are built in; Kubernetes, Modal, Daytona, EC2, Proxmox and others come as extension packages.
• Logs and the viewer. Every run writes a log with every message, tool call, score and timing; inspect view opens them in a browser and a VS Code extension adds authoring and debugging.

In the stack, Inspect is an offline evals tool, like promptfoo and Braintrust's experiments, with a heavier lean toward research-grade agent evaluation. It runs on your machine or cluster and talks to models through their APIs or through local engines such as vLLM.

Pick Inspect when:

• You evaluate agents, not just prompts. Tool-using agents need a safe place to run commands. Inspect gives every sample its own sandbox, with no network by default, and records each step.
• You need statistics you can defend. Epochs with reducers, standard errors, multiple scorers and model-graded panels with majority voting are built in.
• You compare models on public benchmarks. inspect_evals packages over 200 evaluations (GPQA, SWE-bench, ARC and many more) that run with one command against any provider.
• You want logs you can audit. Reviewers can open a transcript and see exactly what the model saw, called and answered, then re-score the same logs with a different scorer.
• You test a third-party agent. The agent bridge and inspect_swe run Claude Code or Codex CLI inside sandboxes and score the result like any other solver.

Pick something else when:

• The team wants config files, not Python. promptfoo's YAML is easier for people who will not write code.
• You need hosted dashboards, review queues and production scoring. Braintrust or Langfuse cover that; Inspect's logs are files.
• You want prompts optimized, not measured. DSPy searches for better prompts; Inspect tells you whether they are better.

A run takes a task, a model and options, and works through every sample:

1. Set up. For agent tasks, Inspect starts a fresh sandbox per sample and copies in the sample's files. With no Dockerfile or compose.yaml in the task directory, the Docker sandbox uses the standard aisiuk/inspect-tool-support image with network_mode: none.
2. Solve. Solvers run in order and share a TaskState. An agent such as react() loops: the model reasons, calls tools in the sandbox, reads results, and calls submit() when it has an answer.
3. Limit. Per-sample limits stop runaway work: message_limit, token_limit, turn_limit, time_limit, working_limit (time minus rate-limit and queue waits) and cost_limit.
4. Score. Each scorer grades the final state against the target. Model-graded scorers call the model bound to the grader role, or the model under test if none is bound.
5. Aggregate and log. Metrics are computed across samples and epochs, and everything goes to a log file in ./logs (or INSPECT_LOG_DIR), in the compact .eval format by default.

Models are addressed as provider/model: openai/..., anthropic/..., google/..., hf/... for local Hugging Face models, vllm/... and more than 20 providers in all. mockllm/model returns canned output, which is how you test an eval's plumbing without spending tokens.