Braintrust: datasets, experiments, scorers and production traces in one loop

Braintrust is where many teams keep the evidence that their AI feature works. You define a dataset, a task and scorers in code; every run becomes an experiment you can compare row by row with the last one; production traces flow into the same projects and get scored automatically; and the bad traces become new dataset rows. It is a hosted product with open-source SDKs and a free tier.

Braintrust covers the loop between building and running an AI system:

• Datasets. Versioned sets of input, expected and metadata rows, created in the UI, by SDK, or from production logs and human review.
• Experiments. Eval() runs a task over a dataset and scores every output. Each run is stored, so you compare a new prompt or model against a baseline and see which rows improved and which regressed.
• Scorers. The open-source autoevals library (Factuality, ClosedQA, Levenshtein, JSON and SQL checks, RAGAS-style RAG metrics and more), your own code scorers, and LLM-as-a-judge scorers built in the UI. Classifiers return labels instead of scores.
• Playground. Run several prompts or models against a dataset side by side in the browser, with scorers attached.
• Tracing and online scoring. SDK wrappers and auto-instrumentation record model calls, tools and your own spans; automation rules score a sample of production traces in the background; human reviewers annotate traces.
• Gateway. One OpenAI-compatible endpoint for many providers, with caching and logging. The older AI proxy is deprecated.
• Loop and Topics. Loop is Braintrust's built-in agent for querying your data and drafting scorers, datasets and prompt changes, with approval before it edits anything. Topics clusters production logs daily by task, sentiment and issue.

Braintrust Data, Inc. runs the platform. The SDKs are open source: Python, TypeScript, Java and autoevals under MIT, Go, Ruby and C# under Apache-2.0. The control plane is always hosted by Braintrust. Your data plane (where traces, datasets and experiments live) can be Braintrust's SaaS in the US or EU, or, on Enterprise, run in your own cloud account: BYOC (Braintrust operates it) or self-hosted (you operate it).

In the stack, Braintrust sits across evals and observability. It overlaps with Langfuse for tracing and with promptfoo for CI evals, and its strength is joining the two: the same scorers run on experiments and on live traffic.

Pick Braintrust when:

• Several people iterate on one AI feature. Engineers run experiments from code, product managers tweak prompts in the playground, reviewers label traces, and everyone sees the same experiment history.
• You want offline and online scores to agree. A scorer you trust in experiments can run on a sample of production traffic without being rewritten.
• You need comparisons, not just pass or fail. Row-level diffs against a baseline show exactly which cases a change fixed and which it broke.
• Your stack is polyglot. SDKs cover six languages, and OpenTelemetry ingestion covers the rest.

Pick something else when:

• Data must stay in your network and you are not on Enterprise. BYOC and self-hosting are Enterprise only. Langfuse (MIT core) and Arize Phoenix (Elastic License 2.0) can be self-hosted without a paid plan.
• You need a free, local CI gate. promptfoo runs entirely on your machine.
• You evaluate agents in sandboxes or run published benchmarks. Inspect AI is built for that, and its logs can stay local.
• Your volume is high and your budget is fixed. Pricing is metered on data processed and scores recorded; heavy tracing with large payloads adds up. Sample and trim before you send.

An eval has three parts, and Braintrust adds the record-keeping around them:

1. Data. A list of rows or a stored dataset (init_dataset(project=..., name=...)). Each row has an input, usually an expected, and optional metadata.
2. Task. Any function that takes an input and returns an output: one model call, a RAG chain or a whole agent. Wrap your model client (wrap_openai) or call auto_instrument() and every call inside the task is traced as a span under that row.
3. Scores. Functions that receive input, output, expected and metadata and return a number from 0 to 1 (or None when they do not apply). trial_count reruns every row to expose nondeterminism.

Running the file with bt eval creates an experiment and prints a link. In the UI, pick a baseline experiment and every row shows the score delta, with regressions highlighted, plus aggregate metrics such as duration, tokens and cost. The same scorer functions can be pushed to Braintrust and attached to an online scoring rule that samples production traces at a rate you set.

Pricing on 2026-10-13. Starter is free and includes 1 GB of processed data, 10,000 scores, 14-day retention and $10 of built-in model credits a month; beyond that you pay $4 per GB and $2.50 per 1,000 scores. Pro is $249 a month with 5 GB, 50,000 scores, 30-day retention and $100 of credits, then $3 per GB and $1.50 per 1,000 scores. Enterprise is custom and adds BYOC and self-hosting, SAML SSO, audit logs, SOC 2 reports and a BAA. Processed data counts everything you send (inputs, outputs, metadata, spans, datasets, attachments) and is measured at ingestion, so deleting data does not reduce it. Experiments are kept up to 365 days on every plan.