Traceloop: LLM monitoring and evaluation on top of OpenLLMetry

Traceloop is the commercial platform built on OpenLLMetry. OpenLLMetry produces the traces; Traceloop stores them, runs evaluators on them in real time, and runs experiments against datasets so you can test a prompt or model change before it ships. In 2026 Traceloop became part of ServiceNow, which affects how you should buy it.

Traceloop is an LLM observability and evaluation platform:

• Tracing. Every request your app makes, instrumented by the OpenLLMetry SDK, shown as a trace with workflow, task, agent and tool spans, token usage and cost.
• Monitors. An evaluator that runs on every span matching a filter, in real time, so quality drops show up as a metric and an alert rather than a customer complaint.
• Evaluators. A library of built-in checks (answer relevancy, faithfulness, answer correctness and completeness, context relevance, instruction and topic adherence, semantic similarity, PII, prompt injection, toxicity, secrets, tone, agent goal accuracy, tool selection and tool trajectory, among others) plus custom LLM-as-a-judge evaluators you write.
• Experiments and datasets. Versioned datasets, a task function, a list of evaluators: run from code or in CI, with results posted to your pull requests by a GitHub App.
• Prompt registry and playgrounds. Versioned prompts per environment fetched at runtime by the SDK, and playgrounds to test prompts and evaluators on rows of data.

In the stack it is the backend and evaluation layer, fed by OpenLLMetry instrumentation, the same SDK that can feed any other OpenTelemetry backend.

On ownership: Traceloop announced in March 2026 that it was joining ServiceNow, and ServiceNow's 5 May 2026 press release describes "the recently completed acquisition of Traceloop" as the source of runtime agent observability in its AI Control Tower. Traceloop told customers it intends "to honor all obligations to customers and continue to sell the Traceloop functionality", and that it will announce roadmap changes in advance. The product is still sold standalone: the site offers a free plan, a 14-day trial and an Enterprise plan, also through the AWS, GCP and Azure marketplaces.

Pick Traceloop when:

• You already instrument with OpenLLMetry and want monitors, evaluators and experiments without building your own judge pipeline.
• You want ready-made evaluators for common failures, including agent-specific ones such as tool selection quality and trajectory against an expected tool sequence.
• You want experiment results in the pull request, so prompt and model changes are reviewed with numbers next to the diff.
• You run ServiceNow and plan to use its AI Control Tower, which now includes Traceloop's runtime observability.
• You need an on-prem or air-gapped deployment with vendor support. The site claims SOC 2 and HIPAA compliance and cloud, on-prem and air-gapped deployment.

Look elsewhere when:

• You want open-source self-hosting. The platform is closed; Langfuse (MIT) and Arize Phoenix (ELv2) self-host for free.
• You need history on the free plan. Free Forever keeps 24 hours of data; anything longer is an Enterprise contract.
• Vendor independence matters for a multi-year choice. Product direction is now set inside ServiceNow. Keep your instrumentation on OpenLLMetry or the OpenTelemetry GenAI conventions so leaving is a configuration change.

Ingestion. The OpenLLMetry SDK exports OTLP to https://api.traceloop.com with your TRACELOOP_API_KEY as a bearer token. Each API key belongs to one project and one environment, and the dashboard shows data per project and environment, so a production key and a staging key never mix.

Monitors pair a span filter with an evaluator. For every span that matches (for example, every draft_reply task in production), the evaluator runs and the result is stored as a monitor metric you can chart and alert on. Monitors can use LLM-as-a-judge evaluators or Traceloop's built-in deterministic checks.

Evaluators have an input and an output schema. To use one, you map your data onto its inputs: in experiments, the keys your task function returns must include every input the evaluator needs. The same evaluator works in a monitor, an experiment, a playground or as a guardrail that runs synchronously in your code.

Experiments take a published dataset version, an async task function that processes each row, and a list of evaluator slugs. client.experiment.run() executes the task on every row, runs the evaluators and records everything under an experiment slug, so reruns line up.

Prompts live in a registry with versions and environments. With Traceloop Sync turned on, the SDK polls for changes (every 60 seconds by default, 5 seconds in the Development environment) and serves prompts from a local cache, so get_prompt() adds no network call to your request.

Deployment comes in three shapes: the cloud service; a hybrid model where traces are stored in a ClickHouse database in your infrastructure while Traceloop runs the processing pipelines and dashboard; and a full platform deployment where pipelines, storage, metadata and monitoring models all run in your infrastructure, with the dashboard still hosted by Traceloop.