Start your learning journeyAI Playbook
Log inStart learning

AI platform tools

The AI platform stack

The tools that sit between your apps and the models, layer by layer. Each deep dive covers what the tool does, when to pick it, and for members the setup, production settings and failure modes.

The AI platform as a stack of layers, from apps and agents at the top down to serving at the bottom, with the cross-cutting layers standing beside the whole stack. Choose a layer to see its tools.

Tap or tab to a layer to see its tools.

Apps and agents

Frameworks and SDKs for agents that plan, call tools and hand work back.

Laminar: open-source observability for long-running AI agents

v0.3.1

Laminar (YC S24) · Apache-2.0

Best for: Debugging long-running and browser agents: full traces, session replay, failure detection across runs, and evals

Apps and agents

LangGraph: stateful agent workflows you can pause, resume and replay

1.2.15

LangChain, Inc. · MIT

Best for: Agents and workflows that must persist state, wait for human approval, survive restarts and mix fixed steps with model-driven ones

Apps and agents

Claude Agent SDK: Claude Code's agent loop inside your own app

v0.2.165

Anthropic · Use under Anthropic's Commercial Terms; Python package code MIT, TypeScript package proprietary

Best for: Agents that read and edit files, run commands and call tools, with Claude Code's permissions, hooks and sessions built in

Apps and agents

Decisions and control

Typed decision models and gates that choose when automation acts and when a person does.

Jev by TypeSafe AI: typed decisions your code can branch on

v0.6.0

TypeSafe AI · Proprietary API; the Python and JavaScript SDKs are MIT

Best for: Routing, gating and scoring text inside your code with calibrated confidence, at request-path speed and low cost

Decisions and control

Gateway

One endpoint in front of every model provider, with keys, budgets and fallbacks.

LiteLLM: one gateway in front of every model provider

v1.105.0

BerriAI · MIT (the enterprise/ directory is under a commercial licence)

Best for: One OpenAI-compatible endpoint in front of many providers, with per-team keys, budgets, fallbacks and spend tracking

Gateway

Serving

Run open models on your own GPUs with batching and caching.

vLLM: serve open-weight models on your own GPUs

v0.31.0

vLLM project (PyTorch Foundation) · Apache-2.0

Best for: Serving open-weight models at high throughput on your own GPUs behind an OpenAI-compatible API

Serving

Observability

Traces, token counts, cost and latency for every model call and agent step.

Helicone: proxy-based LLM logging and gateway, now in maintenance mode

v2025.08.21

Helicone (part of Mintlify since March 2026) · Apache-2.0

Best for: Teams already on Helicone, or logging requests from several services by changing one base URL

Observability

Langfuse: open-source tracing, evals and prompt management

v4.58.0

Langfuse (ClickHouse, Inc.) · MIT (code under ee/ directories has a separate commercial licence)

Best for: Self-hostable tracing, prompt management and evals for LLM apps and agents, with OpenTelemetry ingestion and an MIT core

Observability

Traceloop: LLM monitoring and evaluation on top of OpenLLMetry

0.62.4

Traceloop (ServiceNow) · Commercial platform (SDKs are Apache-2.0)

Best for: Teams on OpenLLMetry who want monitors, built-in evaluators and CI experiments on the same traces, with on-prem options

Observability

Arize Phoenix: open-source tracing and evals on OpenInference

20.21.1

Arize AI · Elastic License 2.0 (OpenInference, phoenix-otel and phoenix-client are Apache-2.0)

Best for: Free self-hosted tracing, evals, experiments and prompt iteration, from a laptop to a Postgres-backed team server

Observability

Evals

Test sets, scores and CI gates that show whether a change helped or hurt.

promptfoo: eval and red-team your prompts from a YAML file

0.124.1

Promptfoo (part of OpenAI since 2026) · MIT

Best for: Prompt and model regression tests in CI, side-by-side model comparisons, and red-teaming an LLM app before launch

Evals

Braintrust: datasets, experiments, scorers and production traces in one loop

0.45.0

Braintrust Data, Inc. · Proprietary platform; open-source SDKs (MIT or Apache-2.0)

Best for: Teams that want eval experiments, datasets, human review and production scoring in one hosted tool with SDKs in six languages

Evals

Inspect: the UK AI Security Institute's framework for LLM and agent evals

0.3.279

UK AI Security Institute and Meridian Labs · MIT

Best for: Rigorous evals of models and agents, especially agent tasks that need sandboxed tools, many epochs and inspectable transcripts

Evals

Optimization

Tune prompts and programs against a metric instead of by feel.

DSPy: write the program, let an optimizer write the prompts

3.4.1

Stanford NLP and the DSPy open-source community · MIT

Best for: Pipelines with a measurable metric and labelled examples, where you want better prompts, cheaper models or a model swap

Optimization

Standards

Open specifications that let tools from different vendors read the same data.

OpenLLMetry: open-source OpenTelemetry instrumentation for LLM apps

0.62.4

Traceloop (part of ServiceNow since 2026) · Apache-2.0

Best for: Vendor-neutral tracing of LLM, vector database and framework calls into any OpenTelemetry backend, set up in one line

Standards

OpenTelemetry GenAI conventions: one trace format for models and agents

1.2b0

OpenTelemetry (CNCF) · Apache-2.0

Best for: Vendor-neutral traces and metrics for LLM calls, agents, tools and MCP that any backend can read

Standards

Want to run this stack yourself? The AI Platform Engineering program builds it in labs.

See the program