LiteLLM: one gateway in front of every model provider

LiteLLM is the gateway between your applications and every model they call. Version 1.105.0 shipped on 11 October 2026. A new stable minor line ships about every week and only the four newest lines get patches, so budget for an upgrade every month.

One repository holds two products:

• The Python SDK. litellm.completion(model="anthropic/...", messages=[...]) calls any of 100+ providers and always answers in the OpenAI Chat Completions shape. Its Router class adds load balancing, retries and fallbacks inside one Python process.
• The AI Gateway (the proxy). A Docker image that serves an OpenAI-compatible and Anthropic-compatible API on port 4000. With Postgres behind it, it issues virtual keys, enforces budgets and rate limits, records spend per request and ships an Admin UI at /ui. Any OpenAI or Anthropic SDK, and coding agents such as Claude Code and Codex, work against it by changing the base URL.

In the stack it is the gateway layer. Applications and agents call LiteLLM; LiteLLM calls hosted APIs and your self-hosted vLLM servers, and sends metrics and traces to your observability backend.

LiteLLM is built by BerriAI. Everything outside the repository's enterprise/ directory is MIT. Enterprise features turn on with a licence key (LITELLM_LICENSE) on the same image, so there is no separate build to migrate to. Two 2026 incidents are worth knowing before you deploy. On 24 March 2026 attackers published malicious PyPI packages litellm==1.82.7 and 1.82.8 (LiteLLM's investigation pointed to a compromised maintainer account); the official Docker image was not affected, versions 1.78.0 to 1.82.6 are verified clean, and releases from 1.83.0 come from a rebuilt pipeline with cosign-signed images. On 18 March 2026 an advisory disclosed that versions before 1.82.3 could leak authorization headers into spend logs and OpenTelemetry traces when a custom guardrail returned the whole request.

Pick LiteLLM when:

• Several teams or services call several providers and you want one place for keys, model access and spend.
• Budgets must deny, not just notify. A key, user or team over its budget is rejected with a budget_exceeded error before the provider is called (HTTP 422 since v1.104.0; set budget_exceeded_status_code: 429 to keep the older code).
• One provider's outage or rate limit must not take your product down. Fallbacks move traffic to another model group automatically.
• You serve some traffic from your own GPUs and want it behind the same API and the same keys as hosted models.

Use the SDK alone when a single Python service needs fallbacks and cost tracking and nothing else. Run the proxy as soon as two services, two languages or two teams share keys and budgets.

Skip it when one service calls one provider and needs no routing, budgets or keys: you would add a network hop, Postgres and Redis for nothing. If all you need is visibility into calls, an observability SDK such as Langfuse or Arize Phoenix is less to run.

The nearest alternatives are Portkey's gateway (MIT), Envoy AI Gateway (Apache-2.0, built on Envoy), Bifrost (Apache-2.0, written in Go), managed gateways such as Cloudflare AI Gateway, and hosted model marketplaces such as OpenRouter. Helicone's gateway has been in maintenance mode since Helicone joined Mintlify in March 2026.

Everything starts from a config.yaml with four sections. Values written as os.environ/NAME are read from the environment, so secrets never sit in the file.

• model_list. Each entry maps a public model_name (the name your apps send) to litellm_params (the real provider/model, credentials, api_base, and per-deployment rpm, tpm or weight). Entries that share a model_name form a model group.
• router_settings. How a model group picks a deployment and survives failures. routing_strategy defaults to simple-shuffle (weighted random), with latency-based-routing, usage-based-routing-v2, least-busy and cost-based-routing available, and per-group overrides through routing_groups. The docs recommend staying on simple-shuffle at high traffic, because usage-based routing adds Redis lookups to every request. Retries (num_retries, default 2), per-error retry_policy, cooldowns (allowed_fails, default 3; cooldown_time, default 5 seconds) and fallbacks live here too.
• litellm_settings. Proxy-wide behavior: request timeout, response caching, logging callbacks, guardrail defaults.
• general_settings. The master key, the database URL and the operational knobs: spend write batching, connection pool size, retention.

Fallbacks come in three kinds: context_window_fallbacks when a prompt is too long for the model, content_policy_fallbacks when a provider refuses the content, and fallbacks for everything else, such as rate limits and outages. A request fails over only after retries on its own group are exhausted.

Access is layered. The master key is the admin credential and doubles as the Admin UI password. Virtual keys are what you hand to services and people, and each carries an allowlist of models, max_budget with a budget_duration reset, rpm_limit, tpm_limit, max_parallel_requests and free-form metadata. Keys belong to users and teams with their own budgets, and you can budget your own customers through the OpenAI user field. Every request is priced from LiteLLM's model cost map and written to the LiteLLM_SpendLogs table in Postgres.

State is the part people underestimate. Keys, spend and stored config need Postgres. As soon as you run more than one worker process, rate limits, budgets, cooldowns and the cache need Redis, or each worker counts on its own and a limit of 100 requests per minute becomes 100 per worker.