* test(e2e): datadog log delivery for successful chat, messages, and responses
Covers logging.datadog.success.exports_metric on all three routes: one
successful non-streaming call must reach the DataDog logs intake as exactly
one log event whose StandardLoggingPayload message carries the model group,
real token counts, and a response cost equal to the x-litellm-response-cost
header of the same response. Delivery is judged at the intake: the compose
stack gains a dd-sink service recording every batch the datadog callback
ships via the DD_BASE_URL testing override, and a typed reader replays it.
Writing these caught a live product bug: /v1/messages double-logs every
success (two byte-identical events per call), filed as LIT-4447; the messages
test tolerates byte-identical duplicates of the one event until it lands,
while a second differing event still fails
* test(e2e): address review findings on the datadog delivery suite
Consolidates the fresh-key first_ok helper into logging_client now that the
otel PR it mirrored has merged (both test files use the shared copy), moves
intake batch parsing into a helper so no path can leave the batch unbound,
and gives the sink's /health endpoint a truthful text/plain content type
* test(e2e): tolerate same-logical-event duplicates by call id, not byte identity
A clean LIT-4447 repro showed the duplicated payload is built twice and can
mint a fresh synthetic completion id per emission, arriving as two separate
intake POSTs with the same litellm_call_id and identical substantive fields.
Byte-identity was therefore a flaky criterion; duplicates now qualify only
when they share the call id, call type, model group, tokens, and cost, and a
second differing event still fails
* test(e2e): assert the scenario strictly; the messages test is the LIT-4447 regression pin
Per review direction the tests now assert exactly what the scenario promises:
exactly one DataDog log event per successful call, on every route. The
/v1/messages test therefore fails on current code against the known
double-log (LIT-4447) and is its regression pin; it goes green when the fix
lands. The duplicate-tolerance machinery is removed
* Simplify docstrings for DataDog log tests
Removed redundant phrasing about cost cross-checking in docstrings.
* Update test_datadog_log_e2e.py
Making the in-memory issuer reflect a trust-on-first-use discovered value fixed
the registry/row token-identity drift, but it overloaded a single field: the
carry-forward gate keyed on issuer truthiness as a proxy for "endpoints are
anchored to a pinned issuer, fail-closed". A discovered issuer is truthy yet not
anchored, so a resource-rooted server that had learned its issuer would drop its
last-known-good endpoints on a transient discovery blip instead of carrying them
forward.
Anchoring is now a first-class property rather than a proxy. MCPServer carries
issuer_is_anchored, set at both build paths from the single _uses_issuer_anchor
definition (a pinned issuer on a discovery auth type). issuer stays the identity
value used by the token-identity tuple and the serializers; issuer_is_anchored is
the provenance value the carry-forward gate reads to decide fail-closed. The two
properties can no longer be conflated, so a discovered issuer keeps its
resource-rooted endpoints carrying forward while a pinned issuer still fails
closed.
Regression tests pin both directions: a discovered-but-not-anchored server
restores its endpoints on a discovery blip, an anchored server does not, and the
build sets issuer_is_anchored true only when the issuer is pinned
Pin a session's first-turn model for the rest of the session by default
instead of reclassifying every turn. Keeps multi-turn sessions on a single
model, preserving provider prompt caches and avoiding cross-model
conversation-history errors (e.g. Anthropic rejecting a thinking block
produced by a different model). Requests without a resolvable session_id are
unaffected. Set session_affinity: false to opt out.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cli): add `lite up`/`lite down` to ambiently route Claude Code through the proxy
Patches ~/.claude/settings.json in place (env.ANTHROPIC_BASE_URL + apiKeyHelper
via `lite auth print-token`) so any `claude` session started afterward, from
any terminal, routes through the local LiteLLM proxy with no wrapper command
needed, unlike the existing `lite claude` subprocess-exec approach. Backs up
the original file first and restores it on Ctrl-C/SIGTERM, or via `lite down`
after an unclean exit. Cursor is not supported: no equivalent file-based config
to patch.
* feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy (#33249)
* feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy
Lets a customer try litellm's complexity_router against models they already
have on their existing, unmodified production proxy, with no config.yaml
edits and no new infra. lite autoroute configure discovers accessible
models via /model_group/info and walks through tier assignment (plus
optional LLM classifier / semantic matching / adaptive selection); every
referenced model becomes its own litellm_proxy/<name> deployment forwarding
back to the real proxy with the real key, so every actual call, routed
completions, classifier calls, embedding calls, still lands on their real
proxy. lite autoroute up launches that generated config as an ephemeral
local proxy, patches ~/.claude/settings.json to point Claude Code at it, and
streams routing decisions live; Ctrl-C/SIGTERM (or lite autoroute down
after an unclean exit) restores everything.
Also adds lite model-groups list (a thin CLI wrapper over the existing
ModelGroupsManagementClient), and generalizes up.py's settings-backup/restore
helpers to take explicit paths so this feature can reuse them instead of
duplicating the logic.
Depends on litellm_lite_up_down (#33231) for that generalization.
* feat(cli): allow multiple models per autoroute tier
complexity_router already supports a pool of models per tier (randomly
picked per request; adaptive mode specifically needs a pool to choose
within), but the configure wizard only ever let you assign one. Tiers are
now a tuple of model names; the wizard prompt accepts comma-separated
indices to pick more than one per tier.
* feat(cli): fuzzy model picker and auto-route Claude Code to autorouter
Numbered-index selection didn't scale past a handful of models, so switch
the tier picker to InquirerPy's fzf-style fuzzy search. Also set
ANTHROPIC_DEFAULT_{SONNET,HAIKU,OPUS}_MODEL to "autorouter" in Claude
Code's settings, since Router resolves auto-router deployments by literal
model name with no wildcard support, so a "*" catch-all model_name would
never match real traffic.
* feat(cli): allow installing lite CLI from source via LITELLM_CLI_REF
Lets testers try an unreleased branch's CLI changes with the same
curl-piped installer, instead of waiting for a PyPI release.
* fix(ci): modernize type hints to clear ruff strict-rule budget
* fix(ci): bump httplib2 and setuptools to patched versions
Clears osv-scan findings for PYSEC-2026-3444 and PYSEC-2026-3447.
* fix(cli): write autoroute's secret-bearing files with mode 0600
commands.py wrote config.yaml (embeds the real proxy key) and Claude
Code's settings.json (embeds the ephemeral proxy's master key) with
plain open(), landing at the umask-derived default (commonly 0644)
until a later chmod call caught up. That window, and the missed case
where settings.json already exists (chmod never ran at all there),
left a credential-bearing file readable by another local account.
secure_create() fixes the mode via fchmod on the fd before any
content is written, covering both the brand-new-file and
already-exists cases, and commands.py/wizard.py now route their
sensitive writes through it.
* docs(cli): warn that a stale Claude Code session can leak to a squatted port
lite autoroute up's master key is embedded statically (unlike lite up's
apiKeyHelper, resolved per request), so a Claude Code session still
running after teardown keeps sending it, along with prompt content, to
a now-unbound loopback port that another local account can bind. This
is the same one-time-patch tradeoff lite up already accepts, just with
a static secret instead of a re-resolved one -- document it in the
README's Caveats section and surface it in the teardown message itself.
* fix(cli): address greptile review feedback on autoroute PR
- terminate the ephemeral proxy child process when its health check
fails, instead of leaking an orphaned, unrecoverable process bound
to the port
- replace bare assert isinstance checks (no-ops under python -O) with
click.ClickException in the model-groups list and configure wizard
code paths
- close launch_proxy's log file handle once the child process has
inherited its fd, instead of leaking it
- add build_generated_proxy_config to config.py's __all__
* fix(cli): close TOCTOU window in lite up's settings backup write
write_backup wrote the backup (which can embed the original
apiKeyHelper/settings content) with plain open() + a chmod call after
the fact -- the same permissive-until-corrected window already fixed
for autoroute's config.yaml and Claude settings writes, and missed
entirely when the backup file already exists with broader permissions.
Moves secure_create (atomic-enough 0600 via fchmod before any content
is written) to up.py, the module both lite up and lite autoroute
share, and has autoroute/process.py import it from there instead of
keeping its own copy.
* fix(cli): refuse autoroute up when a stale backup exists from a crash
The pid-record check only catches a still-live duplicate process; a
SIGKILL'd `up` leaves no live pid but does leave AUTOROUTE_BACKUP_PATH
behind. Without this guard, a fresh `up` overwrote that backup with
the currently-patched Claude settings instead of the true originals,
so `down`/Ctrl-C would restore the wrong content permanently. up.py's
`lite up` already guards the analogous case; mirror it here.
* fix(cli): bind the ephemeral autoroute proxy to loopback only
proxy_cli.py defaults --host to 0.0.0.0 when not passed explicitly.
launch_proxy never passed it, so the ephemeral proxy -- despite every
base_url in this module being built from 127.0.0.1 -- was actually
reachable from other hosts on the network, including its
unauthenticated-until-config-lands routes before the master key is
wired in.
* docs(cli): show curl install for the autoroute QA flow
Points readers at scripts/install-cli.sh's curl one-liner instead of
assuming uv/pip is already set up, and documents the LITELLM_CLI_REF
override for trying an unreleased branch or commit.
* fix(cli): surface a clean error on an empty or corrupt autoroute config
A configure run killed between secure_create's O_TRUNC and the write
completing leaves an empty config.yaml on disk. The next up read that
via yaml.safe_load (None) into the generated-config TypeAdapter
uncaught, surfacing a raw pydantic.ValidationError instead of pointing
the user back at `lite autoroute configure`.
* fix(cli): bind lite up's apiKeyHelper to the proxy it was started against
_ensure_fresh_login only checked token freshness, not which proxy the
cached token belonged to, and resolve_api_key_helper built a bare
`lite auth print-token` command with no --base-url. A user logged into
proxy A who ran `up --base-url proxy-b` (or LITELLM_PROXY_URL=proxy-b)
would silently get proxy A's real token wired into Claude Code's
apiKeyHelper; since apiKeyHelper is invoked bare, print-token's
existing origin check never engaged, so proxy B -- attacker-controlled
or not -- received every subsequent request's Authorization header
carrying proxy A's credential.
_ensure_fresh_login now requires the cached token's base_url to match
before treating it as usable, forcing a fresh login for the selected
proxy otherwise. resolve_api_key_helper now takes that base_url and
threads it through as an explicit --base-url, so print-token's
existing (but previously unreachable in the apiKeyHelper flow)
base_url_explicit check actually enforces the match at request time
too.
* fix(cli): surface clean errors instead of raw tracebacks in lite up/down
load_json_or_empty and read_backup both delegate to pydantic's
validate_json, which raises ValidationError on invalid JSON or a
non-object root -- neither up() nor down() caught it, so a corrupt
settings or backup file surfaced an unformatted Python traceback
instead of a clean CLI error. Both now convert to UpError, and down()
(previously uncaught entirely) and up()'s teardown path now handle it.
restore_claude_settings also gained a parent.mkdir guard before
rewriting CLAUDE_SETTINGS_PATH: if ~/.claude/ was removed while `lite
up` was running, the restore would crash before deleting the backup
file, permanently stranding it and breaking every future `lite down`.
* docs(cli): call out env-var auth for autoroute commands
* fix(cli): clean up leaked proxy and surface clean errors in autoroute
Three related gaps, all following an UpError getting raised somewhere
that wasn't catching it yet:
- up() left the just-launched ephemeral proxy running with no pid
record if load_json_or_empty/write_backup/secure_create raised after
the health check passed, mirroring the existing ProcessLaunchError
cleanup for the health-check-failure branch.
- _teardown() didn't catch restore_claude_settings raising UpError
(e.g. a corrupt backup at stop time), which would otherwise escape
to Click as an unhandled error in the normal-exit path, or print
"Error in atexit" in the atexit path. up.py's own _restore_once
handles the identical case the same way.
- read_pid_record let a corrupt PID file surface a raw
pydantic.ValidationError instead of a clean message, and did so in
down(), the command specifically meant for crash recovery. down()
now clears an unreadable pid record and continues cleanup instead of
aborting, since a corrupt pid file must never block the one command
meant to recover from exactly this kind of crash.
* docs(cli): warn against running lite up and lite autoroute up together
Two lifecycle gaps let the issuer trust anchor drift out of sync with the
endpoints it governs. Changing or clearing a previously pinned issuer left the
authorization_url and token_url that were resolved under the old issuer in the
row, so clearing the anchor could revive stale, possibly untrusted endpoints
instead of re-discovering. And a build that discovered an issuer
trust-on-first-use persisted it to the row while the returned in-memory server
kept the issuer unset, so the registry and the row disagreed and the per-user
OAuth token identity, which includes the issuer, differed between that build and
the next rebuild and forced a spurious re-auth.
update_mcp_server now treats a change to a previously pinned issuer the same as a
url or auth_type change and clears the auth-flow-scoped endpoint fields that were
resolved under it. The trigger fires only when an issuer was already pinned and
is now changed or cleared, so establishing one for the first time, including the
trust-on-first-use discovery write-back, does not wipe the fields it just
resolved.
Both build paths, build_mcp_server_from_table and load_servers_from_config, now
construct the server with effective_issuer = manual_issuer or the discovered
issuer, skipping an origin-fallback guess exactly as the persistence does, so the
in-memory object always reflects what the row will hold.
Regression tests pin each case: clearing and re-pointing a pinned issuer clear
the stale endpoints, a first-time establish preserves the discovered fields, and
a build reflects the discovered issuer while an origin-fallback guess is not
reflected
* fix(complexity_router): return empty dict from _classifier_call_metadata when metadata is absent
The LLM classifier reads request_kwargs.get("litellm_metadata"), but the proxy stores request metadata under "metadata", so this returned None. _classifier_call_metadata then passed None straight through to the classifier acompletion call, which assumes a dict and blows up with 'NoneType' object has no attribute 'update'; the router swallowed it and silently fell back to heuristic scoring, so the configured LLM classifier never ran. Returning an empty dict keeps the classifier call well-formed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover complexity-router LLM classifier routes over the proxy
Add a live e2e regression for the complexity auto-router: a lexically simple but hard prompt ("Is P equal to NP?") is routed by the LLM classifier to the higher-tier anthropic backend, read back from the spend log's model. Before the metadata fix the classifier silently crashed and the router fell back to heuristic SIMPLE scoring on the openai backend, so this test fails pre-fix and passes post-fix.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
When an admin pins an issuer, RFC 8414 section 3.3 makes that issuer the sole
authoritative source of the authorization and token endpoints, so a compromised
or misconfigured upstream cannot smuggle a token endpoint by echoing the pinned
authorize URL. The first cut enforced that only on the database build path; the
carry-forward, persistence, config-load, serialization and sanitization paths
could still restore or emit upstream-derived endpoints for an issuer-anchored
server, which is the class of gap the review flagged.
Every site now routes through one predicate. _endpoints_yield_to_issuer returns
all-None whenever the issuer is the anchor, so both build paths,
has_all_upstream_oauth_fields, needs_discovery and the endpoint merge defer to
the issuer. _carry_forward_resolved_oauth_endpoints carries only scopes for an
issuer-anchored server and fails closed on endpoints.
_persist_discovered_oauth_endpoints skips endpoint writes under the anchor. The
two table serializers round-trip the issuer and both non-admin sanitizers redact
it. Scope selection stays resource-driven per the MCP authorization spec:
_fetch_issuer_anchored_oauth_metadata takes endpoints from the issuer document
and scopes from the resource document.
The OAuth metadata resolution and corroboration gating for the database build
path move into _resolve_table_oauth_metadata so build_mcp_server_from_table
stays within the cyclomatic-complexity budget without changing behavior.
Regression tests pin the invariant at each site: the issuer overrides stored
endpoints even when they are populated, carry-forward does not restore endpoints
under the anchor, persistence does not write endpoints under the anchor, a url or
auth_type change clears stale issuer-scoped fields even when resubmitted
unchanged, the Azure heuristic stays reachable under a required issuer, and
anchored metadata takes endpoints from the issuer while scopes come from the
resource
The issuer anchor is for the token/registration endpoints only (the RFC 9700
mix-up). Scope selection stays resource-driven per the MCP authorization spec
Scope Selection Strategy: _fetch_issuer_anchored_oauth_metadata now validates
the issuer document (RFC 8414 §3.3) for the endpoints and separately fetches the
resource's advertised scopes (WWW-Authenticate challenge, else RFC 9728
scopes_supported) for the scope value, instead of using the issuer document's
own scopes_supported. The resource can influence only the requested scope, which
the authorization server and user consent bound (RFC 6749 §3.3), never the token
endpoint.
Discover the issuer from the upstream and persist it trust-on-first-use (fill-empty-only,
frozen thereafter), so admins do not have to type it; an admin-configured issuer always wins
and is never overwritten. Re-pointing the server url now clears the discovered issuer and
endpoints (matching the existing auth_type-change clearing) so a new upstream re-discovers
instead of anchoring on the previous upstream's issuer. Adds the issuer to the per-user OAuth
token identity so re-pointing it purges stale tokens. Surfaces the issuer as an optional,
auto-discovered, overridable field in the create and edit MCP server forms.
C901 gate shows +1 vs staging; that is inherited from the #33317 stack base (delta 0 against
Adds an admin-configured issuer to MCP servers. When set, OAuth metadata is
fetched from the issuer's own origin and adopted only when the document
self-attests that same issuer (RFC 8414 §3.3), making token_endpoint,
registration_endpoint, and scopes authoritative for the pinned issuer instead
of a document the MCP resource server chose. This closes the mix-up where a
compromised resource echoes a pinned authorization_url to smuggle its own
token endpoint and inflated scopes past the corroboration gate. Discovery is
same-authority against the issuer origin, fails closed on a §3.3 mismatch, and
does not fall back to resource-rooted discovery. Rows without an issuer keep
the existing corroboration-gate behavior unchanged.
Backend + schema only; UI field and live-proxy proof follow.
Reverts the over-correction that restricted a pinned-authorization_url server's
discovered scopes to the authorization server's own scopes_supported. Per the MCP
authorization spec Scope Selection Strategy and RFC 9700 §2.3, the scopes a client
requests are resource-driven: the WWW-Authenticate 401 challenge scope, else the
RFC 9728 protected-resource scopes_supported. The authorization server's RFC 8414
scopes_supported is a non-exhaustive capability list (the server MAY omit supported
scopes) and is never the selection source; scope inflation by a compromised resource
is bounded by the authorization server and user consent (RFC 6749 §3.3), not by the
client restricting the request. The corroboration gate now rejects only the
uncorroborated token_url/registration_url (the RFC 9700 endpoint mix-up) and leaves
scopes untouched. Removes the now-unused authorization_server_scopes field.
Adds a 'passthrough' feature row to the Claude Code compat matrix that
drives the real claude CLI in each cloud's native mode against
LiteLLM's passthrough routes (the LLM-gateway setup from
code.claude.com/docs/en/gateway) instead of the /v1/messages
translation layer:
- anthropic: ANTHROPIC_BASE_URL={proxy}/anthropic, forwarded verbatim
to api.anthropic.com
- bedrock_invoke: CLAUDE_CODE_USE_BEDROCK=1 against {proxy}/bedrock;
the router resolves the alias in /model/{alias}/invoke-with-response-stream
- vertex_ai: CLAUDE_CODE_USE_VERTEX=1 against {proxy}/vertex_ai/v1;
alias, project, location and credentials resolve from the deployment,
which now sets use_in_pass_through: true (and the canonical
vertex_project/vertex_location param names) in test_config.yaml
- azure: CLAUDE_CODE_USE_FOUNDRY=1 against {proxy}/azure via the
AZURE_API_BASE/AZURE_API_KEY fallback (documented in the cron env
example)
- bedrock_converse: not_applicable; Claude Code has no Converse-wire
client
The shared cell body lives in _passthrough.py with injectable runner
and env (no monkeypatching), unit-covered in
_driver_unit_tests/test_passthrough.py including pins on the per-mode
CLI env contracts captured from a real claude CLI (2.1.210) run
against a request-logging sink.
Clients calling the standalone apply_guardrail endpoint had no way to pass
per-request configuration to custom guardrail implementations. This adds an
optional metadata field to ApplyGuardrailRequest and forwards it to
CustomGuardrail.apply_guardrail via request_data, only when the client sends
it. The messages guard is aligned to the same is-not-None semantics so an
explicitly-sent empty list is forwarded instead of silently dropped.
The Admin UI's Guardrail Test Playground gains an optional Metadata JSON
input (validated client-side) wired through applyGuardrail in networking.tsx,
so parameterized guardrails can be exercised from the dashboard.
Tests cover metadata alone, metadata with messages, explicit empty values,
the omitted-field passthrough, and the UI panel's parse/error behavior
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Three process-lifetime retention points kept full request payloads
(messages included) alive after the request finished. Under bursts of
large-token traffic (~73K tokens/request mean) this presented as
stepwise RSS growth that never returned to baseline, ending in OOM:
1. Logging.pre_call/post_call stored their entire locals() (messages,
the Logging object, complete_input_dict) in the module-level
litellm.error_logs dict, pinning the most recent request's payload
per worker forever. Nothing reads that dict; the writes are removed.
2. LLMCachingHandler.request_kwargs kept litellm_logging_obj inside the
stored kwargs while the handler itself hangs off
logging_obj._llm_caching_handler, closing a reference cycle
(Logging -> LLMCachingHandler -> kwargs -> Logging). Cyclic payloads
are only reclaimed by generational GC, so megabytes of dead request
data lingered until a rare gen-2 pass, and the transient copies
fragment the allocator into a permanent RSS high-water mark. The
handler now drops litellm_logging_obj from its stored kwargs; the
caching layer never reads it.
3. The router stored every request's kwargs in the ITPM/OTPM contextvar
even when no deployment configures itpm/otpm. Pooled resources
created mid-request (e.g. redis connections) capture the asyncio
context, extending that pin far past the request. The slot is now
populated only for deployments with io token limits and overwritten
with None otherwise.
Live-proxy verification (bursts of 30 x ~300KB requests, PII guardrail
+ prometheus + redis cache): unfixed grows 16-29MB per burst without
release; fixed grows under 1MB per burst after warmup and flattens.
Resolves LIT-4434
Replace list(set(...)) dedupe with dict.fromkeys so callback insertion
order is preserved deterministically instead of being randomized by set
iteration order (influenced by PYTHONHASHSEED). Applies to both the
Logging and ProxyLogging implementations.
Fixes#33003
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(guardrails): add Compresr guardrail for query-aware context compression
Adds a first-class guardrail that compresses bulky message content (tool
outputs, RAG chunks, search results) through the Compresr API before the
request reaches the LLM, via the apply_guardrail / structured_messages hook
so it covers /chat/completions, /v1/messages, and /v1/responses (the latter
through the texts channel, mirrored only when the replacement is
unambiguous; anything ambiguous is left uncompressed).
Distinct from whole-conversation compressors:
- Query-aware: each message is compressed against the intent that produced
it (a tool output against its originating tool call's name + arguments,
resolved via tool_call_id; otherwise the last user message).
- Recoverable: each compressed message carries a hash marker and the request
gains a compresr_retrieve tool, so the model can pull the original content
back through the agentic loop when the compressed version is not enough.
Originals are cached in-process, scoped to the caller's virtual-key hash
plus the request's litellm_call_id, with a TTL and a per-call byte cap;
recovery is skipped when no caller scope is available so one caller can
never read another's originals. The store is per-process, so multi-worker
deployments need sticky routing (or enable_retrieval=false).
Fail-closed by default (fail_open configurable), SSRF-validated api_base
(alternate IP-literal encodings included), cross-tenant-isolated recovery
store, and upstream errors redacted from client-facing responses. The
outbound client follows redirects and re-resolves DNS per request, so the
api_base host/IP checks are defense-in-depth, not a full SSRF guarantee;
this is documented as a known limitation. Requests where nothing was
actually compressed are returned untouched (same object identity) so
handlers skip the write-back. Auto-discovered via the guardrail_hooks
registry.
* fix(guardrails): cap Compresr recovery store total memory
The recovery store bounded bytes per call and entry count, but had no
aggregate cap: 256 tracked call ids at the 10 MiB per-call default could
retain ~2.5 GiB per worker. A flood of requests with distinct
x-litellm-call-id values and large compressible tool outputs could
exhaust a shared proxy worker.
Add a global byte budget (_MAX_TOTAL_STORE_BYTES, 256 MiB) across all
entries. A running total is maintained on every insert/eviction so the
cap is enforced without re-encoding the whole store on the request path;
oldest entries are evicted once the budget is exceeded, always keeping
the most-recent entry so recovery still works for the request populating
the store. +2 regression tests.
* fix(guardrails): gate and bound Compresr recovery loop
Two hardening fixes to the compresr_retrieve agentic loop:
1. Only run the loop when a retrieve call resolves to recovery state this
guardrail actually created for the request. Previously the gate checked
only that the caller-supplied tool list contained a compresr_retrieve
function and that the model emitted a call, so a caller could define
their own same-named tool and force an extra provider round-trip with
nothing to recover. The plan now returns run_agentic_loop=False when no
requested hash resolves.
2. Bound the follow-up against retrieval amplification: each distinct hash
is expanded at most once (repeats get a short marker) and at most
_MAX_RETRIEVALS_PER_LOOP calls are honored, so prompting the model to
call compresr_retrieve many times with the same marker cannot balloon
the follow-up. _retrieve_original now returns None on miss.
+3 regression tests; two existing security tests updated to assert the
stronger veto behavior (forged/cross-tenant hashes now stop the loop
entirely instead of returning a not-found follow-up).
* fix(guardrails): warn when Compresr recovery is skipped without auth scope
When enable_retrieval is on (the default) but the proxy has no per-key
auth, the request has no caller scope, so recovery is silently disabled:
content is compressed but the compresr_retrieve tool is never injected and
the originals are dropped, with no runtime indication. Emit a one-shot
call-time warning so operators can see recovery is being suppressed and
configure virtual-key auth. +1 regression test.
* style(guardrails): tighten Compresr guardrail comments
Condense the verbose multi-line inline comments and the api_base docstring
to concise form. No behavior change.
* fix(guardrails): keep injected tool on Responses API + bound recovery markers by byte cap
Two fixes for reviewer-flagged defects in the Compresr guardrail:
- Responses API: _merge_tools_after_guardrail iterated only over the
request's original tools, dropping any tool a guardrail appended (the
compresr_retrieve recovery tool) whenever the request already had tools.
Keep the appended tools so recovery works on /v1/responses.
- Recovery markers: markers + originals were built for every compressed
target before the per-call byte cap trimmed the store, so an evicted
original left a marker the model could never retrieve. Attach recovery
only while the store (existing entries under the same key + this call's
originals) stays within the cap, so a shipped marker is always retrievable
-- including on a later turn that reuses the store key.
Adds regression tests for both paths.
* refactor(guardrails): extract _existing_originals to keep apply_guardrail under the complexity gate
The byte-cap fix added a branch to apply_guardrail, tipping it past the
C901 complexity ceiling. Move the store lookup into a small helper; no
behavior change.
* fix(guardrails): harden Compresr SSRF blocklist, re-arm no-scope warning, tolerate odd tool shapes
* fix(guardrails): rerun input guardrails on Compresr retrieval follow-up
* chore: remove unrelated deepkeep files committed by mistake
---------
Co-authored-by: charafkamel <charafkamel@live.com>
* feat(proxy): push-based OTLP billable-request metering for enterprise deployments
Adds opt-in, license-gated metering that counts 2xx HTTP requests to LLM
inference, MCP, and A2A endpoints and exports them over mutual TLS to a global
OpenTelemetry Collector for request-based billing.
A pure ASGI middleware (BillableRequestMetricsMiddleware) classifies each
request by route and records one count per 2xx response via an injected
recorder. The recorder (BillingMetricsRecorder) owns a dedicated OTEL meter
provider and an OTLP/gRPC exporter authenticated with client certificates, kept
isolated from the global meter provider so a customer's own OTEL metrics are
untouched. The recorder is built only when a valid LITELLM_LICENSE is present
and the cert material is configured; otherwise the middleware is a transparent
pass-through.
Deployment identity rides on the mTLS client certificate rather than the
payload, so the secret license key is never sent as an attribute or header; only
the license org id travels as a resource attribute for cross-checking.
Resolves LIT-4089
* fix(proxy): align billable-request metering with the global collector
- switch the exporter to OTLP/HTTP with a TLS client certificate. The
collector front end terminates mutual TLS and validates the client cert
against our CA; server verification uses the system trust store, so the
CA env var is now an optional override for private collectors
- resolve the metrics recorder on the first request via a factory instead
of at import time, so deployments that provide the license and cert env
vars through the YAML config's environment_variables export correctly
- close the metering bypass: classify /images/edits, /images/variations,
/v1/messages, /v1/videos, video remix, /v1/ocr and Gemini generateContent
as billable, and gate LLM routes to POST so GET reads (list videos, fetch
a response) do not bill. Verified live: the collector count matches the
UI usage page successful_requests exactly, with failures excluded on both
sides
* fix(proxy): wrap enterprise billing import in try-except per code-quality gate
The check_unsafe_enterprise_import gate requires every import from an
enterprise-pathed module to be guarded. Annotate the factory with the
middleware's BillingRecorder protocol so no enterprise type import is
needed at type-check time
* chore: satisfy strict lint gates in billing modules
- builtin generics per UP006 (dict/tuple instead of typing.Dict/Tuple)
- noqa the deliberate blind catch that keeps metering from breaking startup
- sort proxy_server import blocks split by the guarded enterprise import
* fix(proxy): bill provider passthrough, search, and rag routes
Route-inventory audit against LiteLLMRoutes.llm_api_routes found more
SpendLogs-producing surfaces the classifier missed: provider passthrough
(/bedrock, /vertex-ai, /cohere and the rest of mapped_pass_through_routes),
/v1/search and vector-store search, and the rag ingest/query routes. All are
counted by the dashboard usage page, so missing them undercounts billing.
The passthrough prefix list is read from LiteLLMRoutes so new providers are
picked up without touching this module. /langfuse is excluded: it forwards
observability traffic and writes no SpendLogs row. Known limitation recorded
in the PR: /v1/realtime is a websocket flow the HTTP middleware does not see
* fix(proxy): bill MCP and A2A requests by protocol transport routes only
The billable-request classifier matched the whole /v1/mcp prefix, so
management and discovery reads such as GET /v1/mcp/tools and GET
/v1/mcp/server counted as billable MCP requests, while real MCP tool
calls on the /{server}/mcp and /toolset/{name}/mcp aliases were missed
because their route handlers rewrite the ASGI scope only after this
middleware has already classified the original path. Classify MCP by the
concrete transport surface (the /mcp streamable-HTTP and SSE sub-app plus
the single-segment server and toolset aliases) and exclude the /v1/mcp
management API. Apply the same shape to A2A, which had the identical
issue: only the /message/send invoke route bills, not /v1/a2a/discover or
the .well-known agent-card reads.
* fix(proxy): harden billable-request classification and recorder lifecycle
Exact-match Anthropic /v1/messages so OpenAI Assistants thread-message
routes no longer bill, add Google Interactions create routes, guard
recorder.record() so a broken exporter can never fail a served request,
lock lazy recorder resolution against concurrent first requests, and
disable metering on empty-string env config instead of accepting a
blank endpoint
* chore(ui): regenerate eslint metrics after staging merge
* docs(proxy): state the lower-bound billing contract in middleware comments
* fix(proxy): bill mcp-rest tool calls and bare a2a agent invokes
POST /mcp-rest/tools/call executes a tool and fires the same MCP spend
logging as the /mcp transport, and POST /a2a/{agent_id} is the JSON-RPC
invoke route whose method (message/send or message/stream) travels in
the body; both returned 2xx without being recorded
* fix(proxy): flush billable-request counts on proxy shutdown
PeriodicExportingMetricReader buffers up to one export interval of
counts; without a final flush every restart silently dropped them. The
factory registers the recorder it builds and proxy_shutdown_event pops
and flushes it, bounded by a 5s timeout so a dead collector cannot
stall shutdown
* fix(proxy): stop billing bare a2a task RPCs and close the shutdown race
POST /a2a/{agent_id} multiplexes JSON-RPC methods off the request body. Only
message/send and message/stream write a SpendLogs row; tasks/get, tasks/cancel
and the pushNotificationConfig RPCs are forwarded upstream and write none.
Classifying the bare path as billable counted those task RPCs and pushed the
metric above the dashboard's successful-request count. Since a path-only
classifier cannot read the body, the bare route no longer bills; the explicit
/message/send routes still do. Missing a bare-path invoke undercounts, which is
the only direction this metric is allowed to drift. The /mcp transport keeps
billing every method because its list path logs a SpendLogs row too.
The billing middleware also sat outside InFlightRequestsMiddleware, and it
records after the inner app returns. A request could therefore be counted as
drained while its record() had not yet run, letting proxy_shutdown_event flush
and stop the exporter underneath it. Registering it before the in-flight
tracker nests it inside, so wait_for_drain covers the record
* test(proxy): stub the OTLP exporter in the recorder-build test
test_premium_with_full_config_builds_recorder built a real MeterProvider, so
the shutdown flush resolved collector.example and opened a TLS connection from
a unit test. The exporter is now stubbed, and a getaddrinfo spy asserts nothing
resolves the collector host so the stub cannot be quietly dropped later
* fix(helm): truncate the helm.sh/chart label to 63 bytes
Kubernetes caps a label value at 63 bytes and .Chart.Version is unbounded. CI
publishes branch builds as 0.0.0-branch-<branch>-<sha>, so helm.sh/chart
rendered as a 64 byte value and the API server rejected every labeled resource
with "must be no more than 63 bytes", including the migrations Job. The
litellm-helm chart already guards this through a litellm.chart helper; this
adds the same helper here.
Swept the rest of the chart for label and name values built from unbounded
input. .Chart.Version appeared only in this label. The remaining candidates all
derive from .Release.Name, which helm itself caps at 53 characters, so they
cannot overflow; three of them are selector labels feeding immutable Deployment
matchLabels, where adding trunc would risk churn for no gain. They are left
alone deliberately.
Verified with a new helm-unittest suite, tests/chart_label_tests.yaml, which
overrides chart.version per test:
helm unittest -f 'tests/*.yaml' helm/litellm # 13 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
The truncation cases fail against the previous helper. Reproduced the original
overflow by rendering with the real branch version and measuring the label:
helm template rel helm/litellm -f helm/litellm/tests/values/required.yaml \
| grep helm.sh/chart # 64 bytes before, 63 after
* feat(proxy): accept inline PEM for the billing-metrics mTLS credentials
LITELLM_BILLING_METRICS_CLIENT_CERT, _CLIENT_KEY and _CA_CERT took a filesystem
path. ECS injects Secrets Manager values as environment content and cannot mount
them as files, so a licensed deployment there could not turn metering on.
Each variable now takes either a path or the PEM itself. Inline PEM, detected by
the "-----BEGIN" prefix, is written once when the recorder is built into a 0700
temp dir as a 0600 file, and the config points at that path. The OTLP exporter
still only ever sees paths. A write failure disables metering through the
existing failure-as-None path rather than raising, and path-valued variables are
passed through untouched, so nothing changes for deployments that mount files.
The mixed case works too: mount the CA, inject the client credentials
* feat(helm): add first-class billingMetrics values to the componentized chart
Turning enterprise billable-request metering on meant hand-rolling the env vars
and the cert volume through gateway.extraEnv and gateway.volumes. This adds a
top-level billingMetrics block, off by default, consumed only by the gateway
since that is the component serving billable traffic.
When enabled it renders LITELLM_BILLING_METRICS_ENDPOINT plus the two cert paths
and mounts secretName read-only at /etc/litellm/billing-mtls. caSecretName is
optional and only needed for private collectors whose server certificate is not
on the public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and
adds the CA env var. exportIntervalMs is passed through only when set.
Enabling without secretName or with an empty endpoint fails the render with a
named message rather than producing a gateway that silently never exports.
The generic gateway.volumes, gateway.volumeMounts and gateway.extraEnv paths are
untouched and still compose with this, so existing overlays keep working.
The chart has no values.schema.json and no README, so there is nothing further to
update. Verified with a new helm-unittest suite:
helm unittest -f 'tests/*.yaml' helm/litellm # 23 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 54 passed
* feat(terraform): billing-metrics variables for the aws and gcp templates
* feat(helm): add billingMetrics values to the classic chart
The componentized chart just gained a first-class billingMetrics block; this
mirrors it in litellm-helm so enabling enterprise billable-request metering no
longer means hand-rolling the env vars and the cert volume through envVars and
volumes.
When enabled the proxy Deployment renders LITELLM_BILLING_METRICS_ENDPOINT plus
the two cert paths, and mounts secretName read-only at /etc/litellm/billing-mtls.
secretName defaults to litellm-billing-metrics-mtls, the conventional name, so
enabling the block is enough once that Secret exists. caSecretName is optional
and only needed for private collectors whose server certificate is not on the
public web PKI; when set it mounts at /etc/litellm/billing-mtls-ca and adds the
CA env var. exportIntervalMs is passed through only when set.
The env entries render after envVars and extraEnvVars, so a user-supplied
LITELLM_BILLING_METRICS_ENDPOINT cannot silently redirect the export under
Kubernetes last-wins duplicate-env semantics; this is the same ordering the
migrations Job relies on for DISABLE_SCHEMA_UPDATE.
Enabling with an emptied secretName or endpoint fails the render with a named
message rather than producing a proxy that silently never exports.
The generic volumes, volumeMounts, envVars and extraEnvVars paths are untouched
and still compose with this, so existing overlays keep working. The chart has no
values.schema.json; README parameters and a setup section are updated.
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 68 passed (54 + 14 new)
helm lint helm/litellm-helm # 0 failed
* test(helm): pin that the migrations job never mounts the billing cert
The componentized chart's suite asserts the backend Deployment stays clear of the
billing wiring, since only the gateway serves billable traffic. The classic chart
has no backend, but it does have a second pod: the migrations Job, which renders
its own env from envVars and extraEnvVars. Nothing today wires the billing
include into it, and nothing stopped a future edit from doing so.
Asserts absence of the env, and that the Job grows no volumes or volumeMounts at
all. Both are notExists rather than notContains because the Job renders neither
key by default, so a notContains would fail on an unknown path instead of
checking the absence it looks like it is checking.
* fix(helm): meter the backend too, it serves the MCP transport
Scoping billingMetrics to the gateway was wrong. Applying each component's own
route allowlist to the proxy app shows the split is 75 billable routes on the
gateway and one on the backend: /{mcp_server_name}/mcp, the named-server MCP
transport, which writes a SpendLogs row on success. Metering only the gateway
would have silently dropped every MCP transport call from the counter, an
undercount proportional to a customer's MCP traffic.
The backend deployment now renders the same env and mounts the same read-only
cert secret. The migrations job still gets neither; it runs prisma and serves no
traffic, and a test pins that.
helm unittest -f 'tests/*.yaml' helm/litellm # 25 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
This also aligns the chart with the terraform templates, which inject the
credentials into both components.
* fix(proxy): never log billing credential values when they fail to resolve
Accepting inline PEM turned the cert env vars into secret-bearing values, but
the disable warning still echoed them. A value that is neither a readable path
nor `-----BEGIN`-prefixed PEM, for example a key with a preamble or a malformed
secret, fell through to the path branch and was written to the proxy logs
verbatim, exposing the client certificate or private key to anyone who can read
them.
The warning now names the offending environment variables and tells the operator
what a valid value looks like, without ever printing one
* Revert "fix(helm): truncate the helm.sh/chart label to 63 bytes"
This reverts commit 4f7f706a63.
Version hygiene belongs to the pipeline that mints chart versions, not to the
chart. The build workflow now caps the version slug so litellm-<version> fits
the 63 byte label budget, which removes the overflow at the source rather than
silently truncating a value operators use to identify the build.
Drops the litellm.chart helper, restores the direct helm.sh/chart printf, and
removes tests/chart_label_tests.yaml. Both chart suites stay green:
helm unittest -f 'tests/*.yaml' helm/litellm # 20 passed
helm unittest -f 'tests/*.yaml' helm/litellm-helm # 69 passed
* feat(helm): default billingMetrics.secretName to the conventional name
The componentized chart required an explicit secretName while the classic chart
defaults to litellm-billing-metrics-mtls. Both now default to it, so the common
path is to create that Secret with tls.crt and tls.key and set enabled: true.
The required() guard stays, and with a default it now only fires when someone
explicitly blanks the override, which the tests pin from both sides
* feat(proxy): log once when billing metrics are actually enabled
build_billing_metrics_recorder returned None silently when the deployment was
not licensed, while every other disable path logged a warning. An operator
reading logs could not tell "metering active" from "metering off because this
component never saw the license", and a component can carry the cert mount and
the billing env and still meter nothing. That is the undercount direction the
metric is not allowed to drift in.
A successful build now emits one info line naming the collector endpoint and the
export interval; neither the certificate contents nor the license appear. The
unlicensed path logs at debug rather than warning, because unlicensed is the
common case and a warning there would be noise on every OSS proxy
* fix(terraform): fail the plan on a partial billing-metrics config
Each PEM secret is created only when its own variable is non-empty, so setting
billing_metrics_endpoint with a certificate but no key applied cleanly and left
the proxy logging "missing config" and never exporting. Silent non-export is the
undercount direction this metric must not drift in, and every other surface
fails fast on a half-configured metering block.
Both templates now carry a lifecycle precondition requiring the client
certificate and its key together whenever the endpoint is set. It lives on the
gateway task definition (aws) and the gateway Cloud Run service (gcp) rather
than on the secret resources, because those are themselves count-gated on the
PEM being present and would never evaluate in the failing case. Cross-variable
`validation` blocks would need terraform 1.9; versions.tf pins >= 1.6, and
preconditions work there.
ca_cert_pem stays optional, so an empty value still falls back to the system
trust store.
endpoint cert key result
"" any any metering off, no secrets created
set set set metering on
set missing either plan fails
Verified each row with `terraform console` against the condition, and reran
`terraform fmt -check` and `terraform validate` in both directories
* docs(terraform): record why the billing guard sits on the gateway resource
The precondition cannot live on the cert secret, which is count-gated on
the cert itself and so has zero instances in exactly the case the guard
must catch. That makes the guard's correctness depend on this resource
staying unconditional, which nothing else records and no test enforces
* fix(terraform): guard the backend against a partial billing config too
The precondition only sat on the gateway, but the backend receives the billing
endpoint as well, because it serves the named-server MCP transport and meters
it. A targeted apply of just the backend task or service would therefore skip
the guard entirely and provision a component holding a billing endpoint with no
credentials to use it, which is the silent never-export failure the guard exists
to prevent.
Both templates now carry the same precondition on the backend resource. The
condition and truth table are unchanged; ca_cert_pem stays optional.
terraform fmt -check and terraform validate clean in both directories
* docs(team): document mcp_rpm_limit in update_team docstring
* chore(ui): regenerate schema.d.ts for update_team docstring change
A whitespace-only authorization_url was truthy to the row/config merges and
has_all check but blank to the corroboration gate, so discovery and
carry-forward adopted token_url/registration_url/scopes as if unpinned while
the broken whitespace value was still used for redirects. Rather than add
another strip() at each site, the pinned authorization_url/token_url/
registration_url are normalized once per build path (DB and config) via
_blank_to_none, so the merge, has_all gate, discovery gate, persist hook, and
carry-forward all see a single notion of blank. Empty and whitespace pins now
behave identically to an omitted field.
Provenance is a property of the whole discovered metadata document, not per
field. Waving scopes through while gating endpoints left a second inflation
vector: a compromised upstream advertises broad scopes via the resource
metadata (RFC 9728 / WWW-Authenticate), the gateway requests them from the
trusted authorization server, and the resulting token flows back to the
upstream. Both that and the token-endpoint mix-up are now one rule: when
authorization_url is admin-pinned, discovered token_url/registration_url are
kept only if the document corroborates the pin, and scopes come from the
authorization server's own scopes_supported (a new authorization_server_scopes
field, trusted tier) rather than the resource-advertised scopes. A document
that does not corroborate backfills nothing. Blank (empty-string)
authorization_url is treated as unpinned so the merge and the gate agree.
Carry-forward, the other non-manual source, drops the same three across an
authorization_url change.
Register bedrock_mantle/openai.gpt-5.6-{sol,terra,luna} with
mode=responses, /v1/responses in supported_endpoints, and
use_openai_responses_path so the data-driven gate routes them through
BedrockMantleResponsesAPIConfig on the openai/v1 Mantle base path.
Without these entries the models fall through to chat-completions
emulation, which the Mantle endpoint rejects.
Pricing and context window sourced from the AWS Bedrock pricing page
and the GPT-5.6 model cards (272K context, OpenAI first-party rates
with the 1.1x in-region US uplift, 90% cached-input discount, 1.25x
cache write).
The corroboration check belongs to adopting a token_url from any non-manual
source, not to discovery alone. Carry-forward is the other such source: it
copied a prior registry entry's token_url/registration_url onto a rebuild
whose authorization_url had been re-pointed to a different server, reviving an
uncorroborated token endpoint the discovery gate would reject. Both sites now
share one predicate, _endpoints_corroborate_authorization_url: previous
endpoints carry forward only when the previous authorization_url corroborates
the authorize endpoint the build will use (absent -> the previous one is
adopted too, a consistent group; else it must match). Endpoint comparison now
elides the default port so :443 and formatting-only differences still match.
Discovery is rooted at the MCP resource, so a compromised upstream can
advertise an attacker-run authorization server. When authorization_url is
manually configured and another field is blank, the per-field merge would
combine the trusted authorize endpoint with the advertised token_url, and
the gateway would redeem authorization codes (with the stored client secret
and PKCE verifier) at that endpoint, then persist it. Discovered token_url
and registration_url are now accepted only when the same metadata document
advertises an authorization_endpoint matching the configured value
(scheme+host+path). Scope backfill is unaffected. Applies to both the DB
and config build paths.
