Profiling the sidecar-enabled gateway at 700 rps showed ~2.4% of all samples
in get_model_group_info called per request from budget reservation, plus
get_deployment_model_info for tiered pricing tables. Both are read-only lookups
over the model list, so serve them from the Router's lru caches and clear the
deployment cache alongside the group cache when the model list changes.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Inference workers publish one compact typed SpendEvent per success over a
unix socket or loopback TCP; an opt-in sidecar (python -m gateway.spend_worker)
runs the unchanged _ProxyDBLogger cost pipeline against the pod's pgbouncer.
Default off (LITELLM_SPEND_WORKER_ENABLED). Also reuses the preset cache key
in get_logging_payload instead of hashing the request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The FK cascade drops the LiteLLM_JWTKeyMapping row, but the cached
jwt_key_mapping:{claim}:{value} entry still resolved to the deleted token
hash, so every JWT call from that identity failed until
virtual_key_mapping_cache_ttl expired instead of auto-registering against a
recreated key. delete_verification_tokens now snapshots the mapping cache
keys before the delete and evicts them across replicas afterwards, the same
way /key/regenerate already does.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
A team-scoped auto-router is stored under an internal
model_name_{team_id}_{uuid} with the caller-facing name in
model_info.team_public_model_name, and the four pre-routing strategy
registries key on that internal name. A team key asks for the public name,
so the strategy lookup missed, the team early-resolve exit handed back the
marker deployment itself, and every call 400'd with "Unmapped LLM provider".
The strategy lookup now resolves the requested name through the same
team-first, then global, then admin-across-teams deployment resolution the
deployment path uses, and looks the registries up under the model_name of
whatever that resolves to. Both exits of _common_checks_available_deployment
drop strategy markers through one helper, so a marker-only resolution is
rejected as uncallable on every path. The request team id has one reader.
Resolves LIT-7363
Claude-Session: https://claude.ai/code/session_01NU97S7d2FUDDvTk59k53Wp
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(proxy): accept non-string callback vars in default_team_settings
A YAML boolean such as turn_off_message_logging: true in a
default_team_settings block failed TeamCallbackMetadata's str-only
callback_vars validation and errored the request before any callback
ran. Stringify the value the same way AddTeamCallback does.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop docstring from default_team_settings bool regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move default_team_settings bool regression test to mapped pre_call_utils suite
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Rust token counter now mirrors _count_input_tokens key precedence
(messages, prompt, input, query/documents) so every LLM route that goes
through budget reservation gets the GIL-free count, not only /v1/messages
and /v1/chat/completions. Objects are serialised like json.dumps before
tokenizing; floats and unknown shapes still decline to Python. The body
model is optional so route-selected models can be matched by the caller
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep a body litellm_session_id in SpendLogs under missing_session_id omit
Under general_settings.missing_session_id: omit, apply_missing_session_id_policy now
mirrors a client-supplied top-level litellm_session_id into metadata.session_id when the
client did not set one there, so SpendLogs.session_id and Langfuse agree with the session
callbacks already report through StandardLoggingPayload.session_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep client metadata.session_id ahead of body litellm_session_id on litellm_metadata routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop docstrings from the missing_session_id omit regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The two test methods, the policy_engine fixture, and the two inner
stubs in TestBackgroundResponseRetrievalGovernance now carry full
parameter and return annotations, closing the Greptile thread that
94f9230d13 left open.
Drop the gateway-side admission layer (identity cache, /key/info lookups,
process-local budget/TPM/RPM limits) and keep only the CPU work in Rust: a
typed litellm-core token counter that parses the raw body once and counts
input tokens with the GIL released, exposed as TokenCounter in the PyO3
bridge. The Python proxy's existing auth dependency passes the raw body
into budget reservation, which uses the Rust count for models on the
Anthropic tokenizer and falls back to Python for anything Rust declines,
when the bridge is disabled, or when the native module is unavailable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Each proxy worker flushed one Prisma upsert per active (date, category, route)
bucket every interval, so the Postgres primary saw workers x routes statements
per interval across the deployment. A flush now builds a single multi-row
INSERT ... ON CONFLICT DO UPDATE, and with use_redis_transaction_buffer on the
workers push snapshots to a Redis list that one lease-holding pod folds and
commits, so the whole deployment costs one statement per interval. The leader
keeps popping until the list is empty so a deployment wider than the dequeue
cap cannot build a backlog, and rows that fail both the commit and the Redis
re-queue fall back to the leader's own accumulator instead of being lost.
Resolves LIT-7371
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): pipeline spend counter increments into one redis call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply surviving spend increments before raising scope error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format spend counter helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): settle inner spend counter gathers and fall back per key on pipeline failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): suppress BLE001 on pipeline fallback catch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): invalidate all batched spend counters on pipeline failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The estimate looked the reported per-token rates up a second time, with the
provider this endpoint resolved rather than the one completion_cost infers.
The provider decides whether a token tier threshold is inclusive, so an
unrouted xai model sitting exactly on 200k billed at the tier rate and
reported the base rate, half of it.
completion_cost now hands back the rates its own lines were billed at, and
the endpoint reports those.
Claude-Session: https://claude.ai/code/session_01RLKy5DMi3XCBUJ37WzfNi1
* feat(guardrails): map each guardrail scan id to its guardrail, stage and provider
Adds the x-litellm-guardrail-scan-metadata response header, a JSON list of
{guardrail, stage, provider, scan_id} entries, next to the existing
comma-separated x-litellm-guardrail-scan-id header. Prisma AIRS records the
execution stage for every scan and OpenAI Moderation now records its
moderation id too. The new metadata key is internal: client-supplied values
are stripped and it is exposed through the UI CORS allow list.
Resolves LIT-6018
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): cap the scan metadata response header at a configurable length
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): hardcode the scan metadata header cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model-management): honor an explicit null as a clear on model update
PATCH /model/{model_id}/update merged the patch with exclude_none and then
popped explicit nulls only for the mirrored pricing fields, so a null sent for
max_input_tokens, mode, supports_vision or any other key was dropped and a value
pinned by an earlier save could never be removed.
The route now follows JSON Merge Patch over both blobs: a key absent from the
body is unchanged, a key sent as null is removed from the stored row, and a key
sent with a value is set. Ownership and identity keys keep ignoring a null, as
do the fields the stored models require, since clearing one writes a row no
reload can rebuild. Mirrored pricing keys still clear from both blobs.
Clearing a price also needed the router to stop merging a deployment's cost-map
entry onto its previous registration, which left the old rate in place and kept
billing at a price the deployment no longer carried.
Adds a create, read, partial-update, clear, enforce, delete lifecycle e2e that
reads back on every replica, and a harness helper for that read-back.
* fix(router): keep a deployment id that names a real model from evicting its catalog entry
Deployments are keyed into litellm.model_cost alongside the built-in catalog, so
evicting a deployment's stale entry by id could take a real model's entry with it:
registering a deployment whose model_info.id is "gpt-4o" stripped that model's
pricing, context window and capability flags process-wide, for every other
deployment of it, until the next price-map reload.
Only evict an entry this registration owns. A colliding id keeps the previous
merge, which pollutes the catalog entry rather than emptying it.
Also pins the Admin UI round trip: the model edit form echoes the whole /model/info
row back on save, and that read reports every key the deployment never stored as an
explicit null, so the clear path has to leave those keys alone.
* fix(router): decide cost-map eviction by what this registrar created
The previous guard read a catalog entry off `litellm_provider`, so a deployment
that declares its own provider in model_info was treated as one and kept billing
at a price it no longer carried. It also only held for a single registration: a
second one under a colliding id saw the id the first merge left behind and
evicted the catalog entry anyway.
Track the cost-map keys this registrar creates instead. A key it created is
evicted before re-registration; one it did not is left to merge, which is what a
deployment id colliding with a catalog model name needs.
Also folds the required-fields comment into the docstring that already gives the
reason.
* fix(router): release a deployment's cost-map key when it is deleted
The ownership ledger only grew. A deleted deployment kept its claim, so if a
later catalog refresh started publishing a model under that same name, the next
registration would treat the catalog entry as the deployment's own and evict it.
Deleting a deployment now gives the key back, which also stops the ledger
growing for the life of the process.
* fix(router): hold a cost-map key while another live router still serves it
The claim is process-wide but the release was per-deletion, so with two routers
serving one deployment id, the first deletion put the survivor back on merging
and the price it had just cleared would keep billing.
Release the key only once no live router still serves that id.
* fix(router): register a router in the live set when it gains a deployment
_live_routers was only joined when a router was constructed with a model_list,
but a router built empty is populated through add_deployment, and the empty
branch exists for exactly that. Such a router was invisible to the live-router
scan, so deleting the deployment from another router released the shared
cost-map key while it was still serving that id.
Joining the set where a deployment enters the list covers every path, and it
also lets a price reload rebuild what a dynamically built router serves.
* fix(e2e): read the stored model row from the control plane, not each gateway
The lifecycle suite polled /model/info on every URL in PROXY_REPLICA_URLS. Those
URLs are the stack's gateways, and gateway/routes/allowlist.py trims them to the
LLM data-plane surface, so /model/info answers only on the backend and 404s on
every replica. All five tests failed at their first read-back in CI while passing
against a monolith, where one process serves both planes.
The stored row has one answer behind it, so it is read through the shared
transport, which routes control-plane paths to the backend. What every gateway
must agree on is which models it serves, so the create and delete steps poll
/v1/models per replica instead, a route the gateway does serve.
read_back_everywhere now rejects a control-plane path outright rather than
timing out on it.
Two things surfaced behind that. /public/ was missing from the transport's
control-plane prefixes, so model_cost_map() was routed to a gateway and 404'd,
and the billing steps needed a data-plane wait: a PATCH lands on the backend and
each gateway picks it up on its own config reload, measured here at 12-24s, so
they now drive calls until the new rate reaches the spend row and let the
deadline fail them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1S92J8gSxxKVe1JBzxWBF
* test(models): keep polling outcomes immutable and document shared ownership
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e
PUT /v1/mcp/toolset dumped its payload with exclude_none, so a field sent as
null looked exactly like one the caller left out and the stored value
survived. An admin could not clear a toolset's description: the save reported
success and the old text came straight back. It now dumps with exclude_unset,
so absent keeps and null clears, which is what PUT /v1/mcp/server already did.
A null tools list clears the selection to empty, and a null toolset_name is
ignored because a toolset always has a name.
Adds create, read, partial-update, clear and delete e2e coverage for MCP
servers and toolsets, with every read-back polled on every replica so an edit
that lands on one replica and not another fails the test, plus an enforcement
test proving a key granted a toolset lists exactly that toolset's tools
against the real Datadog upstream.
* fix(e2e): refuse a read-back that no replica serves
A read-back over an empty replica mapping satisfied every predicate and
returned as if it had converged, so it would have asserted nothing and
passed. No wiring can produce that today, since the replica list always
falls back to at least one URL, but a helper whose whole job is proving a
write reached every replica should not have a shape that passes vacuously.
* fix(mcp): keep a null tools list a no-op on toolset update
Treating a null tools list as a clear meant an existing client that sends
tools=null during a partial update, meaning "leave the selection alone",
silently lost every tool the toolset grants. That is a permission surface,
so the quiet version of it is the worst version.
A toolset always has a tool list, the same way it always has a name, so a
null on either is now a no-op. Emptying the selection is an explicit [],
which cannot be confused with a field the caller left out, and which is
what the dashboard already sends.
* fix(e2e): keep MCP admin routes on the data plane
/v1/mcp/* is a lazily mounted feature, so a gateway registers it on the first
matching request, which happens after the startup route trim that drops
management endpoints. Routing it to the control plane therefore sent every MCP
call to the one backend process: the new lifecycle read-backs proved a single
process rather than every replica, and mcp_client's await_registered barrier
waited on a registry that does not serve the tools/list call it guards, so the
existing MCP suites polled a gateway that had not synced yet until poll_timeout
Verified against a two-gateway split stack (backend on 4001, gateways on 4010
and 4011, one postgres): both gateways answer /v1/mcp/server and /v1/mcp/toolset,
and each served 6 server reads and 7 toolset reads over the run
* fix(e2e): grant the toolset by the tool's own name, not the wire name
tools/list serves a tool as <prefix><tool_name>, but a toolset grants by the
tool's own name: resolve_toolset_permissions reads toolset.tools[].tool_name
straight through, and the prefix is added on the way out. The test built the
toolset from the names tools/list reported, so the grant matched nothing, the
scoped key listed no tools, and await_tools ran out its whole poll_timeout
before failing
Measure the prefix off search_datadog_logs, whose own name is known, rather than
guessing it from the alias, since the proxy can be configured to prefix with a
short server id instead. The expectation compared against tools/list stays in
wire names; only what the toolset stores crosses back
* test(mcp): build immutable lifecycle updates and replica results
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
* test: honor Datadog search rate-limit reset headers
* test: share the Datadog read-back deadline across retries
* test: preserve captured MCP toolset update fields
Every test this PR adds now annotates its fixture and parametrize
parameters. The submit-time warning for a tag-matched deferred policy
names only the policies, since a wildcard attachment pattern would let
caller-provided tag text reach the log.
A guardrail that removes or adds a tool call on an ended stream used to be
silently ignored: every handler substitutes the original list on a count
mismatch and the executor skipped its observer once the translation could
deliver rewrites. The executor now tracks the count change on the observer
and releases the original chunks with the discard warning on every
translation, matching what the merge base did for any tool call rewrite
* test(mcp): exercise /mcp/proxy authorization against the real registry instead of patched manager methods
* fix(mcp): preserve proxy logging and authorization coverage
* test(mcp): respect the proxy FastAPI import boundary
