Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality
sub-routers now take a RoutingContext instead of the Router, and read the router only through
registry(), settings(), health() and the context's forwarding methods. Router-free helpers move
from router_utils to litellm_core_utils, outside callers of router state switch to entry points,
and the surface allowlist shrinks to the endpoint methods plus the entry points
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* fix(guardrails): keep request steps when a response pipeline blocks before a later request pipeline
* refactor(proxy): inject gateway activity through a typed repository
* ci: keep dependency tests in their existing shard
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails (#49263)
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* refactor(proxy): inject database repositories through per-repository dependencies
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): model Prisma client cleanup
* test(rust): include NativeDatabase in bridge API surface
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): give database shutdown one lifespan owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): log database cleanup errors and bound shutdown waits
Database.aclose now logs a failure in either cleanup step instead of raising
out of the lifespan exit, which matches how main handled stop errors and keeps
setup_prisma_client from raising out of cleanup after a failed connect
Shutdown waits for an in-flight reconnect for a fixed five seconds instead of
the reconnect timeout plus grace, and joins engine wait threads with a fixed
limit instead of indefinitely. The threads are daemon threads, so a thread
still alive after the limit cannot keep the process from exiting
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Moves the page-authored English on /policies, /tool-policies and
/guardrails-monitor into react-i18next t() calls backed by en.json, and
adds a small data test for en.json. The rendered English is unchanged.
* fix(batches): cost an unmanaged gs:// batch through the deployment that created it
The create now stores the router deployment id as model_id inside the
managed object row's file_object JSON and CheckBatchCost reads it before
falling back to the first deployment matching the base model in the
gs:// path. Rows without the field keep the old lookup
* fix(batches): tolerate a non-string or malformed file_object when reading the stored deployment
* fix(batches): read the stored creating deployment from the writer, not a lagging read replica
* fix(batches): resolve the writer database once so the type gate count does not grow
* fix(batches): decode a dict-shaped file_object when reading the stored deployment, accumulate wire samples, annotate test locals
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Runs one tool-calling conversation per selected endpoint (/v1/chat/completions,
/v1/responses, /v1/messages) against a saved deployment or unsaved litellm_params
and reports conversation, prompt_caching and cost_tracking verdicts, including
whether LiteLLM_SpendLogs rows landed
* feat(sso): create the team_id_jwt_field team on UI SSO login when team_id_upsert is set
* fix(sso): keep the upserted team_id_jwt_field team out of result.team_ids so restricted_sso_group and the access-token group fallback are unchanged
* fix(sso): only upsert the team_id_jwt_field team on UI SSO login when enable_jwt_auth is on
The API JWT path only reaches team_id_upsert inside the enable_jwt_auth branch, so a deployment
with a leftover litellm_jwtauth block and JWT auth off never created teams. Gate the UI SSO upsert
the same way
* fix(sso): read enable_jwt_auth through the proxy_server module so the basedpyright gate stays at its ceiling
* fix(responses): run the guardrails a Responses WebSocket frame selects
* fix(responses): name the field a malformed frame guardrail selection got wrong
A frame whose guardrails or guardrail_config has the wrong shape now gets
an error event naming the field, instead of pydantic's raw validation dump
* fix(responses): accept every guardrail selection shape a Responses WebSocket frame sends, and run during_call guardrails
A frame can pick guardrails as a list of names or name-to-config dicts, inside metadata or litellm_metadata, with policies validated as a list of strings. during_call guardrails now run before the turn goes upstream, and selection fields sent next to a nested response are rejected instead of dropped
* fix(proxy): keep a key's configured guardrails when policies add guardrails
A policy's guardrails were merged with dict.fromkeys, so a configured guardrail entry ({name: {extra_body}}) in key metadata raised unhashable type: 'dict' and every request naming policies failed with a 500 on HTTP and on the Responses WebSocket. The merge now keeps existing entries and skips policy guardrails already selected by name. Also types the WebSocket test guardrail's hooks.
* fix(responses): run streaming-scoped guardrails on socket turns, keep guardrail fields on blocks, merge guardrail lists by name
* fix(responses): check frame guardrail picks against the team permission, honor picks beside a nested response, prefer a guardrail entry with settings
* style(responses): format the frame guard and give the pass-through guardrail test an enterprise license
* fix(proxy): keep pass-through guardrail settings, gate frame policy picks, skip moderation-only hooks on socket turns
* fix(responses): keep string metadata guardrail values as client metadata on socket turns
* fix(responses): ask no team permission for string metadata named like a guardrail pick on socket turns
* fix(proxy): socket guardrail picks read the key's team metadata when the proxy has no database
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Carry the Z.AI catalog fix from #40088 and move its regression into the mapped unit test file.
Co-authored-by: Chaitanya Laxman <chaitanyalaxman118@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover separate Meta Model API provider onboarding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): add Meta Model API provider onboarding
Co-authored-by: lostmartian <sahilgangurde08@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): align Meta coverage with provider keys and listed registry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show Meta model placeholder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: lostmartian <sahilgangurde08@gmail.com>
Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose
* fix(proxy): raise live budget counters for failure spend rows
A stream that breaks mid-flight writes its recovered cost to the spend log, but the failure hook
had already released the reservation and never touched the live counters, so an over-budget key
kept passing until the counter was next repaired from the database
Every charged failure row with a cost now raises the same counters the success path raises. The
error waits at most a second for them, so a hung Redis cannot hold it back
Fixes#35568
* test(proxy): cover failure spend counters across budgets, endpoints and outages
Integration cells for team, user, end-user, tag and window budgets,
concurrent streams, billed retries, native /v1/messages, Bedrock guardrail
cost, a Redis stall on two workers and disable_spend_updates
* test(proxy): check stalled failure counters and use the real spend writer
- The Redis stall cell now reads the burst key's live counter and requires at least the full failure spend
- Unit tests drop the charging writer double, bare dicts, list mutation and wall-clock timestamps
- Partial-stream test models are frozen
* test(proxy): move the stalled counter bound reason into the assert message
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): apply key and team router_settings to /v1/images/generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format router settings helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve key model_group_alias once on image generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openrouter): list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on
* fix(openrouter): list models off the event loop and keep only chat-callable catalog ids
The model-list builders are sync, and with check_provider_endpoint on they call OpenRouter's catalog with a 10 second timeout and no caching of failures. They ran on the event loop inside /v1/models, /model/info, /model_group/info and the per-user listing, so one slow catalog froze every request on the worker. Those calls now run in a worker thread with asyncio.to_thread
File, batch and RAG credential lookups only need each name's deployment, and a wildcard that is itself a deployment sits in the list ahead of its expansions and resolves its own credentials. get_complete_model_list takes expand_deployed_wildcard_routes, which that lookup sets to False, so a deployed wildcard never fetches its provider catalog there. A key wildcard wider than any deployment, like anthropic/* over a deployed anthropic/claude-*, still expands so it finds the matching deployment's credentials. Every other caller keeps the default
The pattern router's model lookup tolerates a pattern removed between its two reads of the pattern table, which a deployment delete on the event loop can now do while a listing runs in a thread
Listing now reads only OpenRouter's models/user catalog. OpenRouter's own /models list carries no embedding models, the OpenAI-format /v1/models has no mode field so chat pickers cannot tell embeddings apart, /model_group/info labels them mode chat, and one failing catalog emptied the whole list. Ids ending in :batch are dropped because chat completions answers 404 for them, while :free and every other variant stay. An api_base that already ends in /chat/completions, which chat accepts as is, has that suffix stripped before the catalog route is joined
* fix(proxy): keep model listing off the shared thread pool and out of credential lookups
Listings run on their own thread pool so many slow catalogs cannot starve
the default executor that chat and Responses calls share. File and batch
credential lookups match deployed names against a key's wildcard grants
instead of listing any provider catalog. The token counter keeps OpenRouter
deployments on their previous path, and a catch-all "*" over provider/*
lists bare ids that route back through it
* test(proxy): give the off-loop listing tests a generous deadlock guard and typed ids
* fix(proxy): keep routed models in wildcard model info, skip key-only OpenRouter catalogs, cache failed listings briefly
* fix(utils): cache failed model listings under their own key so a late failure never hides a good listing
* fix(proxy): remember only timed-out provider listings so a provider that failed fast is listed again
* fix(proxy): ask each provider catalog once per worker, keep deployed names out of catch-all ids, and match file credentials by name
Listings that start together for one deployment now wait for the first catalog request. A catch-all wildcard no longer lists a name a deployment serves by name. A key wildcard only finds file and batch credentials on deployments named under it, so a bare name keeps the default credentials as before.
* fix(proxy): keep the listing failure log line the same for timed-out listings
* refactor(proxy): fold the cached listing lookup back into the provider listing function
* fix(proxy): list each model id once and keep a named deployment's row over a wildcard's
* fix(proxy): drop a wildcard's model info row when named deployments serve every name it lists
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
