* fix(responses): run the guardrails a Responses WebSocket frame selects
* fix(responses): name the field a malformed frame guardrail selection got wrong
A frame whose guardrails or guardrail_config has the wrong shape now gets
an error event naming the field, instead of pydantic's raw validation dump
* fix(responses): accept every guardrail selection shape a Responses WebSocket frame sends, and run during_call guardrails
A frame can pick guardrails as a list of names or name-to-config dicts, inside metadata or litellm_metadata, with policies validated as a list of strings. during_call guardrails now run before the turn goes upstream, and selection fields sent next to a nested response are rejected instead of dropped
* fix(proxy): keep a key's configured guardrails when policies add guardrails
A policy's guardrails were merged with dict.fromkeys, so a configured guardrail entry ({name: {extra_body}}) in key metadata raised unhashable type: 'dict' and every request naming policies failed with a 500 on HTTP and on the Responses WebSocket. The merge now keeps existing entries and skips policy guardrails already selected by name. Also types the WebSocket test guardrail's hooks.
* fix(responses): run streaming-scoped guardrails on socket turns, keep guardrail fields on blocks, merge guardrail lists by name
* fix(responses): check frame guardrail picks against the team permission, honor picks beside a nested response, prefer a guardrail entry with settings
* style(responses): format the frame guard and give the pass-through guardrail test an enterprise license
* fix(proxy): keep pass-through guardrail settings, gate frame policy picks, skip moderation-only hooks on socket turns
* fix(responses): keep string metadata guardrail values as client metadata on socket turns
* fix(responses): ask no team permission for string metadata named like a guardrail pick on socket turns
* fix(proxy): socket guardrail picks read the key's team metadata when the proxy has no database
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openrouter): list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on
* fix(openrouter): list models off the event loop and keep only chat-callable catalog ids
The model-list builders are sync, and with check_provider_endpoint on they call OpenRouter's catalog with a 10 second timeout and no caching of failures. They ran on the event loop inside /v1/models, /model/info, /model_group/info and the per-user listing, so one slow catalog froze every request on the worker. Those calls now run in a worker thread with asyncio.to_thread
File, batch and RAG credential lookups only need each name's deployment, and a wildcard that is itself a deployment sits in the list ahead of its expansions and resolves its own credentials. get_complete_model_list takes expand_deployed_wildcard_routes, which that lookup sets to False, so a deployed wildcard never fetches its provider catalog there. A key wildcard wider than any deployment, like anthropic/* over a deployed anthropic/claude-*, still expands so it finds the matching deployment's credentials. Every other caller keeps the default
The pattern router's model lookup tolerates a pattern removed between its two reads of the pattern table, which a deployment delete on the event loop can now do while a listing runs in a thread
Listing now reads only OpenRouter's models/user catalog. OpenRouter's own /models list carries no embedding models, the OpenAI-format /v1/models has no mode field so chat pickers cannot tell embeddings apart, /model_group/info labels them mode chat, and one failing catalog emptied the whole list. Ids ending in :batch are dropped because chat completions answers 404 for them, while :free and every other variant stay. An api_base that already ends in /chat/completions, which chat accepts as is, has that suffix stripped before the catalog route is joined
* fix(proxy): keep model listing off the shared thread pool and out of credential lookups
Listings run on their own thread pool so many slow catalogs cannot starve
the default executor that chat and Responses calls share. File and batch
credential lookups match deployed names against a key's wildcard grants
instead of listing any provider catalog. The token counter keeps OpenRouter
deployments on their previous path, and a catch-all "*" over provider/*
lists bare ids that route back through it
* test(proxy): give the off-loop listing tests a generous deadlock guard and typed ids
* fix(proxy): keep routed models in wildcard model info, skip key-only OpenRouter catalogs, cache failed listings briefly
* fix(utils): cache failed model listings under their own key so a late failure never hides a good listing
* fix(proxy): remember only timed-out provider listings so a provider that failed fast is listed again
* fix(proxy): ask each provider catalog once per worker, keep deployed names out of catch-all ids, and match file credentials by name
Listings that start together for one deployment now wait for the first catalog request. A catch-all wildcard no longer lists a name a deployment serves by name. A key wildcard only finds file and batch credentials on deployments named under it, so a bare name keeps the default credentials as before.
* fix(proxy): keep the listing failure log line the same for timed-out listings
* refactor(proxy): fold the cached listing lookup back into the provider listing function
* fix(proxy): list each model id once and keep a named deployment's row over a wildcard's
* fix(proxy): drop a wildcard's model info row when named deployments serve every name it lists
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): name shared-handler errors by the status they answer
handle_exception_on_proxy pinned type=internal_server_error on every
exception it wrapped, so a management route rejecting a caller's own
argument answered {"type":"internal_server_error","code":"400"} and an
SDK branching on the type read its own bad request as a gateway outage.
Derive the type from the status the response actually carries, keeping
a type the exception already names.
* test(proxy): pin a 500 for exceptions whose status is None
* fix(proxy): keep internal_server_error for timeouts and 5xx, redact hosts in shared handler errors
Provider 5xx and timeouts keep internal_server_error so retry loops still retry them,
the shared handler redacts internal hosts from raw exception messages, and /key/list
routes its HTTPException through the shared handler instead of its own copy of the
old label.
* fix(proxy): label shared-handler errors like the chat route, pick the status before the type, redact only gateway failures
* fix(proxy): pass shared-handler error messages through as on main
Redacting timeout and 5xx messages changed what every shared-handler route
returns, which relabeling the error type does not need, so messages go back
to the verbatim text main sends.
* fix(proxy): redact only the status-less failure main never sent
* fix(proxy): answer a generic 500 for a status-less failure and expect auth_error on the /user/new 403
A failure with no status now answers "Internal server error" instead of its own text, since that text can
name a guardrail's private host and pattern redaction misses cluster DNS names and IPv6 addresses. The
/user/new integration test now expects the auth_error type main's route catch-alls give a 403 the proxy raises
* fix(proxy): answer only a 4xx or 5xx carried status and fall back to the route default otherwise
A numeric-string status outside 400 to 599 made /v1/messages, images, rerank and the OpenAI pass-through answer HTTP 200 with an error body or drop the connection, and non-ASCII digits or a 5000-digit string slipped through isdecimal. error_status_code now accepts an int or a three-digit ASCII string in 400 to 599, the range the chat route already enforces
* refactor(proxy): parse a three-digit string status without recursing
Code outside the router now reaches it only through the entry points
(registry, settings, health, select, apply, observe, register_plugin, retire)
or the endpoint methods; the outside-member allowlist shrinks from 139 to 45.
litellm/router.py no longer imports litellm.proxy: it asks the installed
RouterHost (litellm/router_host.py) for vertex pass-through credentials,
member auto-router authorization and auto-router prompt compression, and the
proxy installs its host on import. Call sites where a model name can be
missing keep the old answer instead of reaching an entry point with None.
* fix(proxy): list models from team access groups
* 修复 PR #32599 的类型检查并补齐模型列表回退测试
* fix(proxy): keep a key's own model list narrow when adding team access group models
A key with its own model list was shown every model from its team's access
groups on /v1/models, /v1/model/info and /model_group/info, though auth
denies those calls. Team access group models now widen a key's list only when
the key inherits the team's models. /v1/model/info reuses the shared listing
helper without a catch-all, and route-level tests replace the mock-only ones
* fix(proxy): skip the list build in the unrestricted model info check and test every listing route
The unrestricted check for /v1/model/info only needs to know whether the key
and team lists are empty, so it no longer builds and expands the model list a
second time. The route test now covers /v1/models and /model_group/info next
to both model info routes
* fix(proxy): scope model info for personal and agent keys with no model list
* fix(proxy): keep config keys' team access groups and own-team team_id scope in model listing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): use budget reset window for projected spend alerts
* style(proxy): use PEP 585 tuple annotations in projection helpers
* fix(proxy): derive projection date from budget reset timezone
* fix(proxy): project spend in real time within the budget reset window
* fix(proxy): derive the projection window start from the reset schedule
30d and monthly keys reset on the 1st of the month and unrecognized spellings like 1hr reset at the next midnight, but the window start stepped back the literal duration, so now could land before the window and the elapsed floor inflated the projection into a false alert. The window start now mirrors get_next_standardized_reset_time, and a reset more than one window away no longer projects at all
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* refactor(proxy): expose public names for private proxy helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep original class names behind public aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve internal callback filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep _PROXY_ class names for managed files hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names in package exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve recursive auth helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align MCP limiter tests with server enforcement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): match main's MCP limiter tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names bound in importing modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve compatibility imports through strict lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use exact pyright suppression in password helper test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): add reasons to compatibility import noqa comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update IN-list baseline for renamed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move utils, agent_endpoints and endpoint tests into tests/unit/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): package moved unit test directories
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): exclude proxy-db-owned files from the misc target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop the redundant fixture docstrings in the proxy conftest
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
