* fix(a2a): bill completion-bridge agents at their own prices on /a2a send and stream
message/send on an agent served through the completion bridge (custom_llm_provider
agents such as LangGraph or databricks_agent) logged a $0, 0-token row, and
message/stream logged no row at all, so cost_per_query and per-token prices never
reached spend.
The bridge send path now records counted usage, the agent's pricing keys, and the
agent id on its logging object, the same way URL agents do. The bridge stream is
wrapped in the A2A streaming logger, which now reads the bridge's artifact event
for the reply text and gets the agent name from the proxy. Streamed calls also
count prompt tokens again, since the message text reader now reads pydantic
messages as dicts.
* fix(a2a): count chunked and Task replies once and skip agent prices on error replies
Bridge streams now follow the A2A append flag per artifact, so chunked
pydantic-ai, watsonx, and AgentCore replies count the whole reply once.
Send replies that are a Task count their artifact text. A JSON-RPC error
reply is logged at cost 0 instead of the agent's price.
* fix(a2a): keep one reply text per stream and bill every artifact and status reply
* fix(a2a): ignore a null error field on streams and run the stream per-token billing test
A stream event whose error field is null no longer zeroes the call's cost. The integration test for per-token stream billing now runs instead of skipping, and the URL agent SDK case skips when the optional a2a-sdk is not installed.
* fix(a2a): bill streams the client drops and skip the bridge's working placeholder
A stream the client closes or cancels mid-reply now logs its spend row at the agent's prices on both the bridge and URL paths, and a stream dropped before the agent sends anything still logs nothing. The bridge's "Processing request..." status no longer counts as reply text, so an empty bridge reply bills no completion tokens, as send does. Adds proxy-level tests that a bridge stream bills under the agent's card name.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality
sub-routers now take a RoutingContext instead of the Router, and read the router only through
registry(), settings(), health() and the context's forwarding methods. Router-free helpers move
from router_utils to litellm_core_utils, outside callers of router state switch to entry points,
and the surface allowlist shrinks to the endpoint methods plus the entry points
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* fix(guardrails): keep request steps when a response pipeline blocks before a later request pipeline
* refactor(proxy): inject gateway activity through a typed repository
* ci: keep dependency tests in their existing shard
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails (#49263)
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* refactor(proxy): inject database repositories through per-repository dependencies
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): model Prisma client cleanup
* test(rust): include NativeDatabase in bridge API surface
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): give database shutdown one lifespan owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): log database cleanup errors and bound shutdown waits
Database.aclose now logs a failure in either cleanup step instead of raising
out of the lifespan exit, which matches how main handled stop errors and keeps
setup_prisma_client from raising out of cleanup after a failed connect
Shutdown waits for an in-flight reconnect for a fixed five seconds instead of
the reconnect timeout plus grace, and joins engine wait threads with a fixed
limit instead of indefinitely. The threads are daemon threads, so a thread
still alive after the limit cannot keep the process from exiting
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(batches): cost an unmanaged gs:// batch through the deployment that created it
The create now stores the router deployment id as model_id inside the
managed object row's file_object JSON and CheckBatchCost reads it before
falling back to the first deployment matching the base model in the
gs:// path. Rows without the field keep the old lookup
* fix(batches): tolerate a non-string or malformed file_object when reading the stored deployment
* fix(batches): read the stored creating deployment from the writer, not a lagging read replica
* fix(batches): resolve the writer database once so the type gate count does not grow
* fix(batches): decode a dict-shaped file_object when reading the stored deployment, accumulate wire samples, annotate test locals
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sso): create the team_id_jwt_field team on UI SSO login when team_id_upsert is set
* fix(sso): keep the upserted team_id_jwt_field team out of result.team_ids so restricted_sso_group and the access-token group fallback are unchanged
* fix(sso): only upsert the team_id_jwt_field team on UI SSO login when enable_jwt_auth is on
The API JWT path only reaches team_id_upsert inside the enable_jwt_auth branch, so a deployment
with a leftover litellm_jwtauth block and JWT auth off never created teams. Gate the UI SSO upsert
the same way
* fix(sso): read enable_jwt_auth through the proxy_server module so the basedpyright gate stays at its ceiling
* fix(responses): run the guardrails a Responses WebSocket frame selects
* fix(responses): name the field a malformed frame guardrail selection got wrong
A frame whose guardrails or guardrail_config has the wrong shape now gets
an error event naming the field, instead of pydantic's raw validation dump
* fix(responses): accept every guardrail selection shape a Responses WebSocket frame sends, and run during_call guardrails
A frame can pick guardrails as a list of names or name-to-config dicts, inside metadata or litellm_metadata, with policies validated as a list of strings. during_call guardrails now run before the turn goes upstream, and selection fields sent next to a nested response are rejected instead of dropped
* fix(proxy): keep a key's configured guardrails when policies add guardrails
A policy's guardrails were merged with dict.fromkeys, so a configured guardrail entry ({name: {extra_body}}) in key metadata raised unhashable type: 'dict' and every request naming policies failed with a 500 on HTTP and on the Responses WebSocket. The merge now keeps existing entries and skips policy guardrails already selected by name. Also types the WebSocket test guardrail's hooks.
* fix(responses): run streaming-scoped guardrails on socket turns, keep guardrail fields on blocks, merge guardrail lists by name
* fix(responses): check frame guardrail picks against the team permission, honor picks beside a nested response, prefer a guardrail entry with settings
* style(responses): format the frame guard and give the pass-through guardrail test an enterprise license
* fix(proxy): keep pass-through guardrail settings, gate frame policy picks, skip moderation-only hooks on socket turns
* fix(responses): keep string metadata guardrail values as client metadata on socket turns
* fix(responses): ask no team permission for string metadata named like a guardrail pick on socket turns
* fix(proxy): socket guardrail picks read the key's team metadata when the proxy has no database
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Carry the Z.AI catalog fix from #40088 and move its regression into the mapped unit test file.
Co-authored-by: Chaitanya Laxman <chaitanyalaxman118@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover separate Meta Model API provider onboarding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): add Meta Model API provider onboarding
Co-authored-by: lostmartian <sahilgangurde08@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): align Meta coverage with provider keys and listed registry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show Meta model placeholder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: lostmartian <sahilgangurde08@gmail.com>
* fix(proxy): raise live budget counters for failure spend rows
A stream that breaks mid-flight writes its recovered cost to the spend log, but the failure hook
had already released the reservation and never touched the live counters, so an over-budget key
kept passing until the counter was next repaired from the database
Every charged failure row with a cost now raises the same counters the success path raises. The
error waits at most a second for them, so a hung Redis cannot hold it back
Fixes#35568
* test(proxy): cover failure spend counters across budgets, endpoints and outages
Integration cells for team, user, end-user, tag and window budgets,
concurrent streams, billed retries, native /v1/messages, Bedrock guardrail
cost, a Redis stall on two workers and disable_spend_updates
* test(proxy): check stalled failure counters and use the real spend writer
- The Redis stall cell now reads the burst key's live counter and requires at least the full failure spend
- Unit tests drop the charging writer double, bare dicts, list mutation and wall-clock timestamps
- Partial-stream test models are frozen
* test(proxy): move the stalled counter bound reason into the assert message
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): apply key and team router_settings to /v1/images/generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format router settings helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve key model_group_alias once on image generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openrouter): list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on
* fix(openrouter): list models off the event loop and keep only chat-callable catalog ids
The model-list builders are sync, and with check_provider_endpoint on they call OpenRouter's catalog with a 10 second timeout and no caching of failures. They ran on the event loop inside /v1/models, /model/info, /model_group/info and the per-user listing, so one slow catalog froze every request on the worker. Those calls now run in a worker thread with asyncio.to_thread
File, batch and RAG credential lookups only need each name's deployment, and a wildcard that is itself a deployment sits in the list ahead of its expansions and resolves its own credentials. get_complete_model_list takes expand_deployed_wildcard_routes, which that lookup sets to False, so a deployed wildcard never fetches its provider catalog there. A key wildcard wider than any deployment, like anthropic/* over a deployed anthropic/claude-*, still expands so it finds the matching deployment's credentials. Every other caller keeps the default
The pattern router's model lookup tolerates a pattern removed between its two reads of the pattern table, which a deployment delete on the event loop can now do while a listing runs in a thread
Listing now reads only OpenRouter's models/user catalog. OpenRouter's own /models list carries no embedding models, the OpenAI-format /v1/models has no mode field so chat pickers cannot tell embeddings apart, /model_group/info labels them mode chat, and one failing catalog emptied the whole list. Ids ending in :batch are dropped because chat completions answers 404 for them, while :free and every other variant stay. An api_base that already ends in /chat/completions, which chat accepts as is, has that suffix stripped before the catalog route is joined
* fix(proxy): keep model listing off the shared thread pool and out of credential lookups
Listings run on their own thread pool so many slow catalogs cannot starve
the default executor that chat and Responses calls share. File and batch
credential lookups match deployed names against a key's wildcard grants
instead of listing any provider catalog. The token counter keeps OpenRouter
deployments on their previous path, and a catch-all "*" over provider/*
lists bare ids that route back through it
* test(proxy): give the off-loop listing tests a generous deadlock guard and typed ids
* fix(proxy): keep routed models in wildcard model info, skip key-only OpenRouter catalogs, cache failed listings briefly
* fix(utils): cache failed model listings under their own key so a late failure never hides a good listing
* fix(proxy): remember only timed-out provider listings so a provider that failed fast is listed again
* fix(proxy): ask each provider catalog once per worker, keep deployed names out of catch-all ids, and match file credentials by name
Listings that start together for one deployment now wait for the first catalog request. A catch-all wildcard no longer lists a name a deployment serves by name. A key wildcard only finds file and batch credentials on deployments named under it, so a bare name keeps the default credentials as before.
* fix(proxy): keep the listing failure log line the same for timed-out listings
* refactor(proxy): fold the cached listing lookup back into the provider listing function
* fix(proxy): list each model id once and keep a named deployment's row over a wildcard's
* fix(proxy): drop a wildcard's model info row when named deployments serve every name it lists
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): persist sent model_info fields on POST /model/update
POST /model/update rebuilt litellm_params from the request but dropped the
request's model_info, so changing base_model (or mode, tier, etc.) on an
existing DB model was silently ignored. Clients that only use this endpoint,
like the Terraform provider, could never fix a deployment's base_model
without deleting and recreating it.
Persist the model_info fields the caller actually sent, on top of the stored
ones. ModelInfo defaults never overwrite stored values, server-owned, team,
PTU and pricing fields stay on their dedicated write paths, and values echoed
from a /model/info response are not pinned.
* fix(proxy): keep echoed and blank model_info values out of POST /model/update writes
Blank strings from typed clients no longer overwrite stored values. Values echoed from
/model/info (the stored base model's cost map entry and discovered limits) are not
pinned, and an echoed catalog value resets a differing stored override like PATCH does.
direct_access and access_via_team_ids stay display-only. The tests stop patching the
auth check class and give the team lookup a real row instead.
* fix(proxy): lock the model row and skip unchanged model_info on POST /model/update
* fix(proxy): treat only keyed /model/update bodies as echoes and write the stored model_info as saved
A typed model_info value that matched the cost map, or the previous base model's entry,
was dropped and could erase a stored override. Echo detection now follows PATCH, only a
body carrying the /model/info key counts, compared against the base model the row has
after the write. The locked write patches the stored JSON instead of dumping the hydrated
Deployment, so litellm_params pricing is no longer copied into model_info. Tests run the
real Router with discovery over a mock upstream
* docs(terraform): note that model_info updates now take effect and overwrite values set outside Terraform
* test: expect POST /model/update to save the cache point flag, and wrap long test lines
test_x06 pinned the old contract, where POST /model/update answered 200 and ignored model_info.
This PR makes POST save model_info the way PATCH does, so x06 now asserts what its PATCH sibling
x01 asserts, the stored flag flips to true and cache points come back on.
* fix(proxy): answer 400 when POST /model/update gets an access_groups that is not a list of strings
* fix(proxy): read discovered model info through the router registry entry point
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(keys): allow assigning a project to an unassigned key on /key/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): suppress LIT002 on project assignment error payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): format project assignment error detail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(keys): assert accepted project assignment flows into update payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): drop stale LIT002 suppressions removed by main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(keys): mock writer_db team lookup used by check_db_only path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): write project assignment only while key row is still unassigned
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): satisfy type gate on guarded project assignment write
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): default soft-budget update path to unguarded write
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): run guarded project assignment in one writer transaction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): include object permission link in assignment guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(keys): return permission-linked payload without mutating the update dict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): fix type errors and stale noqa after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): use Sequence in assignment guard model annotation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): type update fields as a mapping in the key update path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): place cast reason on the cast call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): honor explicit null max_budget when assigning a project
* test(keys): assert explicit null max_budget flows into update payload
* fix(keys): match NULL models column in guarded project assignment
* fix(keys): authorize caller before project lookup on /key/update
* fix(keys): drop explicit null object_permission before project assignment write
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): evict a deleted key from every worker's auth cache
/key/delete dropped the key object only from the in-memory cache of the worker
that served the call. Auth serves key objects cache-first, so every other
worker, and the Redis copy when enable_redis_auth_cache is on, kept
authenticating the deleted key until user_api_key_cache_ttl expired. /key/update
already evicts and broadcasts through evict_and_broadcast, delete now does the
same for the tokens it deletes
* fix(proxy): evict only the deleted rows and broadcast them in one batch
Scope the cross-worker eviction on /key/delete to the rows the caller was allowed to delete and send it through delete_cache_key_objects, so a caller-supplied name can no longer drop another team's cached entries on every worker
* refactor(proxy): drop the comment above the delete eviction call
* fix(proxy): run the usage cache eviction beside the auth eviction so a hung Redis costs one wait
* test(proxy): wait on the broadcast instead of a sleep and type the peer eviction tests
* test(proxy): wait for the peer to apply the broadcast instead of polling its cache
* test(proxy): cover hashed keys and aliases when deleting a key warmed on a peer
* fix(key_management): evict every requested key on delete, including keys whose row is already gone
A batch delete that names a live key and a key whose row was removed outside the API evicted only the live key, so the gone key kept authenticating from every worker's cache. Every requested key hash is evicted now, while raw non-key cache keys are still left alone
* test(key_management): split the gone-row eviction assertion into one comprehension per worker
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(passthrough): log the URL model and declared provider on Gemini and Vertex failures
Pass-through failures on Gemini and Vertex routes carry the model only in the
URL, so the failure spend row and the Prometheus failure counter logged an
empty model. The failure payload now falls back to the model parsed from the
URL on those routes, and the Prometheus provider label honors the provider the
request declares before inferring one from the model name
* fix(passthrough): keep model and provider on mid-stream and Vertex failure logs
Build the failure payload from the URL at both streamed relay sites so a
200 stream that dies mid-body still logs the model, derive vertex_ai for
Vertex routes that never declare a provider, and have the Vertex logging
handler reuse get_vertex_model_id_from_url instead of its own regex
* fix(passthrough): bound the Prometheus provider label and name Vertex failures like the success handler
Declared providers only reach the api_provider label when they name a
registered provider, otherwise the label falls back to inference.
Vertex failure rows now use the success handler's model naming
(vertex_ai/ prefix on rawPredict and streamRawPredict, vertex_ai/search_api
on :search) and its host-based provider detection, and the detected-stream
failure sites are covered by tests.
* fix(passthrough): name Google failures from the URL on untracked methods too
A Gemini or Vertex pass-through failure on a method the success handler does
not track (countTokens, for one) still logged model "" and no provider,
although the model sits in the URL. The failure payload now falls back to the
Google AI host for the provider and reads /models/<name> from the URL, so
countTokens 400s name their model and provider like generateContent ones do.
Non-Google hosts keep the tracked-route rule and still log nothing extra.
* fix(passthrough): trust the declared Google provider and Vertex-shaped paths on custom bases
The failure path only named the Google provider from the URL when the
method was one the success handler tracks or the host was a Google API
host, so a Gemini countTokens failure behind a custom GEMINI_API_BASE
(or a Vertex one behind a custom base) still logged an empty model. A
route that declares gemini or vertex_ai now wins outright, and a URL
whose path carries /projects/<p>/locations/<l>/ counts as Vertex on any
host, while a tracked method on a non-Google host no longer passes as
Vertex.
* fix(pass_through): stop the URL model parser at the query string so a configured key never lands in a spend row's model
* test(pass_through): run the URL-model failure tests through the real ProxyLogging with a recording callback
* fix(passthrough): scope declared provider to pass-through failures and match Google hosts only
Prometheus trusts the resolved provider as before and reads the request-declared
provider only on pass-through failures. Vertex attribution matches Google hosts,
strips operation poll suffixes, prefixes vertex_ai/ only for rawPredict, reads the
URL path only, covers Vertex Live WebSocket failures, and labels a router alias
failure with the alias the client requested
* fix(passthrough): attribute failures from the URL only on built-in Google routes and keep other routes' metric labels
A user-defined pass-through route to a Google host keeps its empty model and provider, since its URL names an upstream model rather than a LiteLLM-managed one. Prometheus failure labels read the route's provider and model_group only for gemini and vertex_ai, so Azure, OpenRouter, Bedrock and other pass-through routes keep their earlier labels. The Gemini URL parser drops the query string, Google host matching needs a dot or dash boundary, and the Vertex Claude prefix follows alt=sse like the success handler.
* fix(passthrough): log the URL model on guardrail blocks and the named model on Vertex Live failures
A pre-call guardrail block on a pass-through route now builds its failure
payload the same way upstream failures do, so the model and provider read
from the URL reach spend logs and metrics. Vertex Live websocket failures
(connect errors, upstream rejections, and the credential failure) now carry
the model from the ?model= query parameter under the vertex_ai provider
* fix(passthrough): mark guardrail-block and Vertex Live credential failures as pass-through calls
Prometheus reads a failure's declared provider only on pass-through calls, so
without call_type it fell back to guessing from the model name. That labeled a
guardrail block on the Gemini route as vertex_ai and the Live credential
failure as None
* fix(passthrough): let the route name a failure's provider and give guardrail blocks their own spend row
* fix(pass-through): keep URL model attribution off routes that name another provider
A built-in route that names a non-Google provider, like /openai pointed at
Gemini's OpenAI-compatible host, no longer takes its failure model from the
Google URL. A guardrail block response now returns the x-litellm-call-id of
its spend row.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vllm): send the deployment api_key on /vllm passthrough calls
VLLMModelInfo.get_api_key always returned None, so /vllm/{endpoint} calls to a router
model reached the vLLM server with no auth header even when the deployment had an
api_key, and a server started with --api-key answered 401 (#23443).
Return the deployment key (or VLLM_API_KEY) and send it as Authorization: Bearer, the
header vllm serve --api-key checks. x-api-key was never actually sent before, since the
key was always None.
* fix(vllm): send the pass-through key as a bearer on both /vllm routes
vllm serve --api-key only reads Authorization: Bearer, so a keyed vLLM
answered 401 on both pass-through routes. Send the key as a bearer and
keep x-api-key for gateways that read it. Router models send their own
deployment key and never fall back to VLLM_API_KEY, which belongs to
the /vllm route's own server.
* fix(vllm): send only the bearer on router models and stop doubling /v1 on /vllm
Router models now send the same Authorization: Bearer header /v1/chat/completions sends for that deployment, without the x-api-key copy that route never sent. The VLLM_API_BASE route keeps x-api-key next to the bearer, as it sent it before. A router model whose api_base already ends in /v1 no longer gets /v1/v1 on /vllm/v1 paths.
* fix(vllm): send no placeholder key from keyless router models and stop doubling /v1 on the VLLM_API_BASE route
* fix(vllm): send VLLM_API_KEY when a router model with no api_base is forwarded to VLLM_API_BASE
* fix(vllm): keep a router model's own key on the VLLM_API_BASE server and resolve an oidc VLLM_API_KEY
* fix(vllm): treat an empty key as no key so pass-through sends no empty bearer
* fix(vllm): send no bearer for a Google oidc key and match VLLM_API_BASE across a trailing slash
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): keep a NaN or infinite cost from wedging daily spend writes
A non-finite provider-reported cost now counts as not reported, so the token price bills the request. Any other non-finite cost is logged and recorded as 0, and spend log JSON writes NaN and Infinity as null so Postgres accepts the row.
* fix(spend): price non-finite provider costs on every route and drain parked NaN batches
Perplexity, OpenRouter pass-through, and Responses streaming now treat a NaN or infinite reported cost as unreported, so the token price bills the request. A daily spend batch parked in the Redis buffer by an older version with a NaN spend now drains as 0 instead of failing every flush. The cost header never carries NaN or Infinity, and a set holding a non-finite float no longer breaks spend log JSON
* fix(spend): record non-finite savings as 0 and warn when a parked NaN batch drains
An overflowing price on a compressed request made compression savings infinite, which still wedged the daily spend batch. Every savings column now records a non-finite value as 0 with a warning. The Redis drain warns when it reads a NaN an older version parked, since the spend summed into it is lost from daily spend. The writer test now runs the real payload builder and spend log insert
* fix(spend): keep non-finite costs out of headers, responses, and admin JSON
Writing NaN as null is now opt-in for spend log columns, the Redis buffer, and
streamed frames, so admin writes and logging integrations keep their output.
The cost calculator records an overflowing total as 0 with a warning, cost
headers leave out non-finite values, a non-streaming NaN usage.cost is dropped
instead of failing the response, and a parked NaN spend log row reads back as 0.
* fix(spend): keep caller-set call ids from splitting the non-finite cost warning
The overflow warning strips newlines from litellm_call_id, and neither
non-finite cost warning prints the raw value, which said nothing beyond
"non-finite"
* fix(spend): drop non-finite Responses and OpenRouter costs and the breakdown of an unpriceable request
* fix(types): drop a usage cost breakdown that holds a non-finite number
* fix(streaming): keep a dropped non-finite final usage cost from falling back to an earlier chunk's cost
* fix(spend): keep the dropped-cost flag out of reach of provider usage keys
A provider usage key named dropped_non_finite_cost shadowed the Usage method on the instance, so the stream builder raised and dropped the request's spend. The builder now reads the flag through the class, and Usage sets the flag after copying provider keys, so a _dropped_non_finite_cost key cannot set it either.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover /health scoping for wildcard keys and team model allowlists
* fix(proxy): scope /health with the model access check auth runs on a request
* fix(proxy): scope /health to the model group an allowed name routes to
A key alias, team alias, or router model_group_alias lets auth admit a
requested name by checking its target, so /health also kept the deployment
under the requested name, one the key's requests never reach. Keep only the
served model group each admitted name dispatches to, through the same alias
layers /v1/models mirrors, and let alias_map read its own read-only output.
* test(proxy): cover /health skipping a deployment that has no model name for a restricted key
* fix(health): show deployments reached through wildcard and alias allowlists, and by id when an alias shadows the name
* chore(router): drop should_include_deployment from the outside-use allowlist now that /health no longer calls it
* fix(health): never list a default-fallback deployment for an unserved grant, and keep access-group keys to their group's deployments
* fix(health): resolve ?model= through the caller's aliases the way chat does
* fix(proxy): apply user and agent model limits to /health and yield between checks
/health now runs the same user and agent model checks chat runs, so a personal
key whose owner may call only some models, or an agent key limited by its access
group, no longer sees or probes other deployments. Each per-name check yields to
the event loop, and ?model= reaches the wildcard deployment serving an allowed name
* fix(proxy): check owner and agent grants for wildcard routes in /health and fail on non-access auth errors
* fix(proxy): answer /health auth lookup failures the way chat does and record ?model= checks under the requested name
* test(proxy): pass recorded_model_name in the direct health save tests
* fix(proxy): count alias names as /health candidates so aliased names a key may call get probed
* fix(health): narrow wildcard-served deployments to the key's access groups
* fix(health): narrow every deployment serving a name to the key's access groups, team-owned ones included
* fix(health): narrow a team deployment's internal name to the key's access groups like its public name
* fix(health): keep a team deployment's internal name narrow when a limit holds only that name
An agent or key limit that holds one team deployment's internal name no longer reaches every deployment behind the team's public name. The internal name maps to its public name only when the public name is also admitted.
* test(health): cover all-proxy-models next to * for a key that also holds an access group
* refactor(auth): keep the _as_proxy_exception name so main's tests still find it
* fix(health): keep an internal team name a wildcard grant admits as its own target
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): bill both rounds of a streamed chat completions MCP auto-execute
A streamed /v1/chat/completions request with an MCP tool set to require_approval "never" runs two model calls on one logging object, and the final-stream dedupe kept only the first round's spend row. The final usage chunk also showed only the second round.
The MCP stream iterator now holds the first round's assembled response, prices it, and logs one success row that carries the usage and cost of both rounds, with the fixed margin charged once. The final usage chunk shows the same summed usage. A single-round stream still logs its one row when the answer finishes.
When the follow-up round fails, the first round is logged from a copy of the logging object taken before the follow-up opened, and the failure row carries only the follow-up's partial usage with no second fixed margin. With proxy post-call guardrails armed, only (assembled, cache_hit) tuples are parked for the proxy, and the deferred log is armed only while the iterator reads, so the proxy's orphaned-callback cleanup never sees it after the Router's first-chunk read
* fix(mcp): send the usage chunk of a streamed chat MCP request that ends in one round
When a streamed chat completions MCP request ends in its first round (the model answers without a tool call, or no tool results come back), the iterator drained the inner stream to fire its spend log and dropped the usage chunk that stream produced, so a client that asked for include_usage got no usage chunk at all.
The drained usage chunk is now held and sent to the client after the final content chunk and before [DONE], the same place a plain stream sends it, so the last usage chunk matches the billed row. A two-round request still drops the first round's usage chunk and sends the merged one from the follow-up, and a stream without include_usage still sends none
* fix(mcp): hold round one's log for the guardrail, bill the chunk cost once, and keep round one's id
When the follow-up call of a streamed chat MCP auto-execute fails, round one's
success log now goes through the proxy's deferred guardrail release instead of
straight to the handlers, so a post-call guardrail runs on it and the row
carries it. Without a proxy release it logs directly as before
The last usage chunk now takes its cost from the combined row's billed total,
so a fixed cost margin is counted once in the chunk, as it is on the row. The
follow-up's usage chunk is held until the follow-up ends to make that possible
The combined row now keeps round one's id, the stream's first chunk id, which
matches how a single-round stream is logged
* fix(mcp): bill every chat MCP stream round once on failure, hang-up, and sync drop
When the follow-up call fails, round 1's success row is now logged on a copy of the
logging object that does not share the request's guardrail results, so a post-call
guardrail that reports on the failed request counts once instead of on both rows
A client hang-up during the follow-up now bills round 1 plus the follow-up's partial
answer on one success row with round 1's id and the fixed margin counted once, the
same way a plain stream's hang-up bills its partial answer
The last usage chunk now carries round 1's usage and cost when only round 1 is billed,
and a synchronous reader that drops the stream mid-answer bills round 1 on the loop it
read with instead of discarding it
The deferred release's logged_on argument is typed and every release closure accepts
it, and the parked dispatch constants and protocols both MCP billing paths use now live
in one module. The unit tests wait on a logging signal instead of a fixed delay and
check the billed row through the proxy stream hook instead of private callback state
* test(mcp): drive the unassembled follow-up through an upstream audio delta that is not base64
The test replaced litellm.stream_chunk_builder with a stub that raises. Round 2 now streams an audio delta whose data is not valid base64, so the real builder raises, the stream parks only best-effort usage, and the client's last usage chunk must carry round 1's tokens and cost
* fix(mcp): count the fixed margin once in the combined stream row's usage cost
* fix(mcp): bill round 2's partial answer when a client hangs up on a routed stream
On the proxy the Router wraps the MCP stream, and when a client hangs up during
round 2 the router's own cleanup closes that stream before the proxy's disconnect
cleanup runs. The close logged round 1 alone, so round 2's partial answer was
never billed. The MCP stream now bills round 1 plus round 2's partial answer as
soon as the hang-up's cancellation reaches its read, before anything closes it
* fix(mcp): count the follow-up's own messages in round 2's partial prompt
The follow-up stream shares the request's logger, so it took the client's
original messages as its prompt. When round 2 ended early, from a hang-up or a
broken upstream, its partial usage counted only those messages and left out the
assistant tool call and the tool results the follow-up actually sent. The
follow-up stream now counts the messages the follow-up call sent
* fix(mcp): bill round 2's partial when a routed stream's client hangs up between reads
On a proxy the stream the disconnect cleanup holds is the router's fallback wrapper, so the
cleanup never found the MCP stream's in-flight billing and released round 1 alone when the client
hung up while the proxy sat at a yield. The round billing now registers its in-flight biller on
the request's logging object, next to the parked release, and the cleanup looks it up there, so
any wrapper chain bills round 1 plus round 2's partial once
* fix(mcp): send the MCP stream usage chunk under the billed row id and count the partial off the loop
A streamed chat completion that runs a LiteLLM-hosted MCP tool bills both model calls on one row under round 1's
chatcmpl id, but the usage chunk it sent the client was round 2's held chunk and kept round 2's id, so a caller could
not join the usage chunk to its row. The usage chunk now carries round 1's id in every path that sends it, with or
without a deferred post-call guardrail, while its usage and cost and every content chunk's id stay as they were
When the client hangs up during round 2, assembling round 2's partial answer counts the follow-up messages, tool
result included, with the local tokenizer. That ran on the event loop and stalled every other request on the worker
for about a second on a large tool result. The assembly now runs on a worker thread through asyncify from both the
reader's cancellation and the proxy's disconnect cleanup, shielded from cancellation, and round 1 is checked again
after the await so a release that happened meanwhile is never billed twice
* test(mcp): wait on the loop with a hang guard instead of a 300 ms deadline
* fix(mcp): give a proxy-rejected follow-up's failure row round 1's own cost breakdown
* refactor(streaming): fold the partial response's empty guards into one check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): name shared-handler errors by the status they answer
handle_exception_on_proxy pinned type=internal_server_error on every
exception it wrapped, so a management route rejecting a caller's own
argument answered {"type":"internal_server_error","code":"400"} and an
SDK branching on the type read its own bad request as a gateway outage.
Derive the type from the status the response actually carries, keeping
a type the exception already names.
* test(proxy): pin a 500 for exceptions whose status is None
* fix(proxy): keep internal_server_error for timeouts and 5xx, redact hosts in shared handler errors
Provider 5xx and timeouts keep internal_server_error so retry loops still retry them,
the shared handler redacts internal hosts from raw exception messages, and /key/list
routes its HTTPException through the shared handler instead of its own copy of the
old label.
* fix(proxy): label shared-handler errors like the chat route, pick the status before the type, redact only gateway failures
* fix(proxy): pass shared-handler error messages through as on main
Redacting timeout and 5xx messages changed what every shared-handler route
returns, which relabeling the error type does not need, so messages go back
to the verbatim text main sends.
* fix(proxy): redact only the status-less failure main never sent
* fix(proxy): answer a generic 500 for a status-less failure and expect auth_error on the /user/new 403
A failure with no status now answers "Internal server error" instead of its own text, since that text can
name a guardrail's private host and pattern redaction misses cluster DNS names and IPv6 addresses. The
/user/new integration test now expects the auth_error type main's route catch-alls give a 403 the proxy raises
* fix(proxy): answer only a 4xx or 5xx carried status and fall back to the route default otherwise
A numeric-string status outside 400 to 599 made /v1/messages, images, rerank and the OpenAI pass-through answer HTTP 200 with an error body or drop the connection, and non-ASCII digits or a 5000-digit string slipped through isdecimal. error_status_code now accepts an int or a three-digit ASCII string in 400 to 599, the range the chat route already enforces
* refactor(proxy): parse a three-digit string status without recursing
* fix(sso): bind CLI SSO and Claude Code gateway logins to the first team on the user record
* fix(proxy): keep SCIM and JWT team sync in record order, and look up SSO groups once
SCIM PATCH and JWT team sync saved a user's teams from a set, so the first
team on the record, the one a Claude Code gateway sign-in binds, could change
on any sync. Both now keep the record's order and append new teams in the
order the IdP sent them.
SSO group sync now drops claim groups that are not LiteLLM teams with one
lookup before adding the missing teams one at a time, so a claim full of
non-team groups no longer costs one failed add per group on every sign-in.
* test(sso): expect the gateway team dropdown in the user record's order
* fix(sso): read team existence from the writer, keep new-user and SCIM replace team order
* fix(scim): keep the teams a user already has in record order on PATCH and PUT
* fix(scim): keep record team order when a SCIM POST matches an existing email
* fix(scim): join new teams in request order so a failed roster write keeps it
* fix(scim): let a groups replace win over later group ops in a PATCH again
Applying group adds and removes after a replace made a PATCH that deactivates a user
and adds a team that does not exist answer 404 and leave the user active, where it
answered 200 before. Keep the record's team order, but ignore group ops after the last
replace as the base did.
* test(sso): type the CLI team detail row helper
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: rj-cairn <349020452+rj-cairn@users.noreply.github.com>
* fix(router): apply content_policy_fallbacks when router settings update at runtime
* test(router): annotate the fallback regression test's locals as Final
* fix(proxy): validate router_settings.content_policy_fallbacks on /config/update like its siblings
* test(proxy): annotate the content_policy_fallbacks config update test locals as Final
* fix(proxy): keep litellm_settings content policy fallbacks, accept dict targets, skip invalid stored rules
A runtime content_policy_fallbacks rule now layers over the litellm_settings rules instead of replacing
them, so models the runtime rules do not name keep their file fallbacks. Fallback targets may be
deployment dicts as well as model names. A stored rule that fails validation is logged and skipped at
sync, so the other router settings still apply. Per-user token discovery now reads every fallback key
from key and team router_settings.
* fix(proxy): validate content policy fallback rules one by one and keep a config-owned list alone
An invalid stored rule is now skipped on its own instead of dropping every runtime rule, a router_settings list from the config file stays the router's only rules across DB syncs, and GET /fallback names the model of a dict target.
* fix(proxy): answer an error for a string-valued fallback rule on GET /fallback instead of its characters
* fix(proxy): keep config-file content-policy rules as Router init accepted them and log a skipped rule once per change
* fix(proxy): log a skipped content-policy rule whenever the saved rules change
* fix(proxy): keep general fallbacks when no content-policy rule is saved, name a model-less target's model, and take only model names in key rules
* fix(proxy): reject an empty dict target in a key's content-policy rule
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(usage): show subscription-covered providers and account fees separately
* fix(usage): write only the named fee columns and sync the root schema
The fee update now writes just the columns the request names, so two admins changing
different fields at once both land, and the dashboard edit form sends only what changed.
The root schema.prisma carries the new table like the two copies, and a billing period
anchored in the calendar's last month ends on December 31, 9999 instead of failing.
* fix(usage): register the subscription table span and keep provider logic under llms
* fix(subscription_accounts): register the fee routes as management routes and reject an empty account id
* fix(usage): run the new tests in CI and close subscription card review gaps
Adds the subscription test file to its unit shard, sets the migration lock timeout, rejects infinite fees, keeps the card from rewriting login files, lists each deployment once without aliases, shows the provider badge on entity views, and asks before removing a fee
* fix(usage): resolve named-credential ChatGPT accounts and keep admin viewers read-only on the fee card
* fix(usage): count a subscription fee only for accounts behind a current deployment
Usage listed a fee row for an account no deployment uses while another ChatGPT
deployment existed and dropped it when none did. It now lists only accounts behind a
current deployment in both cases. Adds a real proxy and Postgres integration test for
the fee API and for ChatGPT Responses rows on a deployment's own login, and names the
card's multi-component test with the integration suffix.
* fix(usage): keep a stored subscription fee listed after its account's deployments are removed
The usage route lists every account behind a subscription deployment plus every account carrying a stored fee, and reads the fees on every proxy, so a fee stays visible, counted, and editable for past ranges after its last deployment goes. The integration test now polls every day the calls spanned, so a UTC midnight crossing no longer fails it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): apply published model rows on every worker before the debounced resync
Config sync messages now carry the ids of model rows a create, update, or upsert wrote. Every worker that receives one loads those rows by id right away, including while its debounced full resync is still waiting, so a group added one deployment at a time is complete on every worker as soon as the write returns. Deletes, bulk writes, and messages from older pods carry no ids and keep the full resync path
* fix(proxy): load published model rows from the primary under the reconcile lock
Read the by-id rows through the writer so a lagging read replica cannot put an old row back, and read them inside MODEL_RECONCILE_LOCK so a concurrent delete or reload stays the last word. Look the ids up in bounded chunks, reload DB credentials only when a row names one this worker lacks, and leave auto-router rows to the debounced full resync. Add an integration test that boots two proxies and needs the peer to fall back to a just-added backup within 4 s.
* test(proxy): cover a published model row that names no credential
* fix(proxy): keep the unknown model name read-through on the replica and outside the lock
Only the by-id load that follows a published write reads the primary inside
the reconcile lock. A request for a model name a worker does not know reads
through the replica routing before taking the lock again, as on main, so
unknown names put no load on the primary and do not wait behind a reload.
* fix(proxy): keep access group edits from dropping staying deployments and batch published config ids
* fix(proxy): read access group rows from the primary so quick edits are not lost to a lagging replica
* fix(proxy): install ordinary rows queued with an auto router and check access group edits before writing
A batch of published model ids that held one auto router skipped every row in it, so ordinary
rows written at the same moment waited for the full reload. The by-id load now leaves only the
auto router rows to that reload.
An access group update now reads every deployment from the primary, answers 404 for an unknown
group and 400 for an unknown model id or name before it writes anything, then removes the group
from the deployments that leave before it adds it to the ones that join, so a failed write leaves
the group narrower than asked, never wider.
* fix(proxy): write an access group update in one transaction after a fresh read
The update now checks model names first, reads every deployment from the
primary, and commits every join and leave in one batch right after that
read, so a failed write changes nothing and a field another admin set
during validation is kept. Other workers hear about the rows only after
the commit.
* fix(proxy): lock the model rows while an access group update reads and rewrites them
* test(proxy): run the model resync tests on the real installer and a real router
The by-id resync tests now install rows through ProxyConfig's real installer and credential loader on a real Router, faking only the database, and the config sync test row model is frozen.
* fix(proxy): lock only the rows an access group update touches and answer 404 before 400
An access group update locked every model row, so a lock on any unrelated
row stalled it for 30 s and then failed it with a 500, and unrelated model
edits on other proxies waited behind it. It now takes a per-group advisory
lock, locks only the rows that carry the group or that it selects, and
reads them again once locked. The group existence check runs before the
model name check again, so a missing group answers 404 as on main.
* fix(proxy): never wait on a locked row in an access group update
The update now takes its row locks with NOWAIT. When another edit holds one
of the group's rows, it rolls back, holding nothing, and tries again for about
10 s before answering 409. Holding rows while waiting deadlocked against team
deletes and let a concurrent model edit undo the update's change. The
per-group lock also moves to the two-key advisory space, apart from team ids.
* fix(proxy): retry access group row locks for 30 s with SKIP LOCKED and lock only the rows model_ids select
* fix(proxy): take the access group lock with pg_try_advisory_xact_lock and retry while another edit holds it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep a deployment's own model_id over the request body's
A request body model_id overrode the deployment's pinned model_id, so a client could move SageMaker chat and embedding calls to another inference component. The router now drops the request's model_id when the deployment sets one.
* fix(router): let deployments opt back in to a client model_id and stop client component headers on SageMaker chat
Drop the request's model_id after the router defaults are applied, so a default_litellm_params model_id cannot win
over the deployment either, and apply the same precedence on the sync text_completion path. A deployment that lists
model_id in configurable_clientside_auth_params keeps the client's value. SageMaker chat drops any case variant of a
client's X-Amzn-SageMaker-Inference-Component header when the deployment sets a component
* fix(router): keep a SageMaker deployment's component when it pins it through extra_headers
* fix(router): read the SageMaker component header as a pin only on SageMaker deployments
* fix(sagemaker): send the inference component only as a header, never as a body model_id
SageMaker picks the component from X-Amzn-SageMaker-Inference-Component, so the chat
container body and legacy parameters no longer carry model_id. Nova endpoints accept a
fixed body field list, so a pinned Nova deployment could reject the extra field.
* fix(health): probe header-pinned SageMaker deployments at their component
Health checks bypass the router, so a legacy sagemaker/ deployment pinned only
through its inference component header sent no component on /health while routed
requests now send it. Also update the two SageMaker integration tests that still
asserted the old precedence and the dropped body field.
* fix(health): apply the router's model_id pin rule to health probes
Health probes now share the router's pin function, so a deployment that lets
clients set model_id gets no header pin and its own model_id wins without
reading the header. Header keys that are not strings are skipped when
reading the inference component header instead of failing the call.
* fix(sagemaker): keep header keys that aren't strings and report a string opt-in in health checks
The chat transformation now keeps extra_headers keys that aren't strings, so such a deployment fails the way it did before instead of answering 500, and the model_id pin checks configurable_clientside_auth_params with isinstance, so a string there reaches the health probe's own error capture again.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): restore spend logging for /cursor/chat/completions (#30126)
The /cursor/chat/completions route processes requests through the Responses API pipeline, but _get_metadata_variable_name did not treat it as a litellm_metadata route. Proxy auth metadata therefore landed in the user-facing metadata param, so the spend tracking callback saw user_api_key=None and silently skipped the SpendLogs write. Add the route to LITELLM_METADATA_ROUTES and cover it with regression tests.
* test(proxy): pin provider metadata and key billing on /cursor/chat/completions
A Responses-format body on the Cursor route now sends OpenAI only the metadata the client sent and writes a spend row that bills the key. The chat-body branch, which this fix also moves onto litellm_metadata, still bills the key with its team and request tags. The scripted chat reply gets a unique id per test, since a shared id let only the first spend row per database land
* fix(proxy): pick the Cursor route's metadata field from the body shape
A Responses-style body on /cursor/chat/completions keeps proxy metadata in
litellm_metadata, the same as /v1/responses, while a Chat Completions body
keeps metadata, the same as /v1/chat/completions. Adding the whole route to
the internal-metadata list let a client's chat-body tags replace the key's
tags in spend logs and skip tag budgets.
* fix(proxy): keep get_metadata_variable_name callable with only the request
* fix(proxy): check client tags against tag budgets and read agent limits from litellm_metadata on Responses bodies
A Responses-style body on /cursor/chat/completions now keeps its proxy fields in litellm_metadata, so the pre-call tag budget check and the v3 agent rate limiter stopped seeing the client's metadata.tags and the x-litellm-agent-id value. The pre-auth tag check now merges the caller's metadata.tags into the selected metadata field, sharing one helper with request setup, and the limiter reads agent_id from either field
* fix(proxy): check client tags sent in a JSON-string metadata against tag budgets on Responses bodies
* fix(proxy): answer 422, not 500, when a budget rejects a Responses body whose metadata is a JSON string
* fix(proxy): keep key guardrails, metadata type errors and failure-row client details on Responses-style bodies
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): list only routable models for a wildcard alias over a provider org
A wildcard alias over an org-scoped deployment (together_ai/* over
together_ai/nosuchorg/*) listed the provider's whole catalog on /v1/models
and /model/info, and every listed id routed to a model the provider 404s.
A skipped deployment path segment now counts as a routing prefix only when
the routed model resolves to a catalog id through the get_model_info
lookup order, so responses/, Bedrock converse/, and region paths list as
before.
Together's model list now also carries the cost map's together_ai rows,
skipping the together-ai-* pricing tiers, so an alias over a real org lists
the models Together serves today.
* fix(proxy): keep flat catalog ids under routes the cost map can't parse, drop retired Together rows
* fix(proxy): check skipped path segments only on Together so Hugging Face provider routes keep listing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): count a compaction block instead of answering 500
A message history that replays Anthropic's signed compaction block made
/v1/messages/count_tokens answer HTTP 500: neither the Anthropic nor the
Vertex AI partner count-tokens request carried the compact-2026-09-04 beta,
the provider rejected the block with a 400, the proxy fell back to the local
tokenizer, and that raised on the unknown block type.
Both provider count paths now add the compaction beta whenever the history
carries a compaction block, and the local tokenizer counts the block's
summary text like a text block, skipping the signature.
* fix(token_counter): skip the tokenizer for empty thinking and compaction text
* fix(count_tokens): forward context_management on Anthropic counts and count chat compaction blocks locally
Vertex AI compaction support for inference and counting moves to its own PR, so the
Vertex count handler is back to main here
* fix(anthropic): map list-form context_management to edits for count_tokens and inference betas
Count requests forwarded the OpenAI list form of context_management unchanged, while inference
maps it to Anthropic edits, and the inference beta helper read the list as no edits. Both now
map it once. A replayed compaction block with an empty summary no longer reaches the tokenizer.
* fix(anthropic): forward count context_management only with edits, map Vertex list-form betas, and return provider context_management
* fix(anthropic): drop an unmappable context_management list under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch-interception): stream native server-tool blocks to clients
Streaming native clients (Anthropic SDK, Claude Desktop, Claude Code)
never received the server_tool_use and web_search_tool_result blocks the
interception layer builds for them, so citation panels stayed empty and
result counts read zero. Two gaps in the same flow:
FakeAnthropicMessagesStreamIterator._create_content_block_chunks only
knew text, thinking, redacted_thinking and tool_use; any other block
type emitted a bare content_block_stop with no start and no payload,
silently dropping the block from the fake stream. server_tool_use now
streams as content_block_start plus input_json_delta and any other
server-generated block rides whole in content_block_start, matching
Anthropic's native streaming of these types.
try_short_circuit_search only recognized a native client by finding a
web_search_* tool in the tools list, but the pre-request and deployment
hooks convert that tool to the LiteLLM standard shape before the
short-circuit runs, recording the conversion in
WEBSEARCH_EMIT_NATIVE_BLOCKS_KEY. The flag now counts as native too, so
the synthetic response carries the native blocks on the converted path
instead of only when hook ordering happens to leave the tool untouched.
Fixes#35333
* fix(anthropic): accept server_tool_use and web_search_tool_result blocks when costing a /v1/messages response
The short-circuit now returns native web search blocks, and the cost path validated the served body against a content union that only allowed text, tool_use and thinking blocks, so non-streaming requests answered 500
* fix(anthropic): keep every server_tool_use field when re-streaming a /v1/messages reply
* fix(anthropic): keep tool_use fields, redacted thinking data and text citations when re-streaming a /v1/messages reply
* test(websearch-interception): build the cost test's logging object with a datetime start time
* fix(websearch-interception): send search snippets once in the direct search answer
* fix(guardrails): mask web search queries, titles and urls on /v1/messages and deliver stream rewrites past server_tool_use blocks
Output guardrails now scan the native web search blocks the short-circuit emits, ahead of the reply text so guardrails that read the last text still scan the reply. A guardrail reply of a different length falls back to the reply-only write-back. Ended-stream tool call rewrites land on server_tool_use blocks too, the ended-stream scan key covers the result strings, and legacy-hook pipeline steps pass through the result strings their response view lacks. The wire test reads the streamed server_tool_use input from its input_json_delta.
* fix(guardrails): deliver a legacy post-call hook's tool-call rewrites on streams instead of discarding the stream rewrite
A legacy hook that masked a tool call's arguments on a buffered stream made the whole
rewrite undeliverable, text included, so the original stream went out. With web search
blocks in the stream that now hit any hook masking the search query. The adapter hands
same-count tool-call rewrites to the translation's write-back, as a unified guardrail's
already are, and still discards a rewrite that adds or drops a tool call.
* fix(guardrails): show a legacy post-call hook the web search result blocks of a /v1/messages stream
The legacy hook's view of a buffered Anthropic stream is rebuilt from the assembled
ModelResponse, which keeps web_search_tool_result blocks only in provider-specific
fields, so a hook masking result titles on a stream never saw them. The hook's view now
carries those blocks, its rewrite of them goes back through the ended-stream write-back,
and the adapter no longer needs to pass texts outside the hook's view through untouched.
* test(guardrails): type the legacy hook stream test helpers
* fix(guardrails): scan a streamed web search query as text like the non-streaming reply
* test(guardrails): type the masking guardrail test helpers
* fix(guardrails): keep the block id when scanning a streamed web search query
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(health): probe only models a partial wildcard serves
A partial wildcard health check now probes the cheapest cost-map chat models the wildcard matches, keeps provider route prefixes such as Bedrock converse/ on the probe, and falls back only within the wildcard. A wildcard with no matching cost-map chat model reports an error naming model_info.health_check_model
* fix(health): keep the provider-wide probe for an unmatched wildcard, skip retired models, and ignore Bedrock ARNs
A partial wildcard with no served cost-map chat model falls back to the provider's cheapest chat models
instead of failing, and fails with a message naming model_info.health_check_model only when the provider
has none. The cheapest-model picker skips models past their cost-map deprecation_date, and a Bedrock ARN
segment is never read as a route prefix.
* fix(health): fall back to the provider's cheapest probe when a wildcard's own models fail
A partial wildcard now probes up to three of its own cheapest chat models
first and then the provider-wide cheapest ones, all from one cost-map scan,
so stale or unpriced rows the wildcard matches cannot turn a healthy
deployment red. Retired models are read by their resolved cost-map key,
both deprecation checks use the UTC date, Bedrock only treats converse,
invoke, and region names as route segments, and equal-price ties break by
model id
* fix(health): cap wildcard health probes at three and rank retired cost-map rows last
A partial wildcard now probes two of its own cheapest cost-map chat models and then one
provider-wide model, so a failing deployment makes no more calls than before. Rows past
their cost-map deprecation date go last instead of being dropped, since some still serve,
and the shared cheapest-model picker returns what it returned before.
* fix(health): probe a wildcard's own models first, then the same three models main probes
A partial wildcard's health check tries its two cheapest own cost-map models, then the provider's three cheapest models main already probes, so a row main shows healthy stays healthy. The deprecation check uses the server's local date, like the Baseten listing
* fix(health): keep main's provider-wide probe order so equal-price ties match main
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): keep the upstream error response on mapped exceptions so e.response.text has the wire body
* fix: keep the upstream error body on 429 and 5xx errors
RateLimitError, InternalServerError, BadGatewayError, and ServiceUnavailableError
rebuilt their response and dropped the upstream body. They now keep the body and
exactly the upstream headers, and an unread streaming response no longer raises.
* fix: keep a gzip-encoded 429 or 5xx error readable
The kept body is already decoded, so the rebuilt response drops content-encoding
instead of decoding the body a second time and raising DecodingError.
* fix: keep the upstream headers and drop credentials on every mapped provider error
The copied error response now keeps every upstream header and carries a
request with only the method and the URL, so no provider key, query
secret, or prompt reaches e.request. Timeout and APIError get the
response too, and azure/ chat, streaming, embedding, and image errors
attach it
* fix: keep the upstream error body on gateway timeouts and re-raised gateway errors
* test(azure): cover the 404, 500, 502, and 504 error bodies and the missing-key placeholder
* fix: keep no e.response on timeouts and errors litellm raises itself
* fix: keep the upstream elapsed time on error responses and the upstream body on azure_text errors
* fix: keep the upstream body without credentials on Azure content filter errors and message-matched OpenAI errors
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(types): replace Any with proven types in 35 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): keep anthropic chat transformation json path untyped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): accept null gemini vector store fields and load encryption TypedDicts on python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): validate Python client user responses and type delete_user as the count it returns
* fix(types): keep loose payloads working under the sweep's validators
An off-schema Anthropic stream event is now skipped instead of ending the
guardrail's text and stop_reason scan, and MAI image usage accepts the loose
numbers it did before. New dict and list adapters hide their input from
validation errors, RerankResponse indexing keeps its field types, and the
delete_user test mocks the int /user/delete returns.
* fix(types): make the changed DynamoAI fields ReadOnly and skip non-object A2A stream items
* refactor(types): tighten round 2 parse sites and keep payloads out of errors
Parse Anthropic SSE events with validate_json and ignore extra fields, validate admin team and key rows per row with hidden inputs, type rerank get and __getitem__ per key, and move the Phoenix prompt version response to a frozen BaseModel
* fix(types): keep MAI image calls alive on unreadable usage counts
An unreadable MAI usage count now counts as absent instead of failing a
billed image call, and the provider's usage fields pass through as sent.
The Anthropic stream scan warnings no longer log raw model output, and the
off-schema skip has tests that fail on the old scan.
* fix(types): count non-object MAI usage as absent and parse Anthropic SSE with json.loads
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(batches): read OpenAI batch usage into the batch token counts
* fix(batches): serve OpenAI's batch usage shape from retrieve and the cost poller
* fix(batches): keep the chat usage keys next to OpenAI's batch usage fields
Batch usage now carries every chat key it had plus OpenAI's five batch fields with the same counts, so poller-billed batches keep prompt_tokens and cache-write tokens, and older proxies still read the new rows. Usage with missing or null detail counts reads its real input and output counts.
* test(batches): check the batch usage counts in the e2e terminal-state test
* fix(types): apply nested include and exclude to both batch usage shapes
* fix(batches): log no cost for a batch retrieve that is not final yet
A retrieve now reads OpenAI's batch usage into the chat counts, so a retrieve
that saw the batch completed before its output file arrived priced those counts
for the logging callbacks, and the final retrieve priced the whole batch again.
A batch that is not final now logs no cost, as it did before.
Batch usage dumps also pass exclude_unset and exclude_defaults through to the
usage keys.
* fix(batches): pass through every usage detail field the provider sends
Batch dumps rebuilt input_tokens_details and output_tokens_details from the chat keys, which dropped any detail field past cached_tokens and reasoning_tokens that the merge base passed through. The dump now starts from the provider's stored details and only sets the two counts derived from the chat keys
* fix(batches): skip sync loggers on the cost poller's own retrieve and honor exclude_none on provider details
* fix(batches): keep sync logger events at $0 on poller-owned retrieves, pass unparsed provider counts through, log batch usage in chat shape
* fix(batches): store only chat usage extras on batch spend rows
Spend rows for batch retrieves copied OpenAI's batch fields (input_tokens,
output_tokens and their details) into metadata.additional_usage_values next
to the chat counts they duplicate. Build the row from the batch's chat usage,
shared with the logging payload through batch_chat_usage.
* fix(streaming): rebuild each choice of an n>1 chat stream on its own
stream_chunk_builder folded every choice of an n>1 chat stream into choice 0, so post-call guardrails such as tool_permission re-emitted one choice holding every choice's tool calls, and spend logs and logging callbacks kept only the first choice's text. Each choice is now assembled from its own chunks, while usage, cost, and hidden params still read the whole stream. Cisco AI Defense compares each choice's streamed text on its own so safe n>1 streams still pass through unchanged.
* fix(streaming): keep bridged reasoning summaries on choice 0 and infer an unreported choice finish reason
* fix(streaming): match MCP follow-ups and bridged summary text to the rebuilt choices
MCP auto-execute now answers only the first choice's tool calls, since its
follow-up turn continues that choice alone. The Responses bridge streams a
space before each reasoning summary part after the first, matching the
non-streamed reply. build_base_response now takes the per-choice streams
instead of having its choices replaced afterwards
* fix(streaming): space every later reasoning summary part, deny per judged call, and rebuild in one text pass
* fix(guardrails): key tool_permission denials for custom tool calls too, and type the stream rebuild's messages
* fix(mcp): continue the first choice that called a tool when an n>1 reply auto-executes MCP tools
* fix(noma): scan each choice of a multi-choice reply on its own
* fix(noma): keep masked choices when a later choice's scan fails open on a stream
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): price PDF pages for OpenAI and Azure models the way they bill
* fix(token_counter): count azure, data-zone and dated GPT names with their family's encoding
* fix(token_counter): price only drawn PDF images and keep other gateways' GPT pricing
Detect raster images from the page content stream, including inline
images on pages with empty resources, and walk each form once. Strip
only openai/, azure/ and Azure data-zone prefixes, bound the model
family scan, read Azure-only chat models from the cost map, and keep
cl100k_base for listed non-chat OpenAI models.
* fix(token_counter): price GPT names behind any prefix like the bare name and walk shared forms per resources
Prices a PDF by the model name's last path segment, the same name the proxy counts with,
keeps listed non-chat models on Claude pricing, walks each form once per inherited resources
dictionary so a shared child form drawing an image is not missed, and resolves PDF pricing
only when a message carries a PDF
* fix(token_counter): keep the rendered-page count for image pages, parse each form once, and match chat models only
A PDF page that draws a raster image keeps the rendered-page estimate, since newer GPT models bill such a page near
it. The image walk parses each form once and stops at a fixed number of drawn XObjects, so a form shared under many
resources dictionaries no longer slows the count. The family prefix match skips non-chat entries, and ft: fine-tunes
count like their base model
* fix(token_counter): count Azure-only chat models with o200k like other OpenAI chat models
* fix(token_counter): count reservation and project ITPM with the serving deployment's model
Budget reservation and project ITPM counted with the requested group name, so a
GPT-named group served by Claude reserved a PDF at OpenAI's price. They now count
with each deployment's model minus its provider prefix, the name count_tokens uses,
and take the largest. A custom OpenAI tokenizer with no model counts with cl100k
again instead of raising, and listed Azure gpt- chat entries price as OpenAI.
* perf(token_counter): find drawn PDF images with a byte scan instead of parsing every operator
pypdf parsed every operator of a page's content stream to find the images it draws, so a
compressed page of a million operators took about 30 seconds to price for an OpenAI model.
A linear scan of the decoded stream for name Do and inline images finds the same draws.
* fix(token-counter): count each team deployment by its PDF pricing and honor base_model
Budget reservation and the TPM limiter now look up the deployments that serve the
caller's team, by base_model when set. A deployment that reads PDF pages and prices
them unlike the group name counts with its own model, any other with the group name.
A comment between an XObject name and Do now reads as whitespace, and the timing
tests moved to CodSpeed benchmarks.
* fix(benchmarks): install pypdf in the benchmark environment
The PDF token-count benchmarks parse pages with pypdf, which only the proxy-runtime extra installed, so CodSpeed took the no-parser path.
* fix(proxy): count PDF pages for deployments served under multi-segment model names
* fix(proxy): count a team's wildcard route the way routing serves it and reserve OpenAI chat's per-PDF charge
* fix(proxy): count routing-group and alias members with the caller's team, as routing serves them
