PR #49113 dropped this row without a stated reason while AWS still serves the EU
inference profile. Without it, calls fall back to the base Haiku 4.5 row and bill
at the non-regional price, and exact-key lookups in litellm.model_cost fail.
Restores the row unchanged from 7aa0393cb in both the root and backup maps.
Fixes#49312
* fix(a2a): bill completion-bridge agents at their own prices on /a2a send and stream
message/send on an agent served through the completion bridge (custom_llm_provider
agents such as LangGraph or databricks_agent) logged a $0, 0-token row, and
message/stream logged no row at all, so cost_per_query and per-token prices never
reached spend.
The bridge send path now records counted usage, the agent's pricing keys, and the
agent id on its logging object, the same way URL agents do. The bridge stream is
wrapped in the A2A streaming logger, which now reads the bridge's artifact event
for the reply text and gets the agent name from the proxy. Streamed calls also
count prompt tokens again, since the message text reader now reads pydantic
messages as dicts.
* fix(a2a): count chunked and Task replies once and skip agent prices on error replies
Bridge streams now follow the A2A append flag per artifact, so chunked
pydantic-ai, watsonx, and AgentCore replies count the whole reply once.
Send replies that are a Task count their artifact text. A JSON-RPC error
reply is logged at cost 0 instead of the agent's price.
* fix(a2a): keep one reply text per stream and bill every artifact and status reply
* fix(a2a): ignore a null error field on streams and run the stream per-token billing test
A stream event whose error field is null no longer zeroes the call's cost. The integration test for per-token stream billing now runs instead of skipping, and the URL agent SDK case skips when the optional a2a-sdk is not installed.
* fix(a2a): bill streams the client drops and skip the bridge's working placeholder
A stream the client closes or cancels mid-reply now logs its spend row at the agent's prices on both the bridge and URL paths, and a stream dropped before the agent sends anything still logs nothing. The bridge's "Processing request..." status no longer counts as reply text, so an empty bridge reply bills no completion tokens, as send does. Adds proxy-level tests that a bridge stream bills under the agent's card name.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality
sub-routers now take a RoutingContext instead of the Router, and read the router only through
registry(), settings(), health() and the context's forwarding methods. Router-free helpers move
from router_utils to litellm_core_utils, outside callers of router state switch to entry points,
and the surface allowlist shrinks to the endpoint methods plus the entry points
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* fix(guardrails): keep request steps when a response pipeline blocks before a later request pipeline
* refactor(proxy): inject gateway activity through a typed repository
* ci: keep dependency tests in their existing shard
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails (#49263)
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* refactor(proxy): inject database repositories through per-repository dependencies
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): model Prisma client cleanup
* test(rust): include NativeDatabase in bridge API surface
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): give database shutdown one lifespan owner
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): log database cleanup errors and bound shutdown waits
Database.aclose now logs a failure in either cleanup step instead of raising
out of the lifespan exit, which matches how main handled stop errors and keeps
setup_prisma_client from raising out of cleanup after a failed connect
Shutdown waits for an in-flight reconnect for a fixed five seconds instead of
the reconnect timeout plus grace, and joins engine wait threads with a fixed
limit instead of indefinitely. The threads are daemon threads, so a thread
still alive after the limit cannot keep the process from exiting
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Moves the page-authored English on /policies, /tool-policies and
/guardrails-monitor into react-i18next t() calls backed by en.json, and
adds a small data test for en.json. The rendered English is unchanged.
* fix(batches): cost an unmanaged gs:// batch through the deployment that created it
The create now stores the router deployment id as model_id inside the
managed object row's file_object JSON and CheckBatchCost reads it before
falling back to the first deployment matching the base model in the
gs:// path. Rows without the field keep the old lookup
* fix(batches): tolerate a non-string or malformed file_object when reading the stored deployment
* fix(batches): read the stored creating deployment from the writer, not a lagging read replica
* fix(batches): resolve the writer database once so the type gate count does not grow
* fix(batches): decode a dict-shaped file_object when reading the stored deployment, accumulate wire samples, annotate test locals
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sso): create the team_id_jwt_field team on UI SSO login when team_id_upsert is set
* fix(sso): keep the upserted team_id_jwt_field team out of result.team_ids so restricted_sso_group and the access-token group fallback are unchanged
* fix(sso): only upsert the team_id_jwt_field team on UI SSO login when enable_jwt_auth is on
The API JWT path only reaches team_id_upsert inside the enable_jwt_auth branch, so a deployment
with a leftover litellm_jwtauth block and JWT auth off never created teams. Gate the UI SSO upsert
the same way
* fix(sso): read enable_jwt_auth through the proxy_server module so the basedpyright gate stays at its ceiling
* fix(responses): run the guardrails a Responses WebSocket frame selects
* fix(responses): name the field a malformed frame guardrail selection got wrong
A frame whose guardrails or guardrail_config has the wrong shape now gets
an error event naming the field, instead of pydantic's raw validation dump
* fix(responses): accept every guardrail selection shape a Responses WebSocket frame sends, and run during_call guardrails
A frame can pick guardrails as a list of names or name-to-config dicts, inside metadata or litellm_metadata, with policies validated as a list of strings. during_call guardrails now run before the turn goes upstream, and selection fields sent next to a nested response are rejected instead of dropped
* fix(proxy): keep a key's configured guardrails when policies add guardrails
A policy's guardrails were merged with dict.fromkeys, so a configured guardrail entry ({name: {extra_body}}) in key metadata raised unhashable type: 'dict' and every request naming policies failed with a 500 on HTTP and on the Responses WebSocket. The merge now keeps existing entries and skips policy guardrails already selected by name. Also types the WebSocket test guardrail's hooks.
* fix(responses): run streaming-scoped guardrails on socket turns, keep guardrail fields on blocks, merge guardrail lists by name
* fix(responses): check frame guardrail picks against the team permission, honor picks beside a nested response, prefer a guardrail entry with settings
* style(responses): format the frame guard and give the pass-through guardrail test an enterprise license
* fix(proxy): keep pass-through guardrail settings, gate frame policy picks, skip moderation-only hooks on socket turns
* fix(responses): keep string metadata guardrail values as client metadata on socket turns
* fix(responses): ask no team permission for string metadata named like a guardrail pick on socket turns
* fix(proxy): socket guardrail picks read the key's team metadata when the proxy has no database
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Carry the Z.AI catalog fix from #40088 and move its regression into the mapped unit test file.
Co-authored-by: Chaitanya Laxman <chaitanyalaxman118@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover separate Meta Model API provider onboarding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): add Meta Model API provider onboarding
Co-authored-by: lostmartian <sahilgangurde08@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): align Meta coverage with provider keys and listed registry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show Meta model placeholder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: lostmartian <sahilgangurde08@gmail.com>
Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose
* fix(proxy): raise live budget counters for failure spend rows
A stream that breaks mid-flight writes its recovered cost to the spend log, but the failure hook
had already released the reservation and never touched the live counters, so an over-budget key
kept passing until the counter was next repaired from the database
Every charged failure row with a cost now raises the same counters the success path raises. The
error waits at most a second for them, so a hung Redis cannot hold it back
Fixes#35568
* test(proxy): cover failure spend counters across budgets, endpoints and outages
Integration cells for team, user, end-user, tag and window budgets,
concurrent streams, billed retries, native /v1/messages, Bedrock guardrail
cost, a Redis stall on two workers and disable_spend_updates
* test(proxy): check stalled failure counters and use the real spend writer
- The Redis stall cell now reads the burst key's live counter and requires at least the full failure spend
- Unit tests drop the charging writer double, bare dicts, list mutation and wall-clock timestamps
- Partial-stream test models are frozen
* test(proxy): move the stalled counter bound reason into the assert message
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): apply key and team router_settings to /v1/images/generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format router settings helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve key model_group_alias once on image generations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openrouter): list OpenRouter's live catalog for openrouter/* when check_provider_endpoint is on
* fix(openrouter): list models off the event loop and keep only chat-callable catalog ids
The model-list builders are sync, and with check_provider_endpoint on they call OpenRouter's catalog with a 10 second timeout and no caching of failures. They ran on the event loop inside /v1/models, /model/info, /model_group/info and the per-user listing, so one slow catalog froze every request on the worker. Those calls now run in a worker thread with asyncio.to_thread
File, batch and RAG credential lookups only need each name's deployment, and a wildcard that is itself a deployment sits in the list ahead of its expansions and resolves its own credentials. get_complete_model_list takes expand_deployed_wildcard_routes, which that lookup sets to False, so a deployed wildcard never fetches its provider catalog there. A key wildcard wider than any deployment, like anthropic/* over a deployed anthropic/claude-*, still expands so it finds the matching deployment's credentials. Every other caller keeps the default
The pattern router's model lookup tolerates a pattern removed between its two reads of the pattern table, which a deployment delete on the event loop can now do while a listing runs in a thread
Listing now reads only OpenRouter's models/user catalog. OpenRouter's own /models list carries no embedding models, the OpenAI-format /v1/models has no mode field so chat pickers cannot tell embeddings apart, /model_group/info labels them mode chat, and one failing catalog emptied the whole list. Ids ending in :batch are dropped because chat completions answers 404 for them, while :free and every other variant stay. An api_base that already ends in /chat/completions, which chat accepts as is, has that suffix stripped before the catalog route is joined
* fix(proxy): keep model listing off the shared thread pool and out of credential lookups
Listings run on their own thread pool so many slow catalogs cannot starve
the default executor that chat and Responses calls share. File and batch
credential lookups match deployed names against a key's wildcard grants
instead of listing any provider catalog. The token counter keeps OpenRouter
deployments on their previous path, and a catch-all "*" over provider/*
lists bare ids that route back through it
* test(proxy): give the off-loop listing tests a generous deadlock guard and typed ids
* fix(proxy): keep routed models in wildcard model info, skip key-only OpenRouter catalogs, cache failed listings briefly
* fix(utils): cache failed model listings under their own key so a late failure never hides a good listing
* fix(proxy): remember only timed-out provider listings so a provider that failed fast is listed again
* fix(proxy): ask each provider catalog once per worker, keep deployed names out of catch-all ids, and match file credentials by name
Listings that start together for one deployment now wait for the first catalog request. A catch-all wildcard no longer lists a name a deployment serves by name. A key wildcard only finds file and batch credentials on deployments named under it, so a bare name keeps the default credentials as before.
* fix(proxy): keep the listing failure log line the same for timed-out listings
* refactor(proxy): fold the cached listing lookup back into the provider listing function
* fix(proxy): list each model id once and keep a named deployment's row over a wildcard's
* fix(proxy): drop a wildcard's model info row when named deployments serve every name it lists
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): persist sent model_info fields on POST /model/update
POST /model/update rebuilt litellm_params from the request but dropped the
request's model_info, so changing base_model (or mode, tier, etc.) on an
existing DB model was silently ignored. Clients that only use this endpoint,
like the Terraform provider, could never fix a deployment's base_model
without deleting and recreating it.
Persist the model_info fields the caller actually sent, on top of the stored
ones. ModelInfo defaults never overwrite stored values, server-owned, team,
PTU and pricing fields stay on their dedicated write paths, and values echoed
from a /model/info response are not pinned.
* fix(proxy): keep echoed and blank model_info values out of POST /model/update writes
Blank strings from typed clients no longer overwrite stored values. Values echoed from
/model/info (the stored base model's cost map entry and discovered limits) are not
pinned, and an echoed catalog value resets a differing stored override like PATCH does.
direct_access and access_via_team_ids stay display-only. The tests stop patching the
auth check class and give the team lookup a real row instead.
* fix(proxy): lock the model row and skip unchanged model_info on POST /model/update
* fix(proxy): treat only keyed /model/update bodies as echoes and write the stored model_info as saved
A typed model_info value that matched the cost map, or the previous base model's entry,
was dropped and could erase a stored override. Echo detection now follows PATCH, only a
body carrying the /model/info key counts, compared against the base model the row has
after the write. The locked write patches the stored JSON instead of dumping the hydrated
Deployment, so litellm_params pricing is no longer copied into model_info. Tests run the
real Router with discovery over a mock upstream
* docs(terraform): note that model_info updates now take effect and overwrite values set outside Terraform
* test: expect POST /model/update to save the cache point flag, and wrap long test lines
test_x06 pinned the old contract, where POST /model/update answered 200 and ignored model_info.
This PR makes POST save model_info the way PATCH does, so x06 now asserts what its PATCH sibling
x01 asserts, the stored flag flips to true and cache points come back on.
* fix(proxy): answer 400 when POST /model/update gets an access_groups that is not a list of strings
* fix(proxy): read discovered model info through the router registry entry point
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(keys): allow assigning a project to an unassigned key on /key/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): suppress LIT002 on project assignment error payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): format project assignment error detail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(keys): assert accepted project assignment flows into update payload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): drop stale LIT002 suppressions removed by main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(keys): mock writer_db team lookup used by check_db_only path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): write project assignment only while key row is still unassigned
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): satisfy type gate on guarded project assignment write
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): default soft-budget update path to unguarded write
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): run guarded project assignment in one writer transaction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): include object permission link in assignment guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(keys): return permission-linked payload without mutating the update dict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): fix type errors and stale noqa after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): use Sequence in assignment guard model annotation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): type update fields as a mapping in the key update path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(keys): place cast reason on the cast call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): honor explicit null max_budget when assigning a project
* test(keys): assert explicit null max_budget flows into update payload
* fix(keys): match NULL models column in guarded project assignment
* fix(keys): authorize caller before project lookup on /key/update
* fix(keys): drop explicit null object_permission before project assignment write
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
