* fix(mcp): apply key and team guardrails to MCP tool calls
Guardrails attached to a virtual key or team were only enforced on LLM
routes. The synthetic request built for MCP tool call guardrail hooks
carried no guardrails in its metadata, so a guardrail with default_on
false never ran on tools/call even when the key explicitly listed it.
Resolve key, team, and project guardrails onto the synthetic request
with the same helper the chat path uses.
* fix(mcp): pass project metadata through without a mutable default
* fix(mcp): mark the request dict parameter mutable-ok with a reason
* test(mcp): explain the premium_user patch and tighten the helper docstring
The router code coverage gate reads every function defined in router.py
and fails when no test file names it. _as_retry_skipped_deployment_ids
was only reached indirectly through the retry path, so the gate went red
on this PR's tip.
Test it directly instead: a tuple of strings survives, non-string items
inside the tuple are dropped, and every other shape a caller could send
narrows to an empty skip list.
SPEND_LOGS_URL only diverts spend logs when db_writer_client is set, and nothing in the proxy ever assigns that global, so the queued copy was only ever skipped as a duplicate by the local insert.
The retry skip travels as a request kwarg, and the router forwards keys it
does not recognize, so a client can put _retry_skipped_deployment_ids in its
own request body. The value went straight into a pydantic TypeAdapter and
then into a set(), so an int or an object raised TypeError and a string, a
list, or a dict raised a ValidationError, each of them replacing the 400 the
provider had actually returned.
Every read now goes through one narrowing function that keeps a tuple of
strings and skips nothing otherwise, so a forged value costs the caller
nothing beyond the retry landing on the same deployment again.
disable_spend_logs has to keep meaning that no request gets logged, and the row
that makes a batch chargeable exactly once is the one row it cannot drop, so with
logging off that row now carries only what tells the retrieves apart. SPEND_LOGS_URL
deployments get their copy back too: the claim writes straight to this table, so the
row is queued as well when an external writer is the one that takes the spend logs.
A fake-streamed provider hands the adapter one chunk carrying both the
delta payload and the finish_reason, which is exactly what the combined
chunk splitter exists for, but its content check never listed the refusal.
The translation short-circuits on finish_reason, so that refusal text was
dropped and the client got `stop_reason: refusal` over an empty content
array, the symptom this PR set out to fix.
Both refusal accumulators also drop their `mutable-ok` lists for a plain
string attribute
Before excluding the deployment that just refused, the retry-skip guard asked
whether another one could still answer. It asked by re-running a single routing
filter, the order filter, while deployment selection also applies cooldowns, the
context-window pre-call check, tag routing, and routing plugins.
Any filter the guard did not replicate made it answer yes while the real pick was
left with nothing. A group narrowed to one deployment by tag routing turned the
provider's own 400 into a no-deployments 429.
The skip now runs where every filter has already been applied, and it keeps the
deployments untouched when skipping would leave none. The caller gets the
provider's error either way, and a group with one eligible deployment retries in
place as it did before.
The first-delta guard read `delta.refusal` directly, while the translation
three lines later goes through `openai_chat_refusal_text`, which also reads
the `provider_specific_fields` LiteLLM parks unrecognized fields in. A
provider that sends the refusal that way had its only refusal delta skipped
as blank, so the client got `stop_reason: refusal` over an empty content
array, which is the symptom this PR set out to fix
The /v1/messages adapter lowers a tier the entry does not accept, so dropping max from the astra
rows moves that path from Foundry's 400 to a request at xhigh. Nothing pinned that, and the guard
test's docstring named gpt-6-astra as the only gpt-5 name with an azure_ai row, which 11 rows
contradict.
The retry-skip guard checks that some other deployment could still answer
before it excludes the one that just refused, so a single-deployment group
keeps the old retry-in-place behavior. It asked that question at the group's
minimum order, but the router picks the retry's deployment at the order the
request has already escalated to.
So a group with a primary at order 1 and a backup at order 2 answered "yes,
order 1 still has a candidate" while the retry was pinned to order 2, and the
exclusion left order 2 with nothing. The caller got a no-deployments error in
place of the provider's own 400.
The helper now takes the active target order and filters by it, which is the
same value async_get_healthy_deployments reads off the request.
`vertex_ai/lyria-3-clip-preview` and `vertex_ai/lyria-3-pro-preview` were
registered with `supports_vision`, `supports_image_input`, and an `image`
modality, which contradicts their `gemini/lyria-3-*` siblings and makes
/model/info advertise image input on text-to-music models.
The new Lyria passthrough branch runs before the image-generation branch
and keys on the same `predictions[0].bytesBase64Encoded` shape imagen
returns, so only the cost-map lookup separates them. Cover an imagen
predict response end to end so a future change that drops that lookup
fails here instead of misbilling images as audio.
The takeover of a $0 row an older proxy left behind used to charge the batch when
the update could not reach the database. That leaves the row still reading $0, so
every later retrieve finds the same row and charges the batch again, which is the
repeat charging this PR exists to stop. The retrieve that does take the row over
is the one that charges, and a batch nobody retrieves again after that failure is
never charged, the same as one whose proxy died inside the write window.
CI's router_code_coverage gate wants every function in router.py called by
name from a test file with "router" in its name, and the new helper had no
direct caller, so the check-quality job failed on the first tip.
Covering it directly also turned up a hole. litellm._should_retry compares
the status code to 500, so a provider exception carrying a string status code
raises TypeError instead of answering. should_retry_this_error has the same
call, but the retry policy path skips it, which is exactly the path this
change enables, so the helper was the first to touch that value. Narrowing to
int leaves those exceptions on the old retry-in-place behavior.
Google prices Lyria per generated clip, so every Vertex Lyria entry in the
price map now carries a single output_cost_per_image and both the speech
and the passthrough cost paths read that one field. The old
output_cost_per_second and audio_seconds_per_prediction pair assumed a
30 second clip, which does not match the 32.768 second WAV Vertex returns,
and no other model in the map priced audio that way
Drops max_audio_length_hours and max_audio_per_prompt from the price map,
its schema, the generator, and ModelInfo, since nothing reads them, and
drops the audio_mime_type hidden param for the same reason: the response
already carries the resolved content type on its own header
Folds the per-model bundled catalog lookups into one cached parse of the
local cost map, validated with a TypeAdapter over a ReadOnly TypedDict
A proxy without this fix wrote the batch's cost row on every poll while the batch
was still running, so that row reads $0 and the insert that claims the charge has
nowhere to land. The retrieve that charges the batch now writes its own payload
over that row under a where clause that still names spend 0.0, so exactly one
retrieve takes it over and every later one reads the charge and charges nothing
gpt-6-astra is the only gpt-5-family name with an azure_ai row. Prefixing the rest
cost them every effort flag, since get_llm_provider sends an azure_ai name down the
azure provider when a global AZURE_AI_API_BASE points at an openai.azure.com host and
azure/<model> is not a key either, which turned temperature, top_p and logprobs on
azure_ai/gpt-5.1-chat-latest from accepted into an UnsupportedParamsError.
Both Azure routes refuse it. A live call to the same deployment through
openai/deployments/gpt-6-astra/chat/completions on api-version 2025-04-01-preview
answers reasoning_effort max with a 400 unsupported_value naming none, low, medium,
high and xhigh as the values it takes, and xhigh returns 200, so azure/gpt-6-astra
and azure/us/gpt-6-astra now match the azure_ai row.
BadRequestErrorRetries and ContentPolicyViolationErrorRetries did let a retry
happen, but the retry re-picked the deployment that had just refused, since a
400 never puts a deployment in cooldown. On a weighted model group the caller
got the same 400 back after every configured retry, and the existing 401/403
"retry on another deployment" rule broke the same way
A retry after a non-transient status now carries the deployments that already
answered this request in the per-request exclusion list weighted failover
already honors, so the next attempt lands on a sibling. Single-deployment
groups still retry in place, and 408/429/5xx retries are untouched
Adds live e2e coverage for reliability.retry.context_window.succeeds_within_retries
and renames the two litellm.utils deployment filters that are now called from
outside the module
A proxy running the old code wrote <batch id>_batch_cost at $0 every time it polled a batch that was still running, so after an upgrade the claim found that row and read it as proof the batch had already been charged. Only a row that recorded a charge counts now, which leaves those $0 rows, and any row a client planted under the batch id, to be charged over
disable_spend_logs skipped the claim entirely, so under that setting every retrieve of a finished batch charged again. The claim now runs either way and writes the one row per batch that makes the charge exactly once, while the per-request logs stay off
Both bridges opened an empty text block on a refused streaming turn and
closed it without a single delta, so a client replaying that assistant
turn got HTTP 400 "text content blocks must be non-empty" from Anthropic.
The safeguard-refusal fallback that motivated withholding the text only
runs on the awaited non-streaming response, so nothing needed it withheld
Move the refusal readers into the shared messages/utils helpers so the
adapters stop reaching into each other's private statics, which is also
what put reportPrivateUsage over its budget
Foundry rejects reasoning_effort max on the gpt-6-astra deployment with a 400 that
names none, low, medium, high, and xhigh as the supported values, so the card no
longer lists max. The request path never gated max (only xhigh is opt-in), so this
only changes /model_group/info and router capability gating. The azure/ twin stays
as is because it was not verified on an Azure OpenAI host
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object]
annotations instead of bare dict, and the docstrings added to the new tests go away
since the test names already say what they cover. No behavior change
Foundry deployments of gpt-6-astra reached through azure_ai used the bare OpenAI card
for the reasoning_effort none gates, so temperature and top_p were refused while the
azure_ai card says none is supported. AzureAIStudioConfig now dispatches gpt-5 series
params through AzureAIGPT5Config, which looks capabilities up under the azure_ai/
prefix the way the azure route does
Also carries the search_context_cost_per_query block azure/gpt-6-astra has, adds a
flex service tier cost test that fails at the merge base, and keeps the wildcard test
from stripping azure_ai/gpt-6-astra out of the provider set
The cost callback used to look for an existing `<batch id>_batch_cost` row before charging a
completed batch, which left a window where concurrent retrieves on any instance all charged the
key, and it would honor a row any request had written under that id. The spend update writer now
inserts the batch cost row itself with `create_many(skip_duplicates=True)` and only the retrieve
whose insert lands charges the key, team, and user. An existing row only takes the charge when it
is a successful `aretrieve_batch` row, so a client-chosen `x-litellm-call-id` on another endpoint
cannot suppress billing. Batch cost rows no longer get their own immediate flush path
`batch_cost_is_final` now treats the proxy's normalized `complete` status like `completed`, which
the enterprise batch cost poller relies on when it decides whether a completed batch is safe to
retire. Tests build that status with `model_copy` since the OpenAI `Batch` model rejects it
The `test-quality-ok` markers sit on the `patch(` lines the gate keys on, and the logging tests no
longer wrap the priced retrieve in `contextlib.suppress`
