* fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks
* fix(rate-limit): count embedding media as media only on models that embed it
A data URI sent to a text-only embedding model is read as text, so it keeps
the text estimate there. On models flagged supports_embedding_image_input,
image and audio use the chat media estimates and PDF and video count at most
the model's max_input_tokens. The same estimate now feeds project and key
limits, the usage-based, latency and cost routers' deployment tpm, and the
deployment itpm check.
* fix(proxy): count embedding media only where the backend reads it as media
Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as
media per item. Cohere counts it as media only when the whole batch is
media, and Voyage, OCI and unknown models keep the text estimate. File
blocks carrying a data URI count like the bare URI.
The router context window pre-call check and the batch limiter now use the
same estimate, resolving each deployment or JSONL row model to its
backends. The cost map gains supports_embedding_image_input on
voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0.
* fix(router): hold each embedding media item to the context window on its own
A PDF or video data URI alone already reserves at most the model's
max_input_tokens, but the context window pre-call check then ran the
blanked input through the Responses transform and added a few framing
tokens, so a PDF that Vertex accepts got a 400. When every input item is
media, the check now returns the largest item's media estimate, since an
embedding request embeds each item separately
* fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope
The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped.
* fix(bedrock): bill Cohere embeddings by Bedrock's input token count header
* fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage
The strict base64 check now scans the payload in place instead of decoding
and re-encoding a full copy. Bedrock's billed token header is used only for
image requests, so text-only Bedrock Cohere usage matches main. The deployment
itpm estimator change is reverted, since the router never passes embedding
input to that check.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): let the Admin UI cancel /team/spend/by_user at 60s with an opt-in statement timeout
GET /team/spend/by_user takes an optional timeout_seconds (1 to 600). When set, the raw
LiteLLM_SpendLogs aggregation runs under SET LOCAL statement_timeout and Postgres cancels it,
and the route returns 503 instead of letting the query run for hours. Callers that omit the
param are unchanged. The Admin UI sends timeout_seconds=60 and the Spend Per User Within Team
card shows the server's error instead of an empty table
* fix(proxy): cancel /team/spend/by_user in Postgres after 30s
Prisma's HTTP client already gives up on the statement at 30s, so the
query now runs under SET LOCAL statement_timeout = 30000 for every caller
and the opt-in timeout_seconds param is gone. The Admin UI card shows the
request error instead of the empty-state text.
* fix(proxy): keep /team/spend/by_user on the read replica inside its timeout transaction
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: show user names in access group details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: show user names on project, org, model and prompt details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): look up user names by exact id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fall back to user list for org admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fall back when /v2/user/info returns a different user id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): let org admins read their org members in /v2/user/info
Org admins could list a user through /user/list but got a 404 from
/v2/user/info for that same user, so the dashboard fell back to a
substring /user/list lookup that could miss the user. /v2/user/info
now allows an org admin to read users in organizations they administer,
and the dashboard's single-user lookup calls only that endpoint.
Expected 4xx responses from /v2/user/info no longer log a traceback.
* fix(proxy): drop unbounded IN from org admin check in /v2/user/info
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): record the provider's recommended replacement as successor_model on deprecating rows and report it from /model/deprecations
* fix(cost-map): pick the alias row's successor by its own shutdown date when the family has several
* chore(ui): regenerate schema.d.ts for the /model/deprecations successor_model wording
* fix(proxy): prefer a configured successor_model over the cost map's when deployments share a name, and keep the cost map's under a hand-set deprecation_date
* test(cost-map): allow successor_model in the cost map row schema
* feat(cost-map): infer successor_model as the next version in the same family for 136 dated rows with no published replacement
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit cost headers for unpriced responses
* fix(proxy): preserve declared free response costs
* refactor(proxy): share the cost map pricing lookup for the response cost header
The header check now reuses the same raw entry, fill-missing and rule lookup that
cost model selection uses, so the two can't drift apart. It also counts every declared
rate field, so a free model priced only per request keeps its 0.0 header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(databricks): mark audio and video input support on the Gemini 3.5 to 3.8 Flash rows
* fix(model_prices): add Vertex AI and Together AI retirement dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): correct perplexity agent api pricing and add gemini-4-argon, nemotron-3.5-super
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): consolidate verified pricing and Aplomb metadata
Absorb verified pricing from #49547 and #49548 and Aplomb 1 metadata from #49539. Preserve provider tiers and remove an identical duplicate retirement key.
Co-authored-by: Liutasil501 <schmid.martin501@gmail.com>
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(model_prices): sync generated EmpirioLabs endpoint metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): verify Nebius and EmpirioLabs model metadata
Verify canonical EmpirioLabs rows from #49695 against published model pages, preserving pricing tiers and excluding unverified variants and specialty billing. Add Nebius models and cached-input prices from the official catalog and correct the Voyage legacy input limit.
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): absorb Bedrock Opus 5 batch prices, Vertex Omni Flash limits and Gemini 2.5 shutdown dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): drop the EmpirioLabs rows and endpoint flags from this sync
* fix(model_prices): merge main and dedupe the gemini-2.5 deprecation_date keys
* fix(model_prices): set supports_video_input on the two new Nebius video models
* fix(model_prices): add Azure AI Claude Mythos 5.1 and match Foundry max effort support
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
* fix(proxy): list slash-less wildcard groups like claude-* on /v1/models and /model/info
* fix(proxy): expand team-owned wildcard rows on /model/info through their public name
* fix(proxy): keep listing wildcard names wider than their deployment and skip the Azure placeholder
A group like *gemini-3* over gemini/gemini-3* lists the catalog ids the name matches and routes back, as main did on /model/info, now on /v1/models too. Wildcard listings no longer list the Azure-LLM placeholder. Team-owned wildcard rows on /model/info go back to main's behavior.
* fix(proxy): keep Azure listings on slashed wildcards and list bare deployments through custom_llm_provider
get_provider_models returns main's list again, and only slash-less names skip the Azure-LLM catalog placeholder. A slash-less name over a bare wildcard deployment now lists the catalog of its custom_llm_provider.
* test(proxy): pin custom_llm_provider guards and slashed wider-name listing
* fix(proxy): list bare ids for wider names over a custom_llm_provider deployment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): carry prompt_cache_breakpoint to cache_control on chat completions to Claude
A content part carrying OpenAI's prompt_cache_breakpoint on a /v1/chat/completions request to Claude (Anthropic, Bedrock Invoke and Converse, Vertex AI, Azure AI) now reaches Claude as cache_control {"type": "ephemeral"} on the same block, so prompt caching works for OpenAI-shaped clients instead of the marker being stripped.
Only parts OpenAI accepts the marker on are translated, a client cache_control on the same block wins, and Converse translates only for models that take cache points. A 30m ttl or a malformed marker answers UnsupportedParamsError without drop_params and becomes a default 5m mark or no mark with it.
* fix(anthropic): hold translated prompt_cache_breakpoint marks to the 4-breakpoint cap
Marks past Claude's 4 cache breakpoints now answer 400 without drop_params and
keep the earliest under it, counting the client's own message, tool, and
top-level cache_control marks. The ttl and malformed-marker errors name
drop_params for proxy users too and no longer name a single provider.
* fix(bedrock): count Converse tool cache points in the breakpoint cap and gate unmapped models
A client tool cache_control or a tool_config injection point becomes a Converse tool
cachePoint, so it now takes one of the 4 slots before a translated marker does.
Converse only translates prompt_cache_breakpoint for a model the cost map says takes
prompt caching. An unmapped model such as an application inference profile ARN opts in
through model_info supports_prompt_cache_breakpoint, so a marker sent to every model of a
mixed fleet no longer turns a non-caching profile's 200 into Bedrock's 403.
Router prompt-caching affinity now treats a marked block like a cache_control one, and
requests without a marker skip the translation after one pass over their messages.
* fix(bedrock): count only the tool cache points Converse sends against the breakpoint cap
* fix(anthropic,bedrock): cap only the cache breakpoints each provider sends, and drop marks Nova cannot take on tool results
* perf(anthropic,bedrock): find the breakpoint cut-off by binary search so trimming many marks stays fast
* fix(anthropic): drop the duplicate drop_params left by the main merge
* fix(bedrock): let an inference profile's cache breakpoint flag win over its base_model
* fix(anthropic): keep the first and last prompt_cache_breakpoint marks past the cap
Under drop_params the translated marks that fit the cap are now split between
the start and the end of the request, so the stable prefix and the newest turns
both stay cached. The cap count also includes a native tool search tool that
keeps its cache_control on the wire.
Bedrock Converse now translates the marker only for the Claude and Nova
families, which Bedrock documents as taking cache checkpoints, unless the
deployment sets supports_prompt_cache_breakpoint. A guarded user turn keeps its
cache mark when the request fits the cap.
* refactor(bedrock): decide guarded cache marks with a typed bool helper
* fix(bedrock): validate guarded blocks before reading their cache mark
* fix(anthropic): let injection points replace client marks and keep non-Claude routes untranslated
When a deployment sets cache_control_injection_points, the client's
prompt_cache_breakpoint marks are dropped on targets that do not read them, so
the operator's injection points decide every Claude breakpoint.
OpenCode Go models served over the Anthropic messages endpoint translate the
marker only for Claude, and guarded Converse text keeps a cache mark only on a
model known to take cache points.
* fix(anthropic): keep client marks unless a message injection point replaces them on a translating route
A tool_config-only injection point no longer strips client prompt_cache_breakpoint marks, a flagged Bedrock profile no longer counts as reading the marker natively, and guarded Converse text skips the tool count when no message carries a cache mark.
* fix(anthropic): keep Bedrock and Claude deployments flagged for the marker on cache_control injection
A supports_prompt_cache_breakpoint flag on a Bedrock profile or a Claude deployment made the injection hook write OpenAI markers and prompt_cache_options, which Bedrock answers 400 for. Routes that translate the marker now never take the OpenAI dialect.
* fix(bedrock): reserve only the tool cache points Converse sends when trimming Claude cache marks under drop_params
* fix(anthropic): name the marker's accepted ttl values in the malformed prompt_cache_breakpoint 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API
* fix(logging): log bridged chat completions with the chat response they returned
* fix(logging): log bridged text completions and /v1/messages as served, with the request's options
Bridged rows were translated without the request's options, so structured
output and replay fields could differ from what the client got. Text
completions and non-streamed /v1/messages through the chat bridge also kept
a Responses API object or the encoded id.
* fix(logging): redact reasoning summaries on bridged rows and log bridged text completions as text completions
Redaction now drops reasoning_items on chat messages and deltas, so a bridged row keeps no reasoning summary when message logging is off. A non-streamed bridged text completion logs the text completion the client got, priced like the chat response it came from. The bridge handler's input TypedDict fields are ReadOnly.
* fix(logging): redact bridged text completions for opted-out callbacks and pin the sync logged shape and usage ack
* test(integration): look up bridged chat spend rows by the id the client got
Bridged chat spend rows are keyed by the client-facing id and logged with the caller's call type, so the helpers that decoded the old resp_<base64> key or pinned call_type responses now assert the new contract
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): spell "environment" right in the allowed_ips and AWS KMS V2 license errors
* fix(proxy): read general_settings.litellm_license before the allowed_ips and enforced_params gates
* fix(proxy): read the config license before callbacks and gates, keeping a valid env license
* fix(proxy): keep an invalid config license reported and sync license data with the license in use
A config license that fails no longer overwrites the license string when the env license is not valid either, so /health/license still reports has_license. A failed boot-time env license is not re-checked. Both license writes go through one helper that also refreshes premium_user_data, so /health/license and the seat limits follow the license in use.
* fix(proxy): keep the env license's boot result when a config license fails, without a second license check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure_ai): keep azure_ad_token out of the request body
azure_ad_token is not a litellm-owned param, so get_optional_params routes it
into extra_body and the azure_ai chat and Claude transformations merge it into
the JSON body. Azure AI Foundry then rejects the call with 400 Unknown
parameter / Extra inputs are not permitted. The Authorization header already
comes from litellm_params, so drop the extra_body copy once at the top of the
azure_ai dispatch, as the azure branches do.
Type litellm.open_ai_chat_completion_models and
open_ai_text_completion_models as Set[str] so the basedpyright gate does not
grow from the untyped optional_params read.
* fix(azure_ai): keep Entra credentials out of every azure_ai request body
* fix(azure_ai): keep the Entra token out of the image edit form
* fix(azure_ai): keep Entra credentials out of image edit extra_body too
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000
The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry
An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused
* fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed
When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input
The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts
A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged
The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success
* fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row
A realtime session's success event now reports status failure, with the same
error_information the spend row stores, when the session recorded a provider
failure in its metadata. Usage already streamed stays billed and the router
still counts one cooldown failure
The spend row builder now runs every row's error_information through
sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction),
so the realtime failure rows get the same treatment as the proxy failure hook,
which no longer sanitizes on its own to avoid truncating twice
Drop the selection-failure tag on the committed-session refusal, since the
shared logging object already keeps a retried refusal out of cooldown, and pin
the router's realtime success counting with an OpenAI realtime deployment
* fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops
The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop
* fix(realtime): count a router success only for a session that ended cleanly
A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had
A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers
* fix(bedrock): log a dropped realtime session's usage from its own logging state
The failure event replaced the shared standard_logging_object while the
queued success dispatch was still running its callbacks, so loggers
received $0 and 0 tokens for a billed dropped session.
* fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success
A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed
* fix(rate-limit): settle each token reservation once when a request logs both a failure and a success
A dropped realtime session logs a success for its usage and then a failure. After a success the
limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing
twice. A new reservation written in the same context after a released one now starts clean, so
reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or
28 ITPM for 27 and 20 used
* fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once
* fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge
* test(rate-limit): pin the split hold-release flags across nested MCP pre-calls
* fix(realtime): count a relay that fails on its own as an abnormal end
* test(realtime): pin the relay failure exit with a client frame the provider rejects
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry
Two deployments that share one model string share one key in the live
litellm.model_cost map, and router registration writes each deployment's
supports_prompt_cache_breakpoint into that key. The cache control hook read
the live map, so a flagless deployment picked up a sibling's opt-in and sent
prompt_cache_options and prompt_cache_breakpoint upstream, which the provider
rejects with a 400
When a call carries the routed deployment's model_info, the prompt cache
checks now use its declared supports_prompt_cache_breakpoint, and fall back
to the loaded catalog instead of the live map. This covers the hook on
completion, acompletion, Responses and the /v1/messages route, the azure_ai
chat transform, the OpenAI Responses transform under drop_params, and the
/v1/messages to Responses adapter
* fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers
A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's
prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a
provider other than openai, such as azure_ai OpenAI models and bedrock_mantle.
Bedrock on every route and providers that build Anthropic messages keep
cache_control, which Bedrock Converse turns into cachePoint
The openai chat reader now passes the routed model_info and checks base_model.
The Bedrock cache point reader takes the deployment's flag first and, for a
routed call with no flag, reads the catalog as loaded instead of the live map
that sibling deployments write into. model_info is threaded from the converse
transform, the factory message and tool builders, and the responses bridge
* fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls
A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations.
* fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways
* fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases
* test(anthropic): cover auto caching on an unmapped Claude deployment
* fix(cache_control): keep the merge base's reads where the per-deployment flag changed them
A register_model entry under a provider-prefixed name reaches routed calls again, since the
overlay now resolves the bare and prefixed names the way runtime registrations are stored.
The hook and the openai chat reader no longer judge base_model, as at the merge base, so a
client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint
and get a 400.
A Bedrock deployment flag that is not a real true reads as false again, as at the merge base.
An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again,
as at the merge base. Other wildcards still leave both cache flags to the catalog.
The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag
on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps
its own flag and that a flagless unmapped ARN keeps its cache points after the delete.
* fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name
Routed calls read register_model entries through an overlay that merged the bare
name after the provider-prefixed one whatever order they were registered in, so a
later opt-out under one name lost to an earlier opt-in under the other. Each
registration now moves its key to the end of the kept registrations, and the
overlay merges both names oldest first, which matches what SDK calls read from
the live model map.
Also lists the wildcard cache flag helper in the router coverage script, since
the hook's wildcard tests drive it rather than calling it by name.
* fix(cache_control): keep register_model entries visible while a cost map reload replays them
The previous commit moved a re-registered name to the end of the saved
registrations by removing and re-adding it, and the overlay walked that dict to
order a bare and a provider-prefixed name. A reload replays every saved entry,
so a request on another thread could briefly miss an entry or fail on a dict
that changed size mid-walk. Entries are written in place again, a per-name
sequence records which registration came last, and both the overlay and the
reload replay order by it, so the later registration still wins after a reload.
* fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix
A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(realtime): support latest OpenAI audio models
* fix(transcription): type SDK stream duration metadata explicitly
* fix(realtime): preserve transport settings and translation duration
* refactor(realtime): reuse SDK client initialization
* fix(audio): bill finalized streams and authorize transcription models
* refactor(audio): keep transcription pricing selection together
* fix(realtime): reconcile model metadata and test contracts
* fix(realtime): preserve provider duration and model capabilities
* fix(realtime): keep transcription validation compatible with cost map guard
* fix(realtime): address transcription and billing review findings
* fix(realtime): preserve input usage and provider routing
* fix(realtime): bill retained output and route Azure aliases
* fix(realtime): reject malformed provider durations
* fix(realtime): preserve batch cache rates and Azure v1 routes
* fix(transcription): reject nonfinite provider durations
* refactor(realtime): align annotations with current lint rules
* fix(realtime): preserve translation deployment pricing and input usage
* test(transcription): verify synchronous stream finalization
* fix(realtime): retain missing translation input usage at close
* fix(transcription): reject streams that bypass output guardrails
* fix(realtime): reconcile audio protocols billing and current CI
* fix(realtime): trust confirmed formats and authorize default models
* fix(realtime): preserve absent models and reject unsupported guardrails
* fix(realtime): reject translation policy pipelines
* chore(ci): rerun transient infrastructure timing failure
* fix(realtime): reconcile audio protocols with current upstream
* fix(audio): validate provider responses at SDK boundaries
* fix(realtime): preserve minimum OpenAI SDK compatibility
* fix(realtime): preserve deferred Live session authorization
* fix(realtime): preserve upstream authentication and transcription behavior
* fix(audio): scope provider duration precedence to configured models
* fix(typing): avoid Final declarations inside transcription cost loop
* refactor(audio): reuse transcription mapping and duration resolution
* fix(realtime): enforce model budgets and preserve translation routing
* fix(audio): log streamed transcripts whose done event reports detected languages
* fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers
Providers without a streaming transcription handler got UnsupportedParamsError
for stream=True, languages and keywords, where the merge base forwarded them as
provider-specific params. Only openai and azure validate them now.
* fix(audio): bill streamed gpt-transcribe from the provider's reported duration
The stream path computed cost after logging had replaced the transcription's
duration usage, so it fell back to the measured file length while the
unstreamed call billed the provider's seconds.
* fix(transcription): log streamed transcription usage from the provider response
The stream logging path built the spend payload from the response after
its duration usage was transformed away, so whisper-1 with stream=true and
streamed gpt-transcribe logged an empty usage object. Bill and log from the
provider's TranscriptionResponse in both the sync and async paths.
* fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes
- Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape
- Use the input transcription model as the session model only for transcription sessions
- Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main
- Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set
- Return x-litellm headers on streamed transcriptions and mark the request done
- Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure
- Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing
- Bill translation sessions on output audio seconds only, at the single published rate
- Document the 201 SDP answer on the translation calls routes and the transcription 200 response
* fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients
Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the
guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the
default dated version on the deployments route for every model. The SDK client secret keeps
session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata
keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two
unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the
gpt-transcribe format check only reads the request provider's cost map entry.
* fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it
Transcription reaches every provider as on main again, sends stream only when asked, forwards language and
languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the
new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a
mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from
one source and spend rows carry the billed seconds.
Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward
explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths
are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later
session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA
get beta-shaped session events, and transcription sessions send the deployment model in both shapes.
* fix(realtime): address review findings on transcription billing, streaming and realtime auth
Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata
* fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only
* fix(realtime): preserve prior merge resolutions after rebase
* fix(realtime): enforce translation policies and normalize Azure transcription URLs
* fix(cookbook): stop translation once output idles after the upload ends
* fix(realtime): bill translation on input audio and drop pre-read stream cost headers
Translation spend now uses the input audio seconds the session sent, priced
through input_cost_per_second, instead of the translated output duration.
A streamed transcription no longer returns cost headers before the stream is
read, since the cost is only known after the last event. Also removes an
unused realtime guardrail helper.
* fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields
Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop
* test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription
* fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA
Bill translation sessions from the deployment's own per-second rates, add per-second prices for
Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map
mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to
the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation
calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models
* fix(realtime): bill a translation rate the deployment omits from the session model
* test(e2e): check translated speech is not English instead of exact French wording
* fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices
Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending
* fix(realtime): keep the proxy's realtime session gate off translation sessions
A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects
* fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model
A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params.
* fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree
* fix(model-catalog): accept the realtime translation endpoints in the cost map
* fix(cookbook): close the translation session and read until session.closed
* fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard
An empty `stream` form value runs the transcription unstreamed again, as
OpenAI does, instead of answering 400. The four translate cost-map entries
drop the translation routes from supported_endpoints until the main
branch's catalog parser accepts them, and the nested transcription model
lookup uses the router registry entry point.
* test(proxy): drop the mid-stream error frame pin that main's error shape change replaced
* fix(transcription): bill uploaded seconds when a stream fails mid-transcript
* fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream
* fix(audio): keep transcription's positional arguments and redact streamed error details
- Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning
- Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do
- Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite
* fix(audio): stream the guarded transcript to SDK callers that asked for a stream
* fix(logging): keep a streamed transcript out of callbacks that turned off message logging
A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does
* fix(proxy): stream a transcription the deployment streamed when the client sent no stream field
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): send an https PDF image_url to Claude as a document
A PDF link sent as image_url to Claude on Anthropic went out as an image
block and Anthropic answered 400. A link whose path ends in .pdf is now
downloaded like Vertex links are, and the served Content-Type picks the
document or image block, so CDN links that serve a PNG from a .pdf path
keep working. An explicit format of application/pdf sends a url document
with no download. PDF media types match regardless of case or parameters,
and count_tokens picks the same block as the request.
* fix(anthropic): download https PDF links off the event loop
The proxy downloaded an https link ending in .pdf synchronously while
building the Anthropic request, so on an async call every other request
on that worker waited for the download. Those links are now inlined
before the request is built, the same way http links already are, and
the /v1/messages guardrail write-back does the same. A non-string
format on an image_url is ignored the way it already was inside the
image_url mapping, instead of raising an AttributeError.
* fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent
* test(anthropic): expect the PNG media type the served bytes carry at a .jpg path
* fix(anthropic): bound the PDF link download and keep query-string PDF names
Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the
client's 6000 s default, fall back to main's whole-URL extension check so
download?file=report.pdf still counts as a PDF, treat application/x-pdf as
a PDF, and set the url document's application/pdf format only for the
guardrail write-back so other /v1/messages providers send what main sends
* fix(anthropic): probe a .pdf link and send it to Claude as a url document
Instead of downloading a .pdf link and inlining it as base64, ask the host for
its first kilobyte and send the link as a url document when it serves a PDF.
Every other link goes out as the url image main sends, so robots.txt handling,
caching and the wire format stay as they are on main.
Also restore the guardrail adapter and the content type inference to main,
keep the client's url document blocks (title, context, citations,
cache_control) through the guardrail write-back, treat a PDF filename in the
query string as a PDF link, probe links concurrently on the sync path, cap the
async probe's read at the download limit, and catch only transport, URL and
SSRF failures in the probe
* fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place
The probe now streams the response and keeps the first 1 KB, so a host that
ignores Range no longer fills memory with up to 20 parallel bodies. The
guardrail write-back now matches url documents inside a tool_result and outside
one separately, since the write-back moves tool results ahead of other content
* fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke
* fix(anthropic): keep a PDF probe's error status for a minute, not ten
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(embeddings): honor the caller's timeout on Cohere sync and HuggingFace embedding requests
Bedrock Cohere embeddings ran the sync request with the client default instead of the timeout
passed to litellm.embedding, and HuggingFace embeddings dropped it on both the sync and async paths.
Pass the timeout through to the HTTP request on each path.
Fixes#47301Fixes#47299
* fix(huggingface): pass the caller's timeout to the embedding model lookup too
Without input_type the handler first fetches the model's pipeline tag from huggingface.co, and that GET still used the default timeout. Thread the caller's timeout through both lookup helpers and the transform step so every request the embedding makes honors it.
Refs #47299
* fix(huggingface): keep the unreachable async lookup path untouched
The async model lookup helper and the call_type="async" transform branch are never reached, since the async embedding path transforms through the sync lookup. Leave them as they were so the change only touches requests that actually run.
* feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos
A provider registered in custom_provider_map with a BaseVideoConfig handler now resolves on every
video call type (create, status, content, list, remix, edit, extension, characters), the same way
CustomLLM handlers already resolve on /v1/images/generations. The video entry points and the HTTP
handler stop casting the provider name to LlmProviders, which raised before any lookup ran, and a
provider without a video config gets a 400 instead of a 500.
* fix(videos): type the custom provider lookups so the type gate holds
Pick the CustomLLM for a custom provider through ProviderConfigManager.get_custom_llm_handler,
which narrows the now wider custom_provider_map handler union, read custom_provider_map instead
of the private _custom_providers list, and annotate the untyped custom_llm_provider parameters
on the video entry points
* test(videos): cover a CustomLLM shadowing a built-in video provider name and type the new tests
Mark custom_handler ReadOnly on CustomLLMItem, type the new tests, and assert that a CustomLLM
registered under openai still leaves openai video on the built-in config
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Runs every case through the Python /v1/messages path and the Rust path against one recording upstream and compares dispatch, upstream request, outcome, hidden params, standard logging fields and callback kwargs. Known Rust differences are strict xfails tagged with their inventory ID.
* feat(decisions): add Parallel AI web-grounded decisions
* fix(decisions): name the Parallel model decision
* fix(parallel_ai): restrict environment credentials to trusted decision endpoints
* fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence
Only fields a provider names reach the OpenAI response, so existing providers keep
their base shape. A malformed api_base answers 400 on both credential paths. The
complexity router classifier and the Playground handle a provider that omits
confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types
* chore(cost-map): declare the Decisions routes on parallel_ai/decision
The provider endpoint files are generated from the cost map, so the row must
name both routes for the generated System One flag to hold
* fix(decisions): report no classifier confidence when a provider omits it
A choice answer with no confidence is a valid classifier answer, so the
complexity router picks its tier and leaves classifier_confidence and the
confidence signal out instead of reporting a different metric in its place.
Also show the Parallel AI logo in the provider picker and add a Parallel
translation fixture captured from the live API for both Decisions routes.
* fix(decisions): keep a token-priced row on token billing when it also sets a query price
A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR
* fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route
A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions.
* fix(decisions): keep per-query billing and optional confidence on Parallel only
A query price on any other Decisions provider moved that provider off token
billing, and a choice or score answer with no confidence became a 200 with a
null for every provider. Both now apply to Parallel alone, so other providers
bill and fail as they do on main.
A Parallel deployment with explicit zero token rates and a query price read as
free, so every budget check was skipped while it billed per call. It now counts
as priced.
The dashboard key for Parallel is now PARALLEL_AI so a served bundle that
predates the provider still sends parallel_ai as the provider slug, and the
README table lists the provider.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): expand wildcard models for providers with model info but no cost map entry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): stop requiring an api key to list vllm models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve model ids during wildcard expansion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): forward api key for model discovery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): support bearer auth for model discovery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): list hosted_vllm models from the server chat uses, with a bounded wait
The hosted_vllm listing now resolves HOSTED_VLLM_API_BASE and HOSTED_VLLM_API_KEY like hosted_vllm chat, instead of the /vllm pass-through's VLLM_API_BASE, and waits at most NON_LLM_CONNECTION_TIMEOUT rather than the completion timeout. vllm/* keeps listing nothing, since its chat runs in process. The router pass-through now forwards the deployment's api_key.
* fix(vllm): skip the placeholder key, honor deployment TLS and headers, and retry failed listings after a minute
* fix(discovery): cache only timed-out model listings, so a fast failure is tried again on the next call
* fix(proxy): list wildcard models off the event loop, so a hung provider no longer stalls other requests
* fix(utils): keep a concurrent call's model listing when another call times out
* fix(proxy): run model discovery on its own thread pool and keep credential lookups off the event loop
* fix(proxy): stop the file credential lookup at the first match before listing wildcard models
The fallback lookup now walks the allowed model names lazily, in the same
order as before, so a hung vLLM server is listed only when no configured
name serves the requested provider. The vLLM auth headers no longer
mutate the caller's headers dict.
* fix(proxy): try every configured wildcard route before listing any wildcard's models in the file credential lookup
* fix(proxy): answer model lists from cached listings first and share one listing per server
* test(vllm): give the self-signed cert fixture fixed validity dates
* fix(proxy): skip wildcard servers a key cannot reach on /v1/model/info
/v1/model/info listed every wildcard deployment and filtered by the key
afterwards, so a key limited to one model waited on every wildcard server and
sent listing calls to servers it cannot use. The listing now skips a wildcard
whose patterns match none of the key's models before it asks the server.
A key or team granted hosted_vllm/* with no deployment of that name now lists
no models, like openrouter/*, since chat answers 400 for every id the
environment server would list.
* fix(proxy): match a wildcard's own row pattern before asking its server
The /v1/model/info pre-filter matched a key's models against both the
deployment name and its litellm_params.model, so a key granted
hosted_vllm/* still asked a vllm-secure/* -> hosted_vllm/* server whose
rows it can never see. It now matches only the pattern expansion names
the rows with, picked by one shared helper.
* test(vllm): cover the vLLM listings and no-slash wildcard aliases the patch left unrun
Adds tests for hosted_vllm get_models with and without a server, vllm/*
never asking a configured server, and /model/info pre-filtering of
wildcard aliases whose name has no slash. Drops a test comment that
only repeated a method signature.
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): bill web search interception follow-ups once under the outer request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): keep web search rounds inside interception follow-ups billed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): keep the outer request as parent for nested interception follow-ups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(spend): drop parent override for nested interception follow-ups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): reproduce comparison response ID ownership and wire failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): guard stale retries and malformed comparison envelopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): authorize and unwrap comparison response ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): require comparison IDs to follow response security policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): deny raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): reject raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): enforce managed ownership with unmanaged ID opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date
* fix(proxy): report a deployment's own provider retirement date before the bare model's
An Azure deployment declared as model gpt-4o with custom_llm_provider azure, or as
azure/<deployment> with base_model gpt-4o, now resolves azure/gpt-4o's date instead of
the bare OpenAI row's. The GPT-4 series alias test now reads both cost maps with typed rows
* fix(proxy): read Azure's retirement schedule for azure_ai deployments
An azure_ai deployment with no azure_ai row of its own now reports the azure/<model> date, and a cost-map row wins over the provider-less copy the router registers under the deployment's key, which carries the bare model's date.
* fix(proxy): let custom and registered base_model rows keep their deprecation date
Only the copy the router writes at startup, which has no provider and inherits the bare row's date, gives way to a later candidate
* fix(proxy): report a pinned snapshot's own earlier deprecation date over its base_model's
* fix(proxy): recognize the router's copied cost-map rows by the loaded catalog
* fix(proxy): read Azure's schedule for a provider-prefixed azure_ai base_model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tool_choice): accept OpenAI allowed_tools on chat completions and flatten it for the Responses bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the allowed_tools tool_choice fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: raw regex for the allowed_tools provider gate match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): move the allowed_tools provider gate out of completion() so pyright can still analyze it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): keep the allowed_tools helpers typed so the basedpyright gate stays level
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): type the allowed_tools validation boundary with Any instead of casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tool_choice): widen tool_choice to object at the allowed_tools helpers with a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tool_choice): drop allowed_tools test docstrings, type the new tests, and revert unrelated reformatting
Also reword the allowed_tools 400 so it no longer blames the flat Responses layout for every malformed choice
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool
Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking
sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model
map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The
GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows
* fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool
Native output without thinking now applies to every Bedrock Claude model that cannot
force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5
needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose.
The invoke chat path keeps the native output_format it built instead of inlining it, and
GovCloud region-path ids read their own false rows.
* fix(bedrock): clean caller output_format schemas before sending them natively on invoke
* fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke
* fix(bedrock): keep the faked Converse stream for native output that replaces the json tool
A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture.
* test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(anthropic): pick beta allowlist row per deployment, add mid-2026 beta rows, surface input_transformations
Foundry-hosted anthropic deployments now filter anthropic-beta by the azure_ai row on /v1/messages and chat completions, the bundled config gains the mid-conversation-system-clear-at-2026-08-21 row plus the tool-changes and output-config betas on azure_ai and vertex_ai from a live probe, and Anthropic's input_transformations array reaches chat completions in provider_specific_fields (non-streaming and the first streaming chunk).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): type the provider_specific_fields source dict and beta row test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): input_transformations reaches chat completions and /v1/messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): read the Anthropic response as a Mapping when building provider_specific_fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): fold deployment-row tests into the filtering file and mirror the new beta in the Rust table
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drop the duplicated beta tuple and mark filter results Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): forward the seven betas Foundry accepts on the azure_ai beta row
A Foundry-hosted anthropic deployment now filters through the azure_ai row, which
dropped compact-2026-01-12, fast mode, fine-grained tool streaming, inline tools,
oauth, output-128k, and token-efficient tools. Foundry answers 200 to each, and
without compact-2026-01-12 a compaction request answers 400, so the row now allows
all seven on both the Python map and the Rust table
* fix(anthropic): keep pass-through and beta-less requests off the Foundry beta row
An openai/ deployment that passes /v1/messages through to a Foundry host lost
every beta, and a request without betas read the beta table from GitHub.
Only an anthropic deployment on a Foundry host now takes the azure_ai row,
and the check reads the provider and host before the table.
* test(anthropic): point the beta-gated wire legs whose row lacks clear-at at a native Databricks deployment
* style(rust): wrap the azure_ai beta header test case the way rustfmt does
* fix(anthropic): forward unlisted betas to Foundry and keep Foundry calls off the Rust messages route
An anthropic deployment on a Foundry host now drops only the betas the azure_ai row marks
unsupported on /v1/messages, and forwards betas the table does not list as sent, as main did.
cognitiveservices.azure.com counts as a Foundry host. The Rust messages route sends an async call
with a Foundry api_base to Python, which filters by row. A sync call stays on Rust, since Python has
no sync /v1/messages handler. The json_mode chat response keeps input_transformations,
and the first /v1/messages call with betas loads the beta table off the event loop.
* fix(anthropic): run the advisor loop for an anthropic deployment on a Foundry host
The advisor interceptor treated every anthropic deployment as native, so a Foundry-hosted one
received the advisor_20260301 tool and Foundry answered 400. Interceptors now get the
deployment's beta row from resolve_beta_provider, so that deployment runs litellm's advisor
loop the same way an azure_ai deployment does
* test(anthropic): drive the Foundry advisor loop test through recorded HTTP
The test now intercepts the Foundry and Anthropic HTTP calls with respx instead of patching litellm's own sub-call helper, so it checks what Foundry receives: no advisor_20260301 tool and no advisor beta
* fix(anthropic): leave the advisor loop to the providers it ran for on main and tidy the PR's tests
Reverts 083990f6d7, since review found it reaches a pre-existing gap that is now with security engineering.
The beta row tests drive a real HTTP transport, the beta table fetch test checks the fetch thread instead of a 2 second deadline, and the touched test files keep main's formatting
* test(anthropic): give the Foundry beta row logging fixture a fixed datetime start time
* test(anthropic): check that chat completions pick the beta row by api_base
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: Return Anthropic error envelopes on Messages auth failures
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Answer Anthropic envelopes on Messages auth failures without relabelling unmapped statuses
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): answer the Anthropic error envelope on the /anthropic pass-through
The pass-through relays upstream errors in Anthropic's shape, but the proxy's
own refusals there (bad key, model access, budget) still answered the OpenAI
body, so an Anthropic SDK pointed at /anthropic could not read them
* fix(proxy): answer the Anthropic envelope for 404, 405 and unhandled 500 errors on Messages routes
A disabled Claude Code gateway 404, a wrong-method 405 and an unhandled 500 on the
Messages and /anthropic/ routes kept FastAPI's detail body or the OpenAI shape. The
central HTTP and unhandled-exception handlers now pick the Anthropic envelope by route.
The dashboard playground toast titles Anthropic error types when the body has no code.
* fix(proxy): keep LiteLLM error types and Anthropic envelopes on Messages routes
Error bodies on the Messages routes now carry error.litellm_error_type when the
LiteLLM type differs from the Anthropic type, so clients and the Admin UI keep
causes like budget_exceeded. Unlisted 4xx statuses map to invalid_request_error
per Anthropic's errors page, model-list requests that send anthropic-version get
the Anthropic envelope, and the admission, decompression and request-size
middleware errors use it on these routes too.
* test(proxy): expect litellm_error_type in Messages integration tests and type middleware tests
* fix(proxy): keep handler error bodies unchanged and timeouts as api_error
Errors the Messages and count_tokens handlers already returned in the Anthropic
shape keep their original bodies, so litellm_error_type now appears only on
errors the proxy raises before the handler. A 408 stays api_error as before,
and the playground toast titles the new Anthropic types and ignores long
numbers in front of an embedded error
* fix(proxy): keep the base error types inside the Messages handler and scope toast titles to Anthropic bodies
The wider status table (402 billing_error, 409 conflict_error, 504 timeout_error,
other unlisted 4xx invalid_request_error) now applies only to errors the proxy
raises before the handler. Errors inside the Messages handler, streamed error
frames and the SDK streaming adapters keep the merge base's table.
The dashboard toast reads a status before embedded JSON and titles by Anthropic
type only when the body is an Anthropic error envelope, so other pages keep
their titles.
* test(anthropic): mark the new test variables Final
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): add the Codex file edit and long session rows
The file edit row has Codex change a random value into config.txt with the patch tool and checks the file on disk. The long session row has it follow a chain of random file names one cat at a time. new_workspace now returns the resolved path, because Codex reports patched files by their real path (/private/var on macOS).
* test(e2e): make the Codex long session prove every earlier command output reached the model
* test(e2e): Codex auto-compaction through LiteLLM (#49389)
* test(e2e): add the Codex auto-compaction row
Codex reads a random value, compacts through the gateway with its remote compaction forced on, and is asked for the value after the file holding it is deleted. The row reads the compaction from the thread's rollout file and marks Claude on Bedrock as a strict expected failure, since the Responses-to-chat bridge ignores the compaction_trigger item
* refactor(e2e): validate Codex rollout lines with the models directly
* test(e2e): drive the real Codex CLI through the gateway and add the command round trip row
Adds tests/e2e/codex: a harness that runs codex exec --json with a throwaway HOME and CODEX_HOME, a custom responses provider pointed at the gateway, the model list read from the gateway's /v1/models, the sandbox on and approvals off. Each test registers its setup's deployment via /model/new and calls it with a key scoped to it, both deleted on teardown. The first row checks that a value only a command can print comes back through the gateway on all five setups
* test(e2e): type the Codex self-test event helper as a Mapping
* test(e2e): clean up Codex test keys even when setup fails, and fail runs where Codex exits with an error
* test(e2e): record a step when the Codex harness sets up its home and workspace
The suite rules ask every public harness helper that does something to record a step, so a failed run shows whether it died while setting up Codex or while running it
* Update run.py
wip
* fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): resend a cached-prefix signature inline and cover native generateContent
When the rejected signature sits inside a cachedContents prefix, the
placeholder resend now carries the history inline, since replacing the
signature in the request body cannot reach the cache. The native
generateContent handlers get the same one-time resend.
* fix(gemini): send the history inline when Vertex rejects a signature while creating the cache
* fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(managed_files): keep the upload the managed id encodes when two target models share a deployment
* fix(proxy): skip cost tracking for a file delete routed to a deployment
* fix(managed_files): delete repeat uploads after the upload answers
Run the repeat-copy deletes together in a background task so a slow
provider delete no longer holds the upload's response, log the unbilled
file-delete cost skip at debug, and type the new test helpers
* fix(proxy): run managed file repeat-upload cleanup as a drained internal call
Delete repeat uploads after the record is saved, through the shared background
task list the shutdown drain waits on, and as an internal call so logging,
spend tracking and alerts skip it. Leave litellm_proxy deployments on the old
keep-last behavior, since their per-target copies are distinct files. Revert
the cost tracker skip change, which the internal call makes unnecessary.
* fix(managed_files): delete the copy a managed id names on DELETE and isolate cleanup metadata
* fix(managed_files): read the copy DELETE removes from the id's gateway-written fields
* fix(managed_files): delete the copy a managed id carries as an internal call
* fix(managed_files): pass the parent call id to the cleanup deletes
* fix(managed_files): file each repeat-upload delete under the upload that made its copy
POST /v1/files sets no request call id, so the cleanup delete passed none
and its usage ledger row had no parent. Each copy now keeps the call id of
the router upload that made it, and its delete runs under that id.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Every MCP route (/mcp, /<server>/mcp, /toolset/<name>/mcp, /mcp/proxy) ran LiteLLM auth before looking at the HTTP method, so a bare OPTIONS probe got the RFC 9728 401 OAuth challenge and clients that probe with OPTIONS looped on re-auth. The Streamable HTTP handler now returns 204 with Allow: GET, POST, DELETE, OPTIONS before auth runs. GET, POST and DELETE are unchanged.
Resolves LIT-10231
* fix(proxy): record the route's call type on failed speech spend rows
A failed /v1/audio/speech request never puts a logging object on the request
data, so the failure lift returned nothing and the spend row's call_type came
out empty (or whatever the client sent in the body). Without a logging object,
lift the call type the route already resolved, so a failure row records the
same call type a success row on that route records
* fix(proxy): keep the call type of a failure logged under its own route
An MCP tool listing that fails inside a Responses, chat, or messages call
is logged under /mcp/list_tools, so its spend row keeps list_mcp_tools
instead of taking the outer request's call type
* fix(proxy): take a failed row's call type from its endpoint, not a route path guess
* fix(proxy): log the endpoint call type on failed image generation and rerank rows
* fix(proxy): estimate failed-call ledger usage from the route's call type again
The failure usage snapshot now gets the same call type the spend row uses for
its estimate, so a failed Responses call counts input from the newest
compaction item again, as on main.
The endpoint call type tests now go through real routers whose provider
answers 400, instead of asserting on a mocked failure hook.
* test(proxy): type the failed spend row tests and fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve DB settings during config reload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): mark reload settings variable as final
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Apply database general settings only for managed keys, warn about ignored copies, and allow admins to delete stale stored keys
Move intended database-only settings into config.yaml before upgrading, including authentication and access restrictions
* fix(proxy): charge budget windows when the cached key or team expired before the request ended
The post-call spend update reads a key's or team's budget_limits from
user_api_key_cache and returned early on a miss, skipping both the window
counter increment and the LiteLLM_BudgetWindowSpend row. The cached object
expires after user_api_key_cache_ttl (60s by default), so any request that ran
longer than that without another request refreshing the entry was counted in
the key's lifetime spend but never in its budget windows. With budget
reservation disabled nothing else charges the windows, so keys kept running
past their window caps
On a miss the update now reads budget_limits straight from the key or team row.
It does not re-cache the object, since a write from the spend path could race a
/key/delete and put a just-deleted key back into the auth cache
* test(proxy): assert the budget window fallback never re-caches the key or team
* fix(proxy): read a missing key or team row once per cache TTL in the budget window fallback
The master key and UI session tokens are logged under names no key row
carries, so every one of their requests missed the auth cache and read the
database. A missing row is now remembered for the cache TTL in this process
only. A failed read is not remembered, and a row that exists is read fresh on
every miss so a key update is never served stale.
* fix(proxy): skip the budget window row read for a key or team with no windows
The fallback remembered only a missing row, so a key or team that exists but
holds no budget_limits was read from the database on every request that
outlived the key cache. It now remembers both cases under one cache key built
in user_api_key_cache.py, for the same TTL as the auth cache
* fix(proxy): keep no-window markers in their own cache and clear them when an update adds windows
Markers for keys and teams with no budget windows lived in the shared 200-entry auth cache, so a burst of late costs could evict cached teams and users. They now live in a dedicated in-memory cache on UserApiKeyCache, reached by local and broadcast evictions. /key/update and /team/update now evict the marker on every worker, so a late cost after an update charges the windows the update added.
* test(proxy): give the spend counter tests the proxy's real key cache and type the window spend helper
* fix(proxy): keep a row read that an update outran from restoring the no-window marker
* fix(proxy): clear finished no-window reads and evict a marker-shaped user id from both caches
* fix(proxy): apply other spend counters before a window row read and let no-window notes follow the key cache settings
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
