* fix(azure): cache the remaining per-request Entra credentials and keep token refresh off the event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): await the now async azure health-check auth header helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure): await async azure_ad_token_provider results in the threaded wrapper
* fix(azure): keep transport params when validating async Azure client params on pydantic 2.11
* test(azure): assert the token and validation thread hops by thread identity instead of sleeps
* fix(azure): give the OIDC token exchange its own short-lived HTTP client
The exchange now runs from worker threads, so it no longer shares litellm.module_level_client's connection pool with other threads
* test(azure): check the OIDC exchange closes its client through an in-process transport
* refactor(responses): merge caller extra_headers into a new dict instead of updating validated headers in place
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(search): add DeepSeek as a /v1/search provider
Adds DeepSeekSearchConfig for POST https://api.deepseek.com/v1/tools/search, registers it under search_provider=deepseek, prices deepseek/search at 0.00075 per query and marks search support in provider_endpoints_support.json
* fix(search): reject DeepSeek queries over the 128 character limit before the call
* test(search): escape regex metacharacters in DeepSeek query limit match
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
test_client_disconnect_before_any_content_sends_no_response_scan released the gated upstream right after
the client closed, so the upstream's next chunk raced the disconnect into the proxy. When the chunk won,
the proxy had " tail" on record for the client and the cut-short scan recorded a post_call success on
the spend row, which the test asserts is absent (CircleCI integration-extensions job 2263560 on
68db022d57: `AssertionError: (('success',),)`).
The test now reads the spend row first, which is written once the proxy has settled the disconnect, and
releases the upstream after it. The proxy's behavior is unchanged; only the test's ordering is.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): record cooled deployments on the all-cooled failure row
A RouterRateLimitError now carries the cooled deployments of the route the
request resolves to for the caller's team (model group, alias, wildcard or
team route), each with its masked api_base. get_error_information copies them
into error_information.error_cooldown_deployments so the SpendLogs failure row
and every logging callback can tell which deployments were cooling
* fix(router): name cooled deployments from the router's own resolved route
Use the deployments the router resolved for the request, before health and
cooldown filtering, instead of re-deriving the route. Alias chains, the
provider-model fallback, wildcards and team filtering now match routing by
construction. A routing-strategy refusal records null.
* fix(router): name only the cooled deployments tag and region routing allow
The refusal's cooled-deployment list now leaves out deployments the
request's tags or the end user's allowed region keep it from, so a
tenant's refusal row no longer shows another tenant's api_base
* fix(router): keep failure logging working for refusal subclasses that skip init
A RouterRateLimitError subclass that never calls its parent init now logs a null cooled-deployment list instead of breaking the failure payload
* test(router): call the refusal routing helpers directly
The router coverage check needs each router function called by name from a unit test
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
anyio below 4.14.0 can leave a cancelled waiter queued on an asyncio Lock or
Semaphore, so the next task to acquire waits forever on a lock nobody holds
(agronholm/anyio#1145, fixed in 4.14.0). httpcore takes these locks for every
httpx request, including the prisma query-engine client
litellm imports anyio directly but never declared it, so pip and uv kept an
already installed anyio 4.13.0 when installing litellm and accepted an
explicit anyio==4.13.0 next to it. uv.lock already resolves 4.14.2, so Docker
and CI installs do not change. Both the litellm and litellm-core manifests get
the floor, and the lock only records the new direct dependency
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(openai): drop unsupported reasoning effort for gpt-6.1-sol
gpt-6.1-sol accepts low, medium, high, xhigh, and max. none and minimal were still forwarded, because chat completions ignored an explicit supports_none_reasoning_effort=false and the Responses mapper never applied that contract.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openai): mark the stripped reasoning object as outgoing JSON
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Azure responses reading the OpenAI effort map
gpt-6-astra on Azure accepts reasoning.effort none. The responses gate was checking the OpenAI entry, which rejects it.
Signed-off-by: Paul El Khoury <82139421+pOpO1-9@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(openai): gate every reasoning_effort level from the cost map on chat and Responses
One gate reads the model's cost-map row for whatever effort level a request
asks for, on Chat Completions and the Responses API, for OpenAI and Azure.
xhigh stays opt-in on chat (Azure also keeps none opt-in); every other level
is refused only when its supports_<level>_reasoning_effort flag is false. A
refused level is dropped under drop_params or answered with a local 400, and
Responses keeps the rest of the reasoning object when it drops effort. The
guards that follow judge temperature by the level the provider will see.
Tests move into the mapped test files and derive the expected outcome for
each level from the map row, so a new GPT-6 model needs only its row.
* test(openai): pin gpt-6.1-sol's refused effort levels and prove the row flags alone drive the gate
* fix(cost-map): azure gpt-6-astra refuses none and honors max on Responses, as live calls show
* test(reasoning_effort): pin the astra max and none flips through the adapter and Foundry paths
* fix(reasoning_effort): keep max out of the route gates and correct the gpt-5.3-codex rows
The same model honors max on Responses and refuses it on Chat Completions, so one row flag
cannot gate it on either route. The gates now skip max, the Azure gpt-6-astra rows keep max
off so the /v1/messages adapter degrades to xhigh and /model_group/info stops short of it as
on main, and chat still forwards max for the Responses bridge.
The gpt-5.3-codex rows said none and xhigh off and minimal on; the live model answers the
other way round (2026-10-01), and an opt-out gate would have dropped the levels Codex CLI sends.
* fix(reasoning_effort): bridge function tools to Responses when the chat gate drops none
* docs(reasoning_effort): correct the max advertisement note
* fix(azure_ai): read the Foundry row on the Responses effort gate and turn none off for Azure gpt-6.1-sol
* fix(azure): gate a prefixed Foundry deployment's Responses effort off its row
litellm.responses(model="azure_ai/<deployment>", api_base=<Foundry host>) is
re-routed to the azure provider with the azure_ai/ prefix intact, so the Azure
Responses config saw a prefixed name, skipped it as another provider's
pass-through, and forwarded none and minimal to Azure even under drop_params.
The Azure gpt-5 resolver also read no row at all for a prefixed name.
The Azure Responses gate now detects the family the way the Azure chat gate
does, and an azure_ai/ name on the Azure OpenAI configs reads its azure_ai/ row
when the map has one, else the azure/ twin, the rule the Foundry chat config
already applies to the same deployment.
* fix(cost-map): turn none off on azure/eu/gpt-6-astra like its sibling rows
* docs(reasoning_effort): say how each gate reads none and shorten the module docstring
* fix(azure_ai): call the public effort-level check from the Foundry Responses gate
* test(reasoning_effort): fold the Foundry Responses tests into the mapped file and type the row helpers
---------
Signed-off-by: Paul El Khoury <82139421+pOpO1-9@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* feat(proxy): opt-in self-serve budget edits for key owners on /key/update
Adds general_settings.self_serve_budget_policy (disabled by default) so a
non-admin can change max_budget or budget_limits on a key they created and
still own. lower_only allows adding or lowering limits. ceiling also allows
raising or removing them when the owner could create that key with the new
budget via /key/generate (same member permissions and delegation ceiling).
soft_budget and spend stay admin-only
* fix(proxy): sync dashboard API types and cover the UI session cap on update
* test(proxy): cover team role restriction for owner budget raises
Drop two unreachable branches from the non-raising key generation check.
* fix(proxy): close self-serve budget bypasses for linked budgets, null caps and retained limits
Keys with a linked budget_id need an admin to change max_budget, owners cannot
leave a key uncapped when an upperbound max_budget is configured, and the
ceiling check now runs on the budget the key would carry after the update.
* fix(proxy): keep open key budget windows and the admin path on self-serve budget edits
A resubmitted budget_limits window now keeps its stored reset_at while that period is still
running, so adding or editing one window no longer restarts the others. Team and org admins
answer False on a ceiling loosen so they fall through to _check_key_admin_access as before.
The self-serve tests feed the team through the cache and prisma mocks instead of patching
get_team_object, and every helper is typed
* perf(proxy): group key budget windows by duration in one pass
The per-duration minimum used to rescan the whole window list once per distinct
duration, so a request with thousands of windows pinned the event loop before any
permission check ran. Sort once and take each group's minimum instead
* refactor(proxy): take budget windows as a Sequence in the delegation ceiling checks
* test(proxy): assert allowed self-serve budget edits return no denial
Also drop the unreachable assert_never arm from the exhaustive budget change match.
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Covers token reuse across username/password and oidc/azure deployments, a held
mint not blocking /health/liveliness on chat, Responses, realtime, the first
oidc/azure chat and Bing grounding search, and one Bing provider across searches
async_search called validate_environment on the event loop, so a Grounding with
Bing Search request that mints an Entra token blocked the loop for the token
endpoint round trip. It now runs in a worker thread like the Responses handlers
An oidc/ azure_ad_token was exchanged for an Entra access token while building the
SDK client, so the first async chat request ran the federated token mint and the
token endpoint POST on the event loop. The exchange now runs as the client's token
provider, which async clients already call in a worker thread, and the exchange
cache keeps later calls cheap while letting a cached client pick up a fresh token
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* refactor(count_tokens): hold the tool counting wiring for a follow-up commit
* refactor(count_tokens): hold the chat template tool sections for a follow-up commit
* fix(count_tokens): count tools the way DeepSeek V4 and Qwen3.5 chat templates render them
* fix(count_tokens): skip tool_choice and non-function tools for chat template families, leave out Fireworks V4.1 spelling
* fix(count_tokens): count Anthropic tools in the shape the messages bridge sends them
A missing description is no longer rendered as null and strict is counted when set, matching what DeepSeek V4 and Qwen3.5 deployments receive. Adds tests that pin each template's schema rendering and its system turn.
* test(count_tokens): check the prompt caching threshold agrees with token_counter for DeepSeek V4 and Qwen3.5 tools
* fix(count_tokens): skip Anthropic web search tools and count unmapped Anthropic tool keys for chat template families
* test(token_counter): cover chat-template tool counts across count routes, budgets, ITPM and context edits
Integration cells check that DeepSeek V4 and Qwen3.5 tools are counted the way their chat templates render them on /v1/messages/count_tokens, /utils/token_counter, /v1/responses/input_tokens, strict budget reservation, project ITPM on chat, messages and responses, and the clear_tool_uses trigger, with grok controls keeping OpenAI framing
Show discovered Kubernetes pods on the model page for admins
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Requests with an explicit session id stick to one discovered pod via rendezvous hashing
Requests without a session id keep round robin
Pod routing records are included in logs, spend logs, and traces
Resolves LIT-9168
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(baseten): list current Model APIs instead of retired 2023 deployment IDs
`baseten/*` wildcard expansion and `/v1/models` read a hardcoded set of
three 2023 deployment IDs (Falcon 7B / WizardLM / MosaicML) that no longer
exist, while the model cost map already carries current Baseten entries.
- Populate `baseten_models` from the model cost map, like fireworks_ai.
- Add `BasetenConfig.get_models()` against the Model API `/v1/models`
endpoint and register it in `ProviderConfigManager.get_provider_model_info`,
so `check_provider_endpoint: true` returns the live catalog.
- Remove the retired deployment IDs from `constants.py`.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(baseten): mark retired Model APIs deprecated and skip them in listings
15 Baseten entries in the model cost map are no longer servable: the Model
API returns 410 "model version ... has been deprecated" (DeepSeek-V3.1
returns 400 deactivated), and none appear in `inference.baseten.co/v1/models`.
- Set `deprecation_date` on those entries (kept for spend tracking).
- Skip entries past their `deprecation_date` when building `baseten_models`,
so `baseten/*` expansion only lists models that still serve.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(baseten): assert the custom api_base model listing on the returned models
The custom api_base test only inspected the patched client's call args, which trips the
TQ002 mock-echo gate. Serve the custom URL with respx and assert on the models that come
back instead, so a wrong host or a broken parse both fail.
* fix(baseten): drop newly retired models on cost map reload and fake discovery at the HTTP boundary
A cost map refresh only ever added to the Baseten set, so a model that gained a past
deprecation_date after boot stayed listed under baseten/* until restart. Discard it on reload.
Tests now serve the catalog with respx instead of patching our own HTTP client, the
isinstance registration test is gone since the get_valid_models test already proves
discovery end to end, and new test locals carry Final.
---------
Co-authored-by: AlexKer <AlexKer@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(responses): write one spend-log row per native WebSocket turn
Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns.
* fix(responses): bill the usage a failed native WebSocket turn reports
* fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect
The v3 limiter stashed whichever litellm_call_id the request body carried as
the slot owner, so a client-supplied id on a pass-through body left the
proxy's own success and failure events unable to release the slot. The owner
is now the proxy's logging call id. A WebSocket connection whose turns log
under their own ids never settles the TPM, ITPM, or OTPM estimate the
connection reserved, so the disconnect release now refunds any reservation
no callback settled, once.
* fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect
A failed turn's reported usage now counts toward the key's TPM and the
project ITPM/OTPM scopes the same way a success does, and closing the
socket hands back only the estimate still unused in the window that
holds it, so a window that already rolled is left alone.
An id-less error event is booked against the newest pending turn, the
one the backend rejected at parse time, and a turn that nests its input
under response logs its user message. A connection's one budget
reservation goes to the first turn that settles it, every later turn
charging its cost directly.
* test(proxy): expect the settling call id on a pass-through budget reservation
* fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM
A native Responses WebSocket turn reaches the rate limiter's success
callback with its raw event list as the response and no
combined_usage_object, so the turn's tokens never reached the key's TPM
window or the project's input and output token limits. Read the usage
from the turn's events when the call type is a Responses WebSocket turn.
* fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas
The limiter's failure hook looked up the parallel slot by the request
body's litellm_call_id, while the slot belongs to the proxy's logging
call id, so a pass-through request a pre-call hook rejected held its
slot for an hour when its body carried an id. Both now resolve the id
the same way.
Each response.create frame already charges its estimate to project
ITPM/OTPM, so the turn's own usage no longer adds a second count.
The cost callback's failure hook read the body's litellm_call_id
through a strict model and raised on a non-string value, which dropped
the failure row. It now reads a string id or none.
* fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right
Free a rejected request's parallel slot through the request it runs in, and skip
native Responses WebSocket connections, whose disconnect frees the slot. Count
WebSocket turns toward the connection's tag TPM. In collector mode, claim the
shared budget reservation before the spend event is built, so only the first
turn settles it. Store each turn's own frame as its request body, and send an
id-less error to the oldest waiting turn unless an earlier turn is streaming.
* fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag
* fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls
* fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes
* fix(router): count native Responses WebSocket turns toward deployment limits and budgets
* fix(router): re-check deployment budget and RPM on each Responses WebSocket turn
* test(router): list the WebSocket turn admission method as covered through its partial
* fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn
* fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure
* fix(proxy): log the routed model group on a WebSocket turn's failure row
* fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn
* test(integration): cover per-turn admission on Responses WebSockets
Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the
project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP
controls for TPM counting and the failure row model_group
* fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails
A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success
callback such as the response cache no longer runs each CustomLogger's sync success hook on top of
the async one. That second run counted every turn's tokens twice toward deployment TPM.
A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket
closes with 1011, and the error becomes the socket's failure, on both the native and the managed
relay, instead of the socket dropping with no frame and no failure row.
A refused turn's failure row now carries the request's tags from the router's metadata, the same
way it already carries the model group.
* test(integration): cover a crashed WebSocket turn check and failure-row tags
A turn whose admission check raises gets a 500 server_error frame, the
socket closes with 1011 and its one failure row carries the error, on
both relays. The refused turn's failure row keeps the caller's tags, and
HTTP failure rows on chat, responses and messages keep them too.
* fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows
A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket
now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which
a client that sends its next response.create early would lose.
A refused turn's failure row now takes the router's tags only when every tag is a string, so the
fallback never copies a caller's unvalidated tags onto the row.
* test(integration): cover a turn still streaming when the next WebSocket turn's check crashes
A native turn still streaming when the next turn's admission check raises now has its own cell. It
must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for
one more turn, so a relay that stopped checking turns after a crash goes red.
* fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets
A turn after the first now runs the project quota check, then the proxy's
pre_call guardrails, then the deployment's turn admission, on both the
native and the managed relay, the order HTTP calls use. A turn a guardrail
blocks gets the guardrail's error even past the budget and no longer counts
toward the deployment's rpm or runs its pre-call checks.
* fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it
A turn the per-turn guardrail passes and the deployment then refuses now writes
its own failed spend row with that turn's guardrail information, on both relays,
the way a guardrail-blocked frame and an HTTP refusal already do
* fix(responses): bill a refused WebSocket turn on one row only
A refused turn whose guardrails recorded a cost is already billed on its own row,
so it no longer also becomes the connection's close-time failure row
* fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row
The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them
* fix(responses): bill a WebSocket's refused turns when the session limit closes it
* fix(responses): bill refused WebSocket turns when the session limit cancels the relay
* fix(responses): free the WebSocket slot at the session limit without waiting on the failure row
* fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot
* fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown
* fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after
* fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it
* fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close
* fix(proxy): always commit spend again after the shutdown wait for pending logging tasks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
get_azure_ad_token() built a new DefaultAzureCredential for every request when enable_azure_ad_token_refresh is on, so each Azure Responses call fetched a fresh Entra token synchronously on the event loop. Use the per-scope cached provider the SDK-client path already uses, so the credential's own token cache does its job. Fixes#46419
Azure retired the preview realtime surface, so /openai/realtime/client_secrets
and /openai/realtime/calls with an api-version return 404. Both now target
/openai/v1/realtime and the SDP exchange sends the ephemeral key as a Bearer
token, matching the Azure GA WebRTC contract.
Fixes#24659
Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com>
