mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-14 09:03:16 +00:00
98 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kerry Lu
|
172e16acde
|
feat(proxy): router_settings.fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team model_preferences (#51439)
* feat(proxy): fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team_metadata.model_preferences Team router_settings.fallbacks now fire on proxy-side rate limits: the pre-call fallback walk reads router settings Key > Team > Global instead of the key's column alone, and skips a fallback the caller may not call. A key or team with router_settings.fallback_on_saturation: true walks that same fallback list as soon as the requested model reaches saturation_threshold in dynamic_rate_limiter_v3, and goes back to the requested model when no fallback admits the request. Responses carry x-litellm-fallback-reason: rate_limit:model_saturation_fallback. Removes the team_metadata.model_preferences steering and the x-litellm-model-steered-from header from #51350, which did the same job through a second knob. * fix(proxy): strict bool for fallback_on_saturation, drop a stale import and settle the pre-call tests fallback_on_saturation only turns on for a real boolean true, so a string like "yes" in router_settings no longer enables it. The saturation unit test reads saturation through the limiter's cache, so it asserts the threshold was reached instead of an exact count. The response polling test now hands the pre-call path an awaitable router_settings lookup, as the pre-call fallback walk reads Key > Team settings before the first call. * fix(proxy): pre-call fallbacks pass the budget and access-group checks a direct request does A fallback the pre-call walk picks now has to clear every per-model budget layer and the budgets of the access groups that grant it, and its spend is charged to those groups instead of the requested model's. At saturation a fallback that fails for any reason counts as a miss and the request goes back to the requested model. The client body can no longer opt itself into the saturation walk through router_settings_override, only the saved key or team router_settings can. The integration test for a forbidden fallback waits for the preferred model's own 429 so it cannot pass before saturation. * refactor(proxy): one helper decides which refused fallbacks end the pre-call walk _pre_call_with_fallbacks had grown past the strict C901 ceiling, so the two except clauses fold into one and _fallback_refusal_ends_the_walk carries the rule: at saturation every refusal is a miss, otherwise only a non-rate-limit failure or a hard per-model 429 reaches the caller. * fix(proxy): keep the caller's request object through the pre-call fallback walk Copying the request to drop a client fallback_on_saturation opt-in handed the hooks a different dict than the one the caller holds, so header metadata written into one never reached the other (test_add_litellm_metadata_from_request_headers). drop_client_saturation_opt_in now edits the request in place. * fix(proxy): pre-call fallbacks keep the pricing and budget-reservation rules a direct request passes A fallback is skipped when unpriced models are blocked and it has no pricing, when a zero-cost request would land on a priced model, and when the budget auth reserved for the requested model does not fit it (higher per-token prices, or reserved against an access group that does not grant it). * refactor(proxy): read fallback prices through the router registry entry point router_surface_check only lets the router surface shrink, so the per-token price lookup goes through registry().group() instead of the deprecated get_model_group_info. * fix(proxy): a fallback's group budget check skips the groups the request already holds The auth-time reservation sits in the same counter the plain budget read uses, so a fallback in a group the request already reserved against was refused by its own hold once the group was within one estimate of max_budget. Groups without a hold keep the plain read. * test(dynamic_rate_limiter_v3): the throttle metric test from main uses fallback_on_saturation instead of the removed steering A saturation handoff carries no rate_limited_by reason, so it is never counted as a throttle. * fix(proxy): the saturation handoff only fires under the fallback walk _pre_call_with_fallbacks arms the request before the hooks run, so routes that call the hooks directly (batches, fine-tuning, polling, images) keep admitting or refusing an opted-in key on a saturated model as before instead of surfacing the handoff 429. * style(proxy): keep the cast-ok reason within the line length * fix(proxy): never invent a metadata bucket when arming the saturation fallback walk, take main's limiter metric tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): arm the saturation fallback walk in the request context, refuse fallbacks for agent-owned keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep the saturation walk state out of the body, check the end user's allowlist on fallbacks, release a refused fallback's rate limit hold Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only load the end user for fallback checks when a database is connected Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): grant pre-call fallbacks on the model an alias serves and release the rate limited model's hold before the walk Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): check a pre-call fallback's per-model budgets on its listed name and its served group, as auth does Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert the listed fallback name's budget blocks an aliased fallback, not the order of budget reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): release the dynamic limiter's reservation when the fallback walk leaves a model, and replay an exhausted walk with the requested model The dynamic rate limiter admits through its own inner v3 limiter, so the walk's hold release never reached its team-share reservation. It now exposes async_post_call_failure_hook, which refunds that reservation when the proxy refuses a request after admission, and _release_pre_call_hold calls it next to the registered limiter. An exhausted saturation walk replayed with the already aliased model name, so chained model_alias_map entries moved the request on to a further alias. It now replays with the model argument the first attempt saw. The hold-release test drives the real v3 limiter with one slot per key instead of recording hook calls. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type the dynamic limiter's failure hook request dict for the type-discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
de3f0690e7
|
fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates (#51415)
* fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates The per-team share counters only count requests admitted while the model is saturated, so what a team used while the model was idle is not charged against its share once contention starts. A team below its share is still admitted ahead of a borrower, and the model's own limits bound the total. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): idle fallback targets and WebSocket turns follow the same share accounting A router fallback target attempted while idle no longer holds the team's share, and a Responses WebSocket connection refreshes its share descriptor on every turn, so idle turns are forgiven and saturated turns count. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): an idle fallback target still records its hold Settlement finds the hold, gives the admitting model's reservation back and charges nothing for the idle target, instead of charging the target's tokens to the admitting model's share. The integration fixture builds its configuration without mutating the loaded one. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
7130a5b4fd
|
feat(dynamic_rate_limiter_v3): prometheus share, borrowed and throttle metrics per team and model (#51417)
* feat(dynamic_rate_limiter_v3): Prometheus share, borrowed and throttle metrics per team and model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): name the BLE001 suppressions on the metrics guards Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): count only final refusals, publish the served model's share, join the priority label Review round 1: a steered alternate's refusal no longer counts as a throttle when the preferred model then admits the request, a router fallback's settlement publishes the fallback target's own share under its model group, the gauges carry the same priority label the 429 carries, the gauges aggregate with livemostrecent so a new window replaces an old worker's value, the share read is skipped when no Prometheus logger is configured, the unit tests restore the process registry, and the integration test polls for the settled totals. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(prometheus): type the gauge setter and the label factory so the type gate holds _set_labeled_gauge takes a _MetricLike and sets the series only when it is a real Gauge, prometheus_label_factory declares the dict[str, str | None] it returns, the priority resolver goes back to its original shape, and the label test tolerates the now visible None values. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(prometheus): keep the label test diff to the two None-tolerant assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
2c7a36abe9
|
fix(rate-limit): hold router fallbacks, nested calls, WebSocket turns and oversized prompts to a borrowing team's model share (#51367) | ||
|
Kerry Lu
|
c0114e55aa
|
feat(dynamic_rate_limiter_v3): priority_class_share_splits_equally_among_active_teams splits a priority class share among the teams active on a model (#51366)
* feat(dynamic_rate_limiter_v3): split a priority class share equally among the teams active on a model Adds priority_reservation_settings.priority_class_share_splits_equally_among_active_teams (default false). When on, the class weights are renormalized over the classes with a team active on the model in the current window, each class share is divided equally among its active teams, and once the model saturates each team is held to its own share with a 429 whose rate_limited_by header reads priority_class_share_saturated. Active teams are one Redis sorted set per model with the window as TTL. * fix(proxy): keep basedpyright counts flat for the priority class share setting PriorityReservationSettings is built with model_validate instead of **value, so a new field no longer adds an unknown-argument diagnostic, and the limiter unit test helpers type the handler by its implementation class. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): keep team shares usable when Redis, mixed-priority keys, WebSocket turns or tiny shares get in the way A registry failure falls back to the class pool instead of skipping the model limits, a team's counter is keyed by model and team so keys of different priorities spend one share, Responses WebSocket turns are held to the class share like the team model limit, a share is floored at one request and one token, and the registry declares the rate limits key family. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): catch only Redis and validation errors when the active team registry fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): keys without a team keep their class present, team ids may contain the registry separator, breaker-open counts as a registry failure Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): per-class pool markers that take no team slice, collision-free model and team counter keys, local registry pruned by last-seen time Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): settle tokens only on the stash the call owns, keep a configured class named default apart from teams with no priority Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): tag team registry members and count every unenforced request against the team share A team id can no longer spell a pool marker, since real teams are stored under a team prefix. Below saturation the team share counter is bumped through the atomic increment at the model-wide limit, so the local over-limit short-circuit never skips a request and the counter is complete when the share grows. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): list the caller under its own class and resync the socket stash share on every frame Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dynamic_rate_limiter_v3): keep the connection share across a registry outage and settle fallback tokens on the served model's team counter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
52d1c4cd80
|
feat(dynamic_rate_limiter_v3): steer flexible teams to a permitted alternate when their preferred model is saturated (#51350)
* feat(router): steer flexible teams to a permitted alternate model when their preferred model is saturated
Teams opt in with metadata.model_preferences: [{"model": "luna", "alternates": ["opus"]}].
The dynamic rate limiter v3 pre-call hook reads the saturation cache for the preferred model
and, once it is at or above saturation_threshold, rewrites the request to the first alternate
the team may call that is below the threshold. The response carries
x-litellm-model-steered-from with the preferred model. Rate limits and spend are charged to
the model that served. Teams without model_preferences are untouched.
* fix(dynamic_rate_limiter_v3): type the hook request dicts so the steering rewrite passes the lint gates
* fix(dynamic_rate_limiter_v3): authorize alternates against the key and its model budget, send the steering header on streams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter_v3): enforce member and project allowlists on alternates, skip the steering header for non Latin-1 names
* fix(dynamic_rate_limiter_v3): check every budget layer for an alternate, fall back to the preferred model when the alternate refuses, merge the served model's guardrails
* fix(dynamic_rate_limiter_v3): never steer agent keys, type the steering unit tests
* fix(dynamic_rate_limiter_v3): hold alternates to customer grants, access group budgets, guardrails and the zero-cost waiver
An alternate now passes every check a budget fallback passes. Customer model grants are enforced, the access groups that grant the alternate are budget checked and stamped on the token and request metadata so spend charges them, a zero-cost preferred model that waived the aggregate budget checks is never steered to a priced alternate, and an alternate whose deployments add model-level guardrails the request did not run is skipped.
* fix(dynamic_rate_limiter_v3): stamp the alternate's groups on every bucket and skip alternates the request cannot safely take
The served model's access groups now land on every metadata bucket and the auth copy inside it even when the preferred model had none, so group spend is charged. An alternate is skipped when a budget reservation already holds a different group, when it does not serve the request's endpoint, when unpriced models are blocked and it has no price, or when its guardrails differ from the request's by name or settings. A failing alternate check or alternate admission leaves the request on the preferred model instead of failing it.
* fix(dynamic_rate_limiter_v3): keep a reserved budget off pricier alternates
While the request holds a budget reservation sized for the preferred model, an alternate that costs more per token is skipped, so fail-closed budget enforcement never admits a call the reservation does not cover.
* fix(dynamic_rate_limiter_v3): an unpriced alternate never takes a reserved budget
While a budget reservation is held, an alternate with no input or output price is skipped rather than treated as free.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Kerry Lu
|
ece1484bc7
|
feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity (#51359)
* feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity With team_model_limit_borrows_idle_capacity_from_other_teams on, a team could use everything up to the model's own tpm and rpm while the model was idle. Setting team_model_limit_max_borrow_multiple in the team's metadata caps the team at its model_tpm_limit and model_rpm_limit times that multiple, with a 429 whose rate_limited_by header reads model_per_team_max_borrow_multiple. Unset keeps the previous behavior. Values below 1 or non-numeric are logged and ignored. * fix(rate-limit): reject infinite or oversized team_model_limit_max_borrow_multiple values instead of failing the rate check * fix(team): make team_model_limit_max_borrow_multiple proxy-controlled team metadata so team admins cannot lift their own cap --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
19ffe3a5f2
|
feat(rate-limit): team_model_limit_borrows_idle_capacity_from_other_teams lets a team's model TPM and RPM limits borrow idle capacity in the dynamic rate limiter (#51328)
* feat(rate-limit): opt-in team_model_limit_is_floor turns a team's model_tpm_limit into a guaranteed share that borrows idle capacity With priority_reservation_settings.team_model_limit_is_floor: true, a team's inherited model_tpm_limit on a model the dynamic rate limiter covers stops being a hard cap in the regular limiter. The dynamic limiter tracks it as a team_model_floor descriptor, lets the team past it while the model is below saturation_threshold, and holds the team to it once the model saturates, with rate_limited_by: team_model_floor on the 429. Default false keeps today's behavior. Key-owned model_tpm_limit, model_rpm_limit, the model-wide tpm and customer limits stay hard. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(rate-limit): name the setting team_model_limit_mode with hard and soft values soft replaces the team_model_limit_is_floor flag. The 429 header reads rate_limited_by: team_model_soft_limit and the message says the team soft limit was reached. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(rate-limit): team_model_limit_borrows_idle_capacity_from_other_teams covers team model TPM and RPM Replaces team_model_limit_mode. A team's inherited model_tpm_limit and model_rpm_limit on a model the dynamic rate limiter covers are enforced by that limiter once the model saturates, so the team borrows idle capacity below the threshold. The 429 names the limit and reads rate_limited_by: model_per_team_saturated. Folds in the bot findings: key ownership is checked per requested model, team limits resolve through the model alias like the regular limiter, the team counter is charged only for requests the pre-call hook tracked, every response type that carries usage is settled, and tracking below saturation no longer adds a token per request. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): keep the type gate flat in the dynamic limiter's team limit paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): settle responses usage and null auth metadata, resolve aliases for the borrowing team limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): key the borrowing team counter by resolved model group and read usage through get_usage_object Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): hold Responses WebSocket turns to the borrowing team limit once the model saturates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
fdafffb842
|
feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason (#51138)
* feat(proxy): count a rate-limit fallback in x-litellm-attempted-fallbacks and name it in x-litellm-fallback-reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count every fallback candidate a local rate limit rejected in x-litellm-attempted-fallbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): count fallback attempts without a mutable list and keep self.data on the served request Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
917fd11966
|
fix(rate-limit): settle a fallback's tokens on the model group that served it (#51082)
* fix(rate-limit): settle a fallback's tokens on the model group that served it The v3 limiter pinned post-call TPM settlement to the model reserved at admission, so when the router fell back to another model group the requested model's counter took the usage and the serving model's counter stayed empty. When the served model group differs from the admitted one, charge the served group and refund the abandoned group's per-model reservation. Same-model retries and alias reloads keep the admission target. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): refund an abandoned fallback reservation only on counters the served model does not settle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rate-limit): wait for the primary refund and the fallback charge before reading counters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): only treat a limiter-admitted fallback target as the served model at settlement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rate-limit): drop unrelated formatting from the v3 limiter unit test file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rate-limit): remove helper copies duplicated while appending the fallback settlement tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mika-lt
|
373fe799e7
|
fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order (#50971)
* fix(router): send managed batch uploads to batch deployments when a chat deployment has a lower order A purpose=batch upload on a mixed Azure chat and batch group was order-filtered before batch routing narrowed it, so the file landed only on the order-1 chat deployment and the batch create that followed got Azure's 400 invalid_deployment_type. The upload now runs under the same batch routing as the batch create. * fix(router): keep a failed batch upload's cooldown and order fallback on the batch deployments * fix(router): keep a batch upload off chat deployments while its batch deployment cools down * fix(router): keep a refused batch upload's retry off the chat deployment * fix(router): keep a refused batch upload's error and skip chat while the batch deployment is paused * test(router): pin the order fallback for a batch create on a group with no batch deployment * test(integration): expect an Azure batch upload on the batch deployment alone --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
e2625d0509
|
fix(router): re-check and reserve per-model rate limits on fallback targets (#50994)
* fix(router): re-check and reserve per-model rate limits on fallback targets A provider error fallback handed the caller the fallback model group without its per-model TPM/RPM limits ever being checked. The router gets a fallback_rate_limit_check hook next to fallback_budget_check, the proxy wires it to the v3 limiter, which reserves the target model's per-model descriptors before each fallback hop and skips targets that are over their limit. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): dedupe a repeated fallback target before the reserving rate limit check Both walks now consult the attempted set before the rate limit check, so a chain like ["B", "B"] reserves B once. The deprecated settings parity test also learns the new setting. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): a bidirectional fallback chain never re-reserves the requested model group Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): match the surfaced failure in the fallback reservation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(rate-limit): keep the admission stash untouched while building fallback descriptors Building the fallback target's descriptors could rewrite the request stash's PTU ceiling, which post-call settlement reads for the model the caller named. The fallback check now hides the stash while it builds, like the frame check. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
paveln83
|
a7696a2bc2
|
fix(proxy): send the streamed Responses fallback metadata as a typed Responses event (#50804)
* fix(proxy): send the streamed Responses fallback metadata as a typed Responses event
With expose_fallback_errors_to_caller on and include_fallback_errors set, a streamed /v1/responses
call that fell back sent its fallback metadata as a chat.completion.chunk with no type, so the OpenAI
SDK read it as an audio delta event with type None. On the Responses route the frame is now
{"type": "litellm.fallback_metadata", "litellm_fallback": {...}} and goes through the same Responses
framing as every other event, so it gets an event: line whenever its neighbours do. The chat route
keeps its chat chunk unchanged
* test(proxy): stop pinning the Responses stream terminator in the fallback metadata test
Main ends Responses streams without data: [DONE] since #47980, so the test reads the event frames whether or not a terminator follows them, and passes both before and after that change merges
* fix(proxy): keep the top-level model on the Responses fallback metadata event
* fix(proxy): keep the fallback metadata id on the Responses event and omit an empty model
* fix(proxy): leave the top-level model off the Responses fallback event when raw model names are on
The complexity router's return_raw_model_name leaves every other event unstamped, so the fallback event now matches. A new test streams a raw upstream frame before the event to cover how main names it.
* chore: push again after GitHub dropped the previous push event
* fix(proxy): send the Responses fallback event without an event line so older OpenAI SDKs read it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
yuneng-jiang
|
71c1a2aff6
|
test(integration): repair tests that merged changes left stale (#50782)
Several integration tests on main still assert behavior that later changes replaced on purpose, so they fail on every scheduled run, and one of them stops the whole sdk group at import. The batch cost case expected cached tokens to bill at the Batch input price when a price row has no Batch cache price. Since #50062 they bill at the Standard cache price, so the case now expects 0.000235 input cost and is renamed to say so. The OTel v2 attribute test pinned langfuse_otel and langtrace as carrying no request metadata baggage. Since #50373 every OTel v2 destination carries it, so all seven callbacks share one expectation and the Langfuse baggage test that was deferred under LIT-9128 runs again. The clean-EOF outage test trips its database relay on the text of the spend log insert. The ledger writes that insert without the "public" schema prefix, so the relay never tripped and the test timed out before it checked anything. The trigger now matches the insert on both write paths. The usage routing tests expected cooldown, tpm and rpm keys in one MGET. Since #50170 the cooldown keys are read in their own MGET, at most once per DEFAULT_COOLDOWN_REDIS_READ_INTERVAL_SECONDS. Both the proxy and the SDK test now expect the two MGETs, and the proxy test waits out that interval after its /model/info poll, which reads the same keys. The concurrent deployment delete test held its writer inside Router._update_deployment_indices_after_removal, which went away with the Python router core in #50178, so the file failed at import and stopped the whole sdk group. It now holds the writer where the registry reads the model list again after router-core applied the change. |
||
|
devin-ai-integration[bot]
|
169024c8fe
|
test(routing): expect /v1/responses to pin the served deployment for an hour after #50321 (#50474)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tessellark
|
b5a72c2042
|
fix(router): pin /v1/responses requests for prompt caching affinity (#50321)
* fix(router): pin /v1/responses requests for prompt caching affinity * fix(router): pin previous_response_id turns and keep native tools in the Responses pin key A Responses turn that continues a stored response now goes to the deployment that answered it, unless that deployment already failed this request. Native Responses tools are hashed into the pin key as sent instead of converted, so two requests with different native tools no longer share a pin and a native request no longer logs a false dropped-tool warning. The pin keeps its cacheable deployment ids in a sorted tuple so loggers can serialize request metadata. * fix(router): leave a Responses request with a malformed tool to the provider * fix(router): keep the Responses prompt cache pin out of logged metadata The pre-call check now keeps the pin in a router-owned in-flight store and puts only a random token in litellm_metadata, so loggers that ship metadata never see the routing key or deployment ids. The store is shared across check instances, so turning prompt_caching off and on at runtime keeps pinning, and a retry drops its previous attempt's token. The skipped-input debug log now names only the error type. * fix(router): apply the prompt caching pin after tag, region and pre-call filters, and leave it on retries The prompt caching check no longer narrows the candidates inside the callback filters. It names the deployment it prefers, and the router core applies that preference at the pinned deployment step, after tag, region, pre-call and plugin filters, only when the deployment is still a candidate. A pin can no longer send a request to a deployment its tags do not allow. A retry skips the pin when the pinned deployment was already tried in this request, read from the retry breadcrumbs in both metadata buckets and the excluded and retry-skipped ids. When the retry count outruns the breadcrumbs the check stands down, so a pinned deployment that keeps failing is not retried on every attempt. * test(router): list the prompt caching check as an in-memory cache caller The check's in-flight Responses pins live in a process-local InMemoryCache, so it opens no Redis span and needs no key family, like the other in-memory callers * fix(router): pin a lone healthy deployment, free every in-flight Responses pin, and clear a reused metadata dict's retry records * test(router): lift test cooldowns through the cooldown store's evict_local * fix(router): key Responses tools like their chat twin and drop the pin token after success A Responses function tool now hashes only the function fields it sent, so a chat request and a Responses request with the same tool share one pin. The success event also removes the pin token from the caller's metadata dict. * test(router): cover untyped tool parameters and custom tool grammars in the Responses pin key --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
8ac9da84f7
|
fix(router): drop supports_web_search false deployments for Anthropic web_search_* and Gemini googleSearch tools (#50275)
* fix(router): drop supports_web_search false deployments for Anthropic web_search_* and Gemini googleSearch tools * test(integration): build the web search filter proxy config without mutating the shared config --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Kerry Lu
|
3f13c1ca34
|
fix(router): write the prompt caching pin when turn_off_message_logging is on (#50276)
* fix(router): write the prompt caching pin when turn_off_message_logging is on The success event PromptCachingDeploymentCheck reads had its messages redacted in place by turn_off_message_logging before any callback ran, so the pin was keyed off redacted-by-litellm, never written, and every follow-up bounced across the group. Callbacks that route on the request and never log it now opt out of that redaction and get the request messages back, while every other callback and the spend log keep the redacted payload. * fix(logging): type the request-messages payload helper as dict[str, object] The bare dict return grew the LIT001 count past the merge base and left the returned fields unchecked * test(integration): type the pin rig's Redis client as Redis[bytes] The untyped generic grew the basedpyright reportUnknownMemberType count past the merge base * test(integration): quote the Redis[bytes] annotation so the rig collects at runtime redis.Redis is not generic at runtime, so the unquoted subscript raised TypeError on import * fix(logging): read the request messages once as object in the exempt payload helper Two partially unknown reads of self.messages grew the basedpyright reportUnknownMemberType count past the merge base --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Nate Armstrong
|
300db9f01d
|
test(router): expect no fallback errors header for non-bool include_fallback_errors (#50259)
The Rust core reports fallback errors only when include_fallback_errors is literally true. The routing and SDK wire tests still expected the header for truthy non-bool values (1 and a 5 KB string), which the deleted Python core produced by accident. Every non-bool value now expects no errors, and the flag still must not reach the provider. |
||
|
quietjules
|
5a392b9e5f
|
fix(bedrock)!: keep Claude's thinking when response_format json_schema is set (#49696)
* fix(anthropic): stop forcing the response_format json tool when Claude will think A forced tool call skips Claude's thinking. The json_schema emulation forced json_tool_call unless thinking.type was enabled, so adaptive thinking and the models that think by default (Opus 5, Sonnet 5, Haiku 5.5, and the always-on families) lost their thinking on every structured-output request. AnthropicModelInfo.response_format_tool_can_be_forced is now the one check the Anthropic, Bedrock Converse (including the native output fallback) and Databricks paths use. It leaves the tool unforced when thinking is requested in any form or the model thinks by default, and keeps forcing it when the request turns thinking off (thinking disabled or reasoning_effort none) on a model that accepts forced tools. The new thinking_on_by_default cost map key marks Opus 5, Sonnet 5 and Haiku 5.5 on the providers that reach these paths, and a miss-only fallback rule covers unmapped Claude 5 and later ids. * fix(bedrock): route native output and thinking token counts by whether Claude will think Bedrock's native output opt-in now uses the same will-think check as the json tool, so Claude 5 models that think by default with no thinking param go native. Converse also asks for /usage/output_tokens_details on those models, so reasoning tokens are reported for them. * fix(anthropic): tell Claude to answer through the unforced response_format tool Moves the integration tests that check the forced JSON tool from Claude 5 models to Opus 4.8, since Claude 5 models now keep their default thinking and take the unforced tool or native output * fix(anthropic): keep open-object schemas on the json tool when Claude will think Native output closes every object, so an open object came back empty. Also declare thinking_on_by_default in the Rust model catalog and expect the Converse thinking-token field in test_main * fix(anthropic): hold streamed prose until the JSON tool decides the answer In json mode with the unforced JSON tool, Claude can write its working before it calls the tool. The stream now holds that text, drops it when the JSON tool answers, and sends it when Claude calls a caller tool or skips the JSON tool, matching the non-streamed reply. * test(anthropic): build the stream test events without nested comprehensions * fix(anthropic): keep always-on thinking models unforced, open objects native when thinking, and the tool instruction Claude-only * test(anthropic): keep main's open-object integration cases and cover the unforced tool on a default-thinking model * fix(anthropic): treat thinking False as thinking off so the JSON tool stays forced Cover thinking False on the Anthropic and Converse forcing tests, and pin that a Converse json_schema stream with the unforced JSON tool never sends Claude's working text as content * fix(anthropic): drop the thinking False forcing case that completion never reaches litellm.completion turns thinking False into no thinking param before any route sees it, so the case was dead on the request path and False keeps meaning the model's default * test(bedrock): collect the Converse json_schema stream chunks in one assignment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
71fc204495
|
test(routing): expect 503 server_is_overloaded for an upstream overload (#50155)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Oliver Yu
|
733b776403
|
fix(least-busy): reserve the picked deployment so a burst of requests spreads out (#47484)
* fix(least-busy): reserve the picked deployment so a burst of requests spreads out The in-flight count only went up in log_pre_api_call, which runs after selection and the pre-call hooks. Requests selected in that window all saw the same counts and landed on the same deployment (16 simultaneous requests over 4 deployments went as lopsided as 12/4/0/0 in the proxy). When the router selects for a request it is about to send, least-busy now records a reservation for the chosen deployment in the same synchronous step that reads the reservations, and log_pre_api_call turns it into the normal in-flight count. Selections without request kwargs (token counting, probes) do not reserve. A reservation whose request never reaches the API call expires after 30 s. The shared and per-worker counts and their success/failure decrements are unchanged. * fix(least-busy): prune expired and empty selection reservations The reservation map kept a key per model group forever, so a wildcard route with caller-chosen model names grew it without bound. Drop expired and empty entries on every write. Signed-off-by: Oliver Yu <tc3oliver@gmail.com> * fix(least-busy): keep pick reservations in the shared router cache Selectors that share a router cache dedupe to one registered callback, so only that instance saw log_pre_api_call. Reservations held on the other instances were never released and skewed picks for the full TTL. Storing them in the router cache's in-memory layer lets any selector release them, and the per-key TTL replaces the full-map rebuild on every write. Tests now hold the burst with a barrier instead of a sleep and use a manual clock object. * fix(least-busy): restart the reservation TTL on every write InMemoryCache keeps the TTL a live key was first given, so a reservation key that kept being rewritten still expired 30 s after its first pick and took the newer reservations with it. Drop the key before each write so the TTL counts from the latest one, and leave it deleted once it is empty. Signed-off-by: Oliver Yu <tc3oliver@gmail.com> * fix(router): count least-busy decisions calls and release reservations on cache hits Least-busy now reads the model group from litellm_metadata before metadata, so /v1/decisions calls that carry caller metadata are counted in flight and release their reservation at pre-call. A response cache hit releases its reservation instead of decrementing another request's in-flight count, and Responses WebSocket turn admission selects without reserving. The reservation store is reached through a typed protocol. * fix(decisions): count proxy /v1/decisions calls in least-busy's in-flight totals * fix(least-busy): keep reservations out of the shared router cache and count requests that start during a Redis read Least-busy now reads a call's deployment from the routing metadata the router sets, so /v1/decisions logging params stay as on main and failed decisions calls no longer start cooldowns. * style(router): wrap the least-busy reservation store line * fix(least-busy): lock the reservation read-and-update so threaded sync calls keep each other's picks Router.completion from a thread pool and sync success callbacks can read and rewrite a deployment's reservations at the same time, so one thread's pick or release could be lost. * fix(least-busy): release a pick whose call ends before reaching the provider A call that fails or returns before the provider handoff (a 400 for an unsupported param, token counting, a proxy-gate rejection without a start log, batch create) now drops its reservation instead of decrementing a count it never added. Router.select picks and batch cancel no longer reserve, since neither makes a call that logs. * test(least-busy): give callback fixtures a fixed call-start time The routing-group tests fed least-busy a finished call without the start time litellm stamps before counting it, so the count stayed up after the fix. Fixtures now use a fixed timestamp instead of the clock. * fix(least-busy): free tier-group picks, stop lookups reserving, and count only started attempts Reservations are keyed by deployment id, so a complexity, auto, adaptive or quality router frees its tier pick when the call starts. Each started attempt is recorded under its call id, and only a recorded start lowers the count when the attempt ends, so a fallback hop rejected before the provider frees its pick. Whether a pick reserves now comes from a context variable set by the routed call sites, so get_available_deployment and the deprecated async_get_available_deployment no longer hold a deployment, and custom routing strategies and Router subclasses keep their method signatures. * fix(least-busy): free a request's unstarted picks when it ends Each pick now carries its call id, so a start frees only its own call's pick, and the router frees every pick a request never started once that request ends. A fallback hop rejected before the provider no longer leaves its pick reserved for 30 seconds. * fix(router): free a least-busy pick when a vector store create ends * fix(router): free least-busy picks of calls made without a call id * fix(router): free the least-busy pick of a sync text completion * test(router): use fixed start times in least-busy fixtures * fix(router): free the least-busy pick of a sync generic call that fails before its provider * fix(router): read in-flight counts and start attempts under one lock so a busy deployment never looks free * fix(router): keep pass-through lookups from reserving least-busy picks * test(least-busy): wait for the helper thread to start before the timed join * fix(least-busy): keep realtime sessions out of in-flight counts --------- Signed-off-by: Oliver Yu <tc3oliver@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
d5ca88900c
|
fix(anthropic): answer an upstream Anthropic 529 as 529 overloaded_error on /v1/messages (#46063)
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
tessellark
|
ff91b89b8d
|
fix(router): keep encrypted reasoning when a fallback hops between same-key deployments with no api_base (#49396)
* fix(router): keep encrypted reasoning on a hop between same-key deployments on the provider's default endpoint * test(router): type the default-endpoint boundary test helpers * fix(guardrails): restore the google_text_moderation embeddings fix an earlier main merge dropped * fix(router): match encryption boundaries on the key, account, and region the request really uses A deployment's named credential now fills only the values the deployment leaves out, through the same helper the request path uses, so the boundary names the key and base the request sends. A deployment with no api_key takes the provider's default key, the boundary carries the OpenAI organization and project a deployment sends, and a Bedrock Mantle region named in the model routes through the provider resolver before the Responses URL builder * fix(router): keep header-signed accounts apart on a shared boundary and skip boundary work no reasoning needs Headers the request logger masks as secrets (Authorization, api-key, x-api-key) now join the boundary's account scope, with extra_headers overriding headers the way the request merges them, so deployments that sign in through a header on one base never share encrypted reasoning. Target boundaries are built only when a reasoning item from another deployment needs the comparison, and the Responses input strip asks only about encrypted reasoning items, so a pinned turn or fallback hop with nothing to compare makes no secret-manager reads. * fix(router): resolve default boundaries only for openai and drop the unsent organization param Aggregators and Bedrock Mantle serve several vendors on one key, so a missing key or base there no longer yields a shared boundary. The Responses request never sends the organization param, so it no longer splits the boundary, while a Cookie header now does. * fix(router): match explicit pairs as on main and key default boundaries by model An explicit api_base plus api_key pair is the whole boundary again, so tag headers no longer split deployments that main kept together. The default OpenAI boundary now carries the model, since one OPENAI_BASE_URL can front several vendors, and a deployment that declares another provider gets no boundary before any secret lookup. * fix(router): match an explicit key and the environment key on one api_base A deployment with an explicit api_base and api_key now shares a boundary with one on the same api_base that takes the same key from OPENAI_API_KEY, when the model and account headers also match. Two explicit pairs still match on base and key alone, as on main. * fix(router): skip a deployment whose params cannot be read in boundary checks A named credential with a value of the wrong type, or an extra header whose name is not a string, made the boundary check raise and fail the whole request. That deployment now has no boundary, so the others still keep the reasoning they can decrypt. Two explicit api_base plus api_key pairs also compare without resolving the provider, so they read no secrets, as on main. * fix(router): ask a secret manager for default boundaries at most once a minute per deployment * fix(router): resolve a changed named credential at once in the cached boundary and take an injected clock --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
quietjules
|
cee72ffdd6
|
fix(proxy): count native Responses WebSocket turn tokens toward key and tag TPM limits (#49257)
* fix(proxy): count Responses WebSocket turn tokens toward key and tag TPM limits A native Responses WebSocket turn logs its raw events instead of a usage-carrying response object, so the v3 rate limiter read no usage and never charged the key or tag token window. Read the turn's usage from its events the way the cost calculator does, and charge the TPM-limited tags the socket was opened under. * fix(proxy): refuse socket turns past TPM and count managed, failed, and realtime usage A later response.create turn on an open Responses WebSocket now gets a read-only check of every TPM window it counts toward (key, team, user, org, end user, team member, model-scoped limits and the socket's tags). A turn with no tokens left gets the 429 rate_limit_exceeded frame and the socket stays open. The check never reserves or increments a counter and leaves RPM, budget, PTU and project windows to their own paths. It runs on native and managed sockets. Managed turns carry the socket's call id in their metadata, so they charge the socket's TPM-limited tags. A failed native turn charges the partial usage its failure row already carries. A socket refused at open by a rate limit gets the same refusal frame and closes 1008, logged once without a traceback. /v1/realtime sessions now count toward TPM through the realtime usage combiner, and the v1 limiter reads socket usage through the same helper * fix(proxy): count failed socket turns, keep socket usage off team PTU, check agent windows per turn A native socket turn that ends in response.failed now counts the usage that event reports toward spend and TPM, as the HTTP stream path does. Native socket and realtime usage no longer charge the team PTU counter, matching the merge base. The per-turn TPM check now also reads the agent and agent session windows the socket opened with. * fix(proxy): start missing legacy user, team, and end-user TPM buckets at zero * fix(proxy): stop charging a realtime session's whole usage into its close minute Realtime and live sessions log once at close, so counting them put a long session's tokens into one TPM window and refused calls that stayed under the limit. They count no TPM again, as before this branch. Also covers the legacy v1 count-once fix with an HTTP regression case. * test(proxy): check each per-turn TPM window on its own * fix(proxy): send a fail-closed per-turn socket refusal with its 503 status --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
fbd126e518
|
test: fix integration tests that fail on test order, races and intended changes (#49310)
Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose |
||
|
moe-berri
|
67954b5c3a
|
fix(proxy): apply key and team router_settings to /v1/images/generations (#49286)
* fix(proxy): apply key and team router_settings to /v1/images/generations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): format router settings helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve key model_group_alias once on image generations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tavren27
|
43c07a0f12
|
fix(router)!: check and count deployment itpm and otpm on every Responses WebSocket turn (#49069)
* fix(router): check and count deployment itpm and otpm on every Responses WebSocket turn The success hook only reconciled an ITPM/OTPM reservation found in top-level metadata or the standard logging object, but on the proxy the reservation lives in litellm_params.metadata, so no reservation was ever settled there. The hook now always calls the reconcile, which already returns early without a reservation and clears what it settled, so a second success event is a no-op. A Responses WebSocket turn after the first now has its deployment's current-minute ITPM and OTPM counters checked and is refused with a 429 frame once a counter is at its limit. An admitted turn carries a zero reservation on the current minute's keys, so its actual tokens land on those counters on both the native and the managed relay. * fix(responses): settle the socket open's itpm/otpm reservation on the first turn only After the first turn takes its logging copy, both websocket relays drop the open's ITPM/OTPM reservation keys from the connection metadata, the same way the budget reservation is detached. Later turns, unsolicited backend events, turns after limits are removed mid-socket, and sockets with no turn admission no longer settle or refund the open's reservation a second time. * refactor(responses): read the itpm/otpm reservation keys from core metadata helpers instead of router_utils * fix(responses): keep a socket open's hold for its first real turn and check tokens before request limits A turn that will settle the socket open's itpm/otpm reservation no longer counts that reservation against itself or swaps it for a zero one, so Codex's warmup frame on a managed provider, or a refused first frame on the native socket, no longer blocks the first real prompt or strands the hold for the minute. The token check now runs before the rpm/rpd checks, so a turn refused for tokens never uses up a request slot. Token-check tests pin the clock, and the managed relay settle test drives a real streamed call through a custom provider. * test(router): pin the router clock in the socket turn admission tests Both tests seeded this minute and the next to dodge a minute boundary. They now use the file's pinned-minute fixture and seed only that minute, so neither depends on when the test runs. * fix(router): check a responses socket's open without reserving and read shared token counters first A Responses WebSocket open now runs the same itpm/otpm check a turn runs and reserves nothing. It used to reserve an estimate that falls back to the model's max output, so a socket could not open under any otpm below that. The open still records a zero reservation for its own minute, so the first frame, which the relay forwards without a turn check, counts its actual tokens like every later turn. This drops the plumbing that only carried the open's hold into the first turn. The turn admission callable takes no arguments again, and the native relay detaches only the budget reservation, as before this change. The itpm/otpm turn check reads Redis first and falls back to this instance's in-memory count only without Redis or when the circuit breaker is open. These counters go down when a call settles, so an in-memory copy at the limit could refuse turns after the shared count dropped. The tpm check keeps its in-memory shortcut, since its counter only grows within a minute. Responses usage that reports input_tokens_details.cached_tokens now leaves cached input out of itpm, as chat usage already did. * fix(router): leave a socket turn's cached input out of itpm The native Responses WebSocket relay logs a turn with its list of events, so the deployment rate limit hook finds no usage on the response and falls back to the standard logging payload. That fallback read cached input from metadata.cache_read_input_tokens, a key the payload's cleaned metadata never carries, so every cached token of a socket turn was counted against itpm. The fallback now reads the cached count from the payload's usage_object, the same usage the logging path already built for cost, so a native socket turn counts only its uncached input. The managed relay and HTTP /v1/responses already counted correctly through the response's own usage * fix(router): mark only responses socket opens as check-only and count a turn on the minute it finishes The check-only branch for a socket open keyed on a websocket value in the request kwargs, which any HTTP body can carry and which the realtime route also sets. The router now passes an opens_responses_socket flag into set_io_token_rate_limit_request_kwargs only for its _aresponses_websocket call, and the flag lives in the same context value as the request kwargs, so no request body can set it and realtime opens reserve and get refused as before A zero reservation holds nothing to refund, so a socket turn now counts its real tokens on the minute the call finishes, read from the clock at settle time. A non-zero reservation still settles on the keys it reserved on so its refund hits the same counter When the shared Redis read returns nothing, the check now falls back to this instance's in-memory count instead of admitting the turn with no count at all * fix(router): keep io token reservation keys out of the metadata sent to providers The itpm/otpm pre-call check wrote its reservation keys into both metadata and litellm_metadata. On the routes the router serves through its generic helper (Responses, Anthropic Messages, and the rest), metadata is the client's own field and goes to the provider as is. So _litellm_itpm_cache_key and _litellm_otpm_cache_key, which carry the deployment id and the model name, reached the provider, came back in the echoed response metadata, and counted toward the provider's 16-key cap. The managed Responses socket relay merged each turn's reservation into the frame's metadata the same way, and a native socket opened with metadata sent the keys on every turn through its request defaults The router now passes the limiter the metadata field it uses for its own fields on that call (metadata for chat, litellm_metadata for the generic routes), and the reservation goes only there. metadata_channel is a required keyword, so a new caller has to name the field it writes. The managed relay puts each turn's reservation only on litellm_metadata With the reservation only on litellm_metadata, a retried call on the generic routes refunded its first attempt twice and pushed the deployment's counters below zero. The failure hook read the reservation from the logging copy in litellm_params.metadata and cleared only that copy, so the router's litellm_metadata still held it and the retry refunded it again. The hooks now also read and clear litellm_params.litellm_metadata, which is the router's own dict The proxy does not strip client keys that start with _litellm_ on the way in, so dropping that prefix at the provider boundary would also drop keys clients send today The io token tests now run Redis in process with fakeredis instead of a TCP fake server * test(responses): count socket turns through real logging callbacks instead of a Logging subclass The native relay test now hands the rate limit check to a real Logging object as its success and failure callback, so the turn's tokens go through the same dispatch path production uses. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
paveln83
|
b91fc92b6d
|
fix(router): cap the default OTPM reservation and reconcile token counters after each call (#49054)
* fix(router): cap the default OTPM reservation for requests without max_tokens * fix(router): reconcile ITPM and OTPM reservations after every successful call * fix(router): read the default OTPM reservation from the served model and make its fraction configurable The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check. * fix(router): refund a failed attempt's io token reservation in make_call The proxy passes one logging object to every retry, and it runs failure callbacks only for the first attempt, so the last attempt's ITPM and OTPM reservation stayed held until the minute ended. Each attempt that raises now gives its reservation back before the retry or fallback * fix(router): count Anthropic cache writes once in ITPM reconcile Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens. * fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction * fix(router): reserve with the deployment cap only through the param the request's API sends * fix(router): never let a deployment's output cap raise the reservation above the default * test(router): move the reservation fraction env var test to the integration suite The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation. * test(router): retry the reservation fraction scenario when the minute rolls over The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute. * test(router): give the rollover retry its own deployment --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
feda51453a
|
fix!: retain unattributed daily activity across reports and exports (#48925)
* fix: retain unattributed daily activity breakdowns * fix: scope daily activity bucket drilldowns consistently * fix: preserve all additive daily activity metrics * test: cover unattributed reporting end to end * fix: preserve daily activity bucket identities across exports and key details * test: drop SQL-text assertions changed by daily activity bucket fix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
6cdb9959df
|
feat(router): opt in to non-chat deployment filtering (#46257)
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior. * fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions * fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog * fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags * fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released * fix(proxy): walk fallback layers without recursion and gate the compact Responses route The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes * fix(proxy): keep chat on rows whose cost map entry serves chat A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s * fix(proxy): check chat sent to a deployment id or backend model name * fix(proxy): check a model the read-through loads from the database * fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400 * fix(proxy): ignore a client model_group that is not a string when building rate-limit headers A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells. * fix(proxy): keep a non-string client model_group out of spend rows * fix(proxy): validate standard fallbacks before the typed fallback-group lookup * feat(router): opt in to non-chat deployment filtering * fix(router): skip unresolved local rate-limit fallbacks * fix(router): ignore caller-provided endpoint selector values * fix(router): normalize declared chat endpoint prefixes * test(router): cover preflight error-value behavior * test(router): derive replay metadata defaults from model info * fix(router): preserve fallback aliases and check synchronous text calls * test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission * test(routing): keep the endpoint gate's owned proxies on the worker's database * fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None so an omitted flag keeps the stored value The same-group retry of a stream dropped before content called the provider without the endpoint selection context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside endpoint_selection like every other router attempt * fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow Main's router fence lets code outside the router use only listed Router members and router_utils modules. The proxy's fallback probe now calls the listed get_routable_deployments with the request's call type, and the gate's 400 moves to litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's UnsupportedEndpointError, which the router retries on the group's other deployments, while the gate refuses the whole group * test(responses): cover the WebSocket error frame for a model that refuses the endpoint * fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup * fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame * fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path * test(router): type the load-warning test's caplog and router * test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket --------- Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain> |
||
|
Kerry Lu
|
0f786fa1c7
|
feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits (#48875)
* feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): route rate limit cache increments through typed helpers * chore(ui): regenerate schema.d.ts for deployment rpd and tpd fields * test(types): add rpd and tpd to the owned litellm_params inventory * fix(router): run daily rpd/tpd checks after the minute checks and increment rpm and rpd in one cache trip * fix(router): flatten the async rpm reject branch for the ruff strict gate * fix(router): hand rpm and rpd back on any rate limit reject A request the rpm, rpd or tpd check rejects no longer keeps its request counts, so retries against a deployment whose rpd is spent cannot fill its rpm and block the in-group fallover with the minute 429. * test(router): a minute-rejected request no longer keeps its rpm count --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tessellark
|
41c4c3788e
|
test(routing): audit #46601 (#48910)
* test(routing): audit #46601 Adds integration cells for the falsy budget 400, the empty and none-only declarations under drop_params, Vertex AI Claude, Databricks Claude, and the Bedrock native output retry * test(routing): expect the Anthropic default max_tokens on Bedrock Invoke chat Reads the default from litellm instead of pinning 4096, which #48502 raised to 128000 * test(routing): assert whole sent bodies in the Vertex AI and Bedrock native output cells --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
ca4a2ba006
|
fix(router): drop the deployment rpm filter from enable_pre_call_checks (#48873)
enable_pre_call_checks filtered deployments whose in-memory minute count
reached litellm_params.rpm, undocumented since
|
||
|
tavren27
|
826859588c
|
test(router): audit #41368 (#48794)
* test(router): cover silent_model shadow fan-out and streaming end to end Proxy cells check that each silent_model target gets one request in the caller's stream mode, that the caller sees only the primary, and that every shadow lands its own spend row under its own model group. SDK cells check the sync Router.completion thread path streams each shadow and fires its success callback. * test(router): flatten the spend row lookup with itertools.chain --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
norkelbyte
|
64d545ba7c
|
test(router): audit #47376 (#48792)
* test(router): audit #47376 batch create routing and supported_endpoints listing * test(router): make the batch mode cells deterministic and assert concurrent creates --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
mika-lt
|
9546f9a69f
|
test(router): audit #47498 (#48799)
Pin the queue route contracts #47498 changed: the priority header on streamed answers, the 400 for a request with no priority to queue at, a provider bad request keeping its own status, and a deployment-id request served ahead of the queue Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
8f31c7171d
|
fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception (#48644)
* fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): give cooldown fixtures a sibling deployment so allowed_fails=0 still forces a cooldown * fix(router): keep benching a lone deployment that has a fallback group or an empty deployment policy * fix(router): accept dictionary fallback targets in the lone deployment check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
9e01b3c37b
|
fix(complexity_router): keep the held user_turn route for Codex compaction requests (#48627)
* test(complexity_router): Codex compaction requests hold the user_turn decision Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(complexity_router): treat Codex compaction traffic as user_turn continuation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(complexity_router): validate Codex turn metadata with frozen models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(complexity_router): flatten body system text through _message_text Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(complexity_router): drop unrelated reformatting from the mapped test file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(complexity_router): skip pin escalation on Codex compaction turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b5befe2dd0
|
test(routing): a team's wildcard serves its keys before a proxy-wide wildcard of the same pattern (#48501)
Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
dd511a2b22
|
fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)
* fix(logging): fire one CustomLogger hook per async call outside the async flag list aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are missing from is_sync_litellm_request's flag list, so the sync success and failure handlers called a CustomLogger's sync hook on top of the async hook the async handler had already called for the same request. The sync handlers now skip a CustomLogger's log_success_event and log_failure_event when that run is paired with an async handler run (for_async_request) and the async handler gives the logger its own async hook. The async client wrapper, dispatch_failure_handlers, the proxy-only failure dispatch and the native bridge's sync failure step for an async call mark their sync failure run as paired. Resolves LIT-9486 * fix(logging): keep the sync CustomLogger hook when no async success runs The async client wrapper skips async success logging for calls carrying a fallbacks kwarg, and the native bridge skips it for internal and fallbacks calls, yet the paired sync success run still dropped a both-hooks logger's sync hook, so that logger got no success event at all. The sync success run now suppresses the sync hook only when the async success handler was actually scheduled for the call: the client wrapper passes async_handler_runs=not is_completion_with_fallbacks, and the native host records per call whether it enqueued or deferred the async success. Tests that pinned the doubled failure hook on native aocr, messages and responses calls and on proxy-rejected requests now expect the async hook alone. The integration test adds /v1/messages and /v1/responses cells and runs the key-auth rejection on every route. * fix(logging): type dispatch_success_handlers kwargs The new async_handler_runs keyword on success_handler made the untyped **kwargs that dispatch_success_handlers forwards an unknown argument for it, which grew the reportUnknownArgumentType count past its merge-base ceiling. Typing the forwarded kwargs as Any, as _success_handler_body already does, removes those diagnostics without changing what is forwarded. * test(integration): cover CustomLogger hook counts across modes, chaos and router strategies Moves the hook count rig into tests/integration/_support/hook_counts.py and extends the cells it drives: - streamed chat, Messages and Responses requests, success and failure - loggers the async handler never reaches (sync-only overrides, sync event lists only) keep one sync hook per request on every surface outside the async flag list, key-auth rejections included - a concurrent burst while the logger's destination stalls and then fails, and a burst while a worker is killed, never give a request two hooks - least-busy keeps counting a request in flight while async speech requests finish around it, and usage-based routing v2 counts an async image request's tokens once * test(integration): Datadog and LangSmith deliver one record per async request Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local sink and checks every async surface outside the flag list: one Datadog log entry and one LangSmith run per request (two on the merge base), and never two OpenTelemetry spans, which its own emit-once guard already ensured. * fix(logging): give a both-hooks logger its sync hook when the async run never reaches it The sync hook was skipped whenever the logger sat in the async callback list, so a logger the async run never reached got no hook at all: a sibling stalled past the logging worker timeout, a batch poll that skips default logging, or an async failure run cut off. The sync run now holds the hook and the async run hands it back unless it reached the logger * fix(logging): run the held sync success hooks when the proxy drops a blocked response's success A post-call guardrail block drops the parked async success, which left the sync success hook held for it unrun. Base gave a both-hooks logger that hook on these call types, so the drop now ends the pairing and the held hook runs as before * fix(logging): run held sync hooks only after every sync run of the request has closed A held hook released by the async run could run on another thread while the sync run was still walking its other callbacks. The pairing now waits until the async run has ended and every sync run has closed, and the last sync run to close runs the held hooks itself * fix(logging): end the async run when its coroutine is dropped before it starts A logging queue that drops the async success coroutine, or a deferred success that fails to schedule, left the sync hooks held for it unrun. The enqueued async run now ends when its coroutine is discarded, and the proxy ends it when firing the deferred success raises * fix(logging): leave a still-queued async run to the exit flush The discard finalizer ran at interpreter exit before the logging worker's exit flush, so a queued request could get its held sync hook and then its async hook * test: keep the router and file-stream tests in step with main Main gives each owned proxy a private Redis by default, so the router cells share the worker's Redis to read its counters, and the file-stream stub takes the hook pairing the stream now passes * fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up A during_call guardrail that raised after the LLM call parked its deferred success skipped the proxy's flush, and async runs the exit flush never started were never ended, so a both-hooks logger lost its sync hook in both cases * fix(logging): skip the sync failure hook when the async failure run logs to the logger An async call outside is_sync_litellm_request's flag list ran both failure handlers, so a CustomLogger with both failure hooks got both. The sync run of an async request now skips log_failure_event for a logger that overrides async_log_failure_event and is registered for async failures. The four sync runs of async failures (dispatch_failure_handlers, the @client async wrapper, the proxy-only failure thread and the Rust host) pass for_async_request=True. Drops the HookPairing fallback. Successes are left to #47192, which already gives such a logger only async_log_success_event. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
e78f7b4602
|
test(integration): repair seven stale integration tests on main (#47600)
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply |
||
|
Mateo Wang
|
f925c067d2
|
fix(tests): read owned proxy rows from the owned proxy's database (#47333)
* fix(tests): read the stream stall cooldown spend rows from the owned proxy's database The cooldown cells boot an owned proxy, which writes its spend logs to its own scratch database since owned proxies went private, while the cells polled the worker's database and never saw a row. Read each row through the gateway that served the request, so the cells check the proxy they called. * fix(tests): read passthrough spend rows from the owned proxy's database The vertex regional uplift and passthrough stream pricing spend files boot private owned proxies, then polled the worker database for spend rows that only reach the proxy's own database. Read them through the owned proxy's gateway instead. Also hold the stream stall file's hold_headers peer past the longest row wait, so a slow spend row cannot fail the peer teardown. * fix(tests): check agent deletion on the owned proxy's database The agent least privilege test deletes its agent through an owned proxy, then checked the worker database for the row, which is always empty there. Read it through the owned proxy's gateway so a delete that leaves the row fails the test. * fix(tests): read turn admission spend rows from the owned proxy's database |
||
|
Mateo Wang
|
d698106ed7
|
fix(router)!: schedule default_priority requests once and reject a malformed priority (#47498)
* fix(router): schedule default_priority requests once and reject a non-integer priority A router configured with default_priority failed every request that carried no priority with a 500 naming a missing 'priority' argument, and an explicit priority re-entered acompletion and hit the same error. acompletion now resolves the priority once (the request's integer, else the default, with 0 kept as a real priority) and schedules through _schedule_factory directly, while schedule_acompletion delegates to acompletion so the two paths share one queue pass. A non-integer priority answers 400 naming the param, or is dropped under drop_params, instead of leaking to the provider or crashing the scheduler. * fix(router): honor router-level drop_params and schedule prompt-management models under a priority * fix(router): drop a non-integer priority when the model's deployments set drop_params * fix(router): resolve the drop_params deployment scope for team routes and deployment ids * fix(router): read the request and router drop_params flags without a loop-assigned Final * refactor(router): type the prompt-management wrapper's return and mock the provider in its test * perf(router): consult drop_params for the priority only when it is not an integer * test(integration): assert the new priority contract for non-integer and zero priorities * docs(router): describe default_priority as the priority every request without one takes * fix(router): parse default_priority once and apply it to text completions and the queue route A digit string default is taken and any other non-integer default is ignored with a warning instead of failing every request. atext_completion and /queue/chat/completions resolve the priority the way acompletion does: a request without one takes the default, an integer queues once, and a non-integer answers 400 or takes the default under drop_params. Integration cells cover every default-priority surface on an owned two-instance rig. * fix(router): type the priority dispatch so the lint budgets hold * fix(router): read priority like the scheduler did, require a priority on the queue route, and keep litellm's drop_params order * fix(router): drop a priority named in additional_drop_params and report the queued priority on streamed queue responses * fix(router): serve deployment-id requests without waiting behind the queue and ignore a boolean default_priority A request whose model is a deployment id now counts that deployment as healthy in the scheduler, since routing by id already ignores cooldown. Before, the scheduler found no healthy deployment for an id, so the request waited behind every entry ahead of it. A boolean default_priority (true, or YAML off) is ignored at startup with a warning, like any other value that is not a whole number. The scheduler integration tests now share worker state for the defaulted pair, cool a group through /v1/responses so no stale in-memory queue is left behind, and use "high" instead of "1" for the non-integer cases. The concurrent burst test no longer asserts that the queue drains, matching the explicit-priority burst test, since the scheduler's read-modify-write queue can race across workers. * test(router): keep the queue drain check as a BUG-skipped integration test * fix(router): queue default_priority chat requests once instead of failing them * test(router): open with the default_priority regression tests alone * fix(router): honor deployment drop_params for a malformed priority on a request routed by provider model name |
||
|
yucheng-berri
|
7ae9a1c350
|
fix(router)!: only prefixed caller tags route once tag_routing_prefix is set (#47428)
* fix(router): scope tag routing to prefixed tags once tag_routing_prefix is set With router_settings.tag_routing_prefix configured, an unprefixed caller tag (for example a spend tag such as user_id:234) no longer drives tag routing, so it can no longer refuse a model group whose deployments are untagged. Key/team/project tags (metadata.inherited_tags) keep applying with or without the prefix. Rows of batches the proxy runs itself now carry those tags as inherited_tags too, so they keep routing on them once the prefix is set. Takes over #39903. Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com> * test(integration): pin that proxy-run batch rows keep key/team tag protection * refactor(router): filter routing tags from the raw request tags; deterministic tests and key-tag cells * test(integration): audit cells for tag_routing_prefix: project tags, forged inherited_tags, odd tag values, outage, fallback, user-agent routing, concurrent burst * test(integration): proxy-run batch rows keep key and project tags too --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> Co-authored-by: Deepanshu Lulla <deepanshu.lulla@gmail.com> |
||
|
ekpog200
|
ba61ecdf57
|
fix(videos)!: keep model_id in proxy video ids so per-model keys resolve on status/content (#44614)
* fix(videos): keep model_id in proxy video ids so per-model keys resolve on status/content
Provider transforms encode returned video ids with an empty model_id, so
GET /v1/videos/{id} and /content cannot resolve the deployment (and its
per-model api_key) and fail with invalid_api_key.
Re-stamp the returned VideoObject id with the resolved model_id in
video_generation, video_status, video_remix, video_edit and
video_extension.
Fixes #33423
* fix(videos): return a copy instead of mutating the response, read hidden params via helper
* test(videos): fully type the new video id re-stamping tests
Annotate every parameter and return added by this PR (hidden_params,
harness, response, -> None), drop Any from the test helper, mark locals
Final, and narrow the endpoint/helper results with isinstance(VideoObject)
instead of reading .id off an untyped value. Route the mocked processor
response through a typed Harness.respond_with, read NotRequired
DecodedVideoId keys with .get, and name the rule and reason on the
_hidden_params suppressions.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(videos): fill only an empty model in returned video ids
Keep an id that already names a model as is, so the asked-for name,
routing group and served group in it still drive access checks and
routing. Fill an empty model from the model the caller asked for
before the routed deployment id, and read the request model through a
typed view so the type gate does not grow
* fix(videos): route video calls to the deployment that created the video
Video ids now carry the id of the deployment that created them. Status,
content, remix, edit and extension calls go to that deployment when it
belongs to the requested group, so a group spread over two provider
accounts no longer polls the wrong one. A provider model shared by
several groups resolves to the creating deployment's group, and the
asked-for model is written into an id only when the router can route it.
* fix(videos): pin video calls only to deployments the caller may use
Pin a video id's creating deployment at deployment selection, and only when the caller may use that
deployment in the requested group, so aliases, routing groups and access groups still apply and the
spend label and fallbacks keep the requested group.
A status poll keeps the deployment its id names and takes the deployment that answered when the id
names none. An alias that routes to the creating deployment's group is kept on status and content.
* test(router): reset module fallback globals between unit tests
* fix(videos): check and route polls by the name the router serves
A poll on a video created through a provider wildcard is checked against the name the wildcard serves, a team model is checked by its public name, a poll stamps the deployment that answered, and the sync Router pins video calls the same way the async path does
* fix(videos): pin polls only among the deployments routing allows
Video calls route by group again and the creating deployment is picked only from the deployments left after access, team, health, cooldown, budget and tag filters, so a pinned poll no longer skips them. Wildcard groups now pin too. Adds tests for cooldown, wildcard pinning and character lookups through a provider wildcard, and return types on the auth tests
* fix(videos): scope the video pin to video calls, keep fallback-served videos reachable, skip cooldown on video 404s
* fix(videos): stamp the requested group only when one of its router fallbacks served the create
* fix(videos): follow chained router fallbacks for fallback-served videos, test with a real Router
* test(router): list the video pin helper with its sibling in the router coverage exemptions
* refactor(auth): walk chained router fallbacks without recursion
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Mateo Wang
|
96f1bc02d8
|
feat(router)!: apply each deployment's reasoning_effort_levels on every provider and route (#46601)
* feat(router): apply each deployment's reasoning_effort_levels on every provider and route A deployment's declared reasoning_effort_levels now decide the effort sent upstream on /v1/chat/completions, /v1/messages and /v1/responses for every provider. Requests round up by default, litellm.reasoning_effort_rounding and model_info.reasoning_effort_rounding switch to down, a declaration lets providers like azure_ai accept reasoning_effort, and each remap logs one INFO line with structured fields. * fix(router): round reasoning effort only on a deployment's own declared levels Cost map levels no longer round requests or force-allow reasoning_effort, so undeclared sibling deployments and catalog-only models behave as on main. A declaring deployment lets azure_ai accept any level the resolver sends, and a hosted_vllm chat thinking budget is rounded too. * fix(router): drop undeclared reasoning effort under drop_params and round native /v1/messages effort Under drop_params a level outside the deployment's reasoning_effort_levels is dropped instead of sent. Providers that take /v1/messages natively now round output_config.effort onto the deployment's declaration. The rounding setting ignores case and surrounding spaces. * fix(router): strip line breaks from every field in the reasoning effort remap log * fix(router): round thinking-budget efforts onto a declaring deployment on every provider * fix(router): keep the caller's thinking display when rounding a thinking budget on adaptive Claude Warn on a reasoning_effort_rounding value other than up or down instead of silently rounding up * test(integration): cover deployment reasoning_effort_levels rounding on chat, messages and responses Wire cells assert the exact body each provider receives when a deployment declares reasoning_effort_levels, plus undeclared controls, malformed input, drop_params, rounding settings, fallbacks, model groups, live updates and the response cache. Chaos cells cover a killed worker, a proxy restart, an upstream outage, a slow stream and the INFO remap log. * test(integration): cut held streams when the upstream stops in the reasoning effort chaos cell The upstream-stop cell waited for the held streams before stopping the upstream, so they only ended on a client or proxy timeout and the result depended on which fired first. The upstream now stops while the streams are held, and each cut stream must carry an error and no answer * fix(router): round thinking budgets only on declaring deployments and round Converse output_config An undeclared hosted_vllm deployment serving a Claude model name now maps a thinking budget the same as any other undeclared hosted_vllm deployment. A model_info that echoes a cost map key no longer borrows that entry's levels. A non-numeric thinking.budget_tokens answers 400, or is dropped under drop_params, where main answered 500. A declaring Bedrock Converse Claude deployment now rounds output_config.effort, which /v1/messages sends for Claude Code * test(integration): cover output_config rounding and non-numeric thinking budgets in the reasoning effort cells * test(integration): keep an explicit reasoning_effort winning over a malformed thinking budget on hosted_vllm * test(integration): cover output_config rounding on openai, azure and azure_ai chat * fix(hosted_vllm): let an explicit effort win over a malformed budget and reject one on /v1/messages An explicit reasoning_effort beside a non-numeric thinking.budget_tokens keeps main's behavior again. A declared hosted_vllm deployment with a Claude model name maps a budget through the generic table instead of the adaptive Claude one. /v1/messages to a non-Claude target hands a malformed budget to completion(), which answers 400 or drops it under drop_params, where it answered 500 * fix(router): drop thinking beside an explicit effort when a declaring deployment's provider cannot read it * test(integration): cover malformed thinking budgets on /v1/messages to non-Claude targets * fix(anthropic): answer 400 for a malformed thinking budget on /v1/messages to any non-Claude model, or drop it under drop_params * test(integration): cover an explicit effort beside thinking on azure_ai and drop_params on the /v1/messages bridges * fix(anthropic): answer 400 for a malformed thinking budget through adapter_completion without drop_params * fix(anthropic): type the adapter_completion request so the drop_params read adds no unknown argument * fix(router): a default effort beside a thinking budget sends no effort on chat Chat now leaves the budget-derived effort out when the caller's effort resolves to none or asks output_config for one, matching /v1/messages. The chat thinking decision moves into resolve_chat_reasoning so completion() stays under basedpyright's complexity limit. * test: type respx_mock and mark the adapter result Final in the reasoning effort tests * fix(router): keep the caller's thinking beside a kept effort and the budget effort beside a dropped one A non-default output_config.effort beside a thinking budget now leaves the thinking as sent, so Bedrock Invoke keeps display updates. A level drop_params drops counts as never sent, so the budget still sets the effort. output_config.effort replaces a thinking budget the provider cannot read, and a non-numeric string budget on adaptive Claude answers 400 in place of 500, or drops under drop_params. * fix(router): send a kept output_config effort as reasoning_effort where the provider cannot read thinking On a declaring hosted_vllm or azure_ai deployment, a thinking budget goes out as reasoning_effort. An output_config.effort beside it now takes that slot, rounded onto the declared levels, instead of leaving only output_config, which those backends ignore. * fix(reasoning-effort): honor global drop_params for a non-numeric budget and keep the budget effort beside a dropped /v1/messages level * fix(reasoning-effort): send a level a gpt-5 deployment declares even where the cost map does not list it * fix(reasoning): apply declared levels on azure_ai gpt-5 and reject a bool thinking budget on adaptive Claude * fix(router): drop a thinking budget's effort that a deployment declaring no ranked levels excludes A deployment declaring [] or ["none"] skipped the budget check, so adaptive Claude still sent the budget-derived output_config.effort under drop_params on chat, Bedrock Converse, and native /v1/messages. The adaptive thinking now goes out without that effort * fix(anthropic): reject a false, null or empty thinking budget on adaptive Claude |
||
|
devin-ai-integration[bot]
|
c80f1c2ec8
|
fix(router)!: count native Responses WebSocket turns toward deployment limits and budgets (#45727)
* fix(responses): write one spend-log row per native WebSocket turn Each response.create turn on a native Responses WebSocket connection now logs its own call as it finishes, with its own usage and the resp_ id the client saw as request_id. A failed turn is booked alone and no longer flips the other turns. * fix(responses): bill the usage a failed native WebSocket turn reports * fix(proxy): pin the limiter slot owner to the proxy call id and refund unsettled reservations on disconnect The v3 limiter stashed whichever litellm_call_id the request body carried as the slot owner, so a client-supplied id on a pass-through body left the proxy's own success and failure events unable to release the slot. The owner is now the proxy's logging call id. A WebSocket connection whose turns log under their own ids never settles the TPM, ITPM, or OTPM estimate the connection reserved, so the disconnect release now refunds any reservation no callback settled, once. * fix(proxy): bill failed WebSocket turns and refund only the reserved window on disconnect A failed turn's reported usage now counts toward the key's TPM and the project ITPM/OTPM scopes the same way a success does, and closing the socket hands back only the estimate still unused in the window that holds it, so a window that already rolled is left alone. An id-less error event is booked against the newest pending turn, the one the backend rejected at parse time, and a turn that nests its input under response logs its user message. A connection's one budget reservation goes to the first turn that settles it, every later turn charging its cost directly. * test(proxy): expect the settling call id on a pass-through budget reservation * fix(proxy): charge successful Responses WebSocket turns to TPM and ITPM/OTPM A native Responses WebSocket turn reaches the rate limiter's success callback with its raw event list as the response and no combined_usage_object, so the turn's tokens never reached the key's TPM window or the project's input and output token limits. Read the usage from the turn's events when the call type is a Responses WebSocket turn. * fix(proxy): release a rejected pass-through's slot and count each websocket turn once toward project quotas The limiter's failure hook looked up the parallel slot by the request body's litellm_call_id, while the slot belongs to the proxy's logging call id, so a pass-through request a pre-call hook rejected held its slot for an hour when its body carried an id. Both now resolve the id the same way. Each response.create frame already charges its estimate to project ITPM/OTPM, so the turn's own usage no longer adds a second count. The cost callback's failure hook read the body's litellm_call_id through a strict model and raised on a non-string value, which dropped the failure row. It now reads a string id or none. * fix(proxy): keep per-turn WebSocket limits, request bodies and collector reservations right Free a rejected request's parallel slot through the request it runs in, and skip native Responses WebSocket connections, whose disconnect frees the slot. Count WebSocket turns toward the connection's tag TPM. In collector mode, claim the shared budget reservation before the spend event is built, so only the first turn settles it. Store each turn's own frame as its request body, and send an id-less error to the oldest waiting turn unless an earlier turn is streaming. * fix(proxy): store a failed WebSocket turn's own request and count tag TPM only for calls that carry the tag * fix(responses): log the client's turn frame, keep the payload id and reservation shape for non-WebSocket calls * fix(proxy): keep the legacy limiter's WebSocket slot until the socket closes * fix(router): count native Responses WebSocket turns toward deployment limits and budgets * fix(router): re-check deployment budget and RPM on each Responses WebSocket turn * test(router): list the WebSocket turn admission method as covered through its partial * fix(responses): give a refused WebSocket turn's error frame a 429 status so Codex ends the turn * fix(router): leave ITPM/OTPM to socket open and record a refused WebSocket turn as the socket's failure * fix(proxy): log the routed model group on a WebSocket turn's failure row * fix(router): count WebSocket turn tokens toward TPM and re-run routing strategy checks on each turn * test(integration): cover per-turn admission on Responses WebSockets Adds integration cells for budget, RPM, TPM, a mid-socket TPM edit, usage-based-routing-v2, the project quota frame, and the open-time failure frame on the native and managed relays, plus HTTP controls for TPM counting and the failure row model_group * fix(responses): count WebSocket turn tokens once and end a socket whose turn check fails A native Responses WebSocket turn is now logged as an async call, so a proxy with a sync success callback such as the response cache no longer runs each CustomLogger's sync success hook on top of the async one. That second run counted every turn's tokens twice toward deployment TPM. A turn whose admission check raises an unexpected error now gets a 500 error frame, the socket closes with 1011, and the error becomes the socket's failure, on both the native and the managed relay, instead of the socket dropping with no frame and no failure row. A refused turn's failure row now carries the request's tags from the router's metadata, the same way it already carries the model group. * test(integration): cover a crashed WebSocket turn check and failure-row tags A turn whose admission check raises gets a 500 server_error frame, the socket closes with 1011 and its one failure row carries the error, on both relays. The refused turn's failure row keeps the caller's tags, and HTTP failure rows on chat, responses and messages keep them too. * fix(responses): keep a WebSocket open after a crashed turn check and keep only string tags on failure rows A turn whose admission check raises an unexpected error still gets a 500 error frame, but the socket now stays open, like a refused turn. Closing it cut any turn still streaming on the same socket, which a client that sends its next response.create early would lose. A refused turn's failure row now takes the router's tags only when every tag is a string, so the fallback never copies a caller's unvalidated tags onto the row. * test(integration): cover a turn still streaming when the next WebSocket turn's check crashes A native turn still streaming when the next turn's admission check raises now has its own cell. It must complete and be billed at the turn's price. The crashed-check cell also keeps the fault set for one more turn, so a relay that stopped checking turns after a crash goes red. * fix(responses): run per-turn guardrails before the deployment's turn check on WebSockets A turn after the first now runs the project quota check, then the proxy's pre_call guardrails, then the deployment's turn admission, on both the native and the managed relay, the order HTTP calls use. A turn a guardrail blocks gets the guardrail's error even past the budget and no longer counts toward the deployment's rpm or runs its pre-call checks. * fix(responses): bill a WebSocket turn's guardrail cost when the deployment refuses it A turn the per-turn guardrail passes and the deployment then refuses now writes its own failed spend row with that turn's guardrail information, on both relays, the way a guardrail-blocked frame and an HTTP refusal already do * fix(responses): bill a refused WebSocket turn on one row only A refused turn whose guardrails recorded a cost is already billed on its own row, so it no longer also becomes the connection's close-time failure row * fix(responses): bill a refused WebSocket turn's guardrail cost on the socket's failure row The turns a deployment refuses after their guardrails passed now add their guardrail records to the socket's close-time failure row instead of a row of their own, so that row's failure hooks, model group and call type cover them * fix(responses): bill a WebSocket's refused turns when the session limit closes it * fix(responses): bill refused WebSocket turns when the session limit cancels the relay * fix(responses): free the WebSocket slot at the session limit without waiting on the failure row * fix(responses): keep the session limit's second cancel so a relay that swallows the first frees the slot * fix(responses): free the WebSocket slot before the close-time failure row and drain pending rows at shutdown * fix(proxy): commit spend before waiting on pending logging tasks at shutdown and again after * fix(responses): start the close-time failure row before the slot release so a late cancel cannot drop it * fix(responses): build the close-time failure row inside its task so a bad guardrail record cannot skip the crash close * fix(proxy): always commit spend again after the shutdown wait for pending logging tasks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
44d99475a5
|
test(integration): stop GET /__observations from draining the upstream log (#47256)
* test(integration): stop GET /__observations from draining the upstream log GET /__observations used to empty the queue, so a second read in a test saw nothing and a "never reached upstream" check could pass whether or not the request was forwarded. GET now returns every request since the last reset and DELETE /__observations is the reset. An autouse fixture resets the worker upstream before each test, which also covers tests that read without the gateway fixture. Discarded reads used as resets become DELETEs, helpers that accumulated drained reads now read one snapshot, and tests with several phases reset between them. The three vacuous negative checks (rag allowlist, guardrail opt-out chaos, xAI batch output) now assert the absence on the same read that shows a request which did reach the upstream. * style(integration): space the observations unit test like its neighbours |
||
|
yuneng-jiang
|
fc27dfdc7d
|
test(integration): give each owned proxy a private database and Redis by default (#47213)
* test(integration): give each owned proxy a private database and Redis by default owned_proxy and owned_proxy_process now boot on a scratch database and a per-proxy redis-server unless the caller opts into the worker's state with state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips DATABASE_URL. Under the replica profile the private database gets the writer and reader roles so the reader path stays covered. Gateway carries the database_url its proxy writes to, and the Scenario cleanup checks and delete_key_if_present read from it, so they keep checking something on a private database. The 189 test files with owned proxy sites that depend on the worker's database or Redis opt in with a shares_worker_state module marker, which a module-scoped conftest fixture turns into the shared state. The 7 helper modules with such sites opt in per call, since conftest fixtures and other test files reach them. * test(integration): reach session fixtures with the shared marker and keep the worker retry default The module-scoped fixture that applied shares_worker_state ran after session-scoped fixtures, so proxies those booted went private in marked files. The marker is now applied around each marked item's setup, call and teardown, which covers fixtures of every scope. Owned proxies on the worker database inherited the worker gateway's router_settings.num_retries of 0 from it. A private proxy now gets the same default in its config copy unless the config sets num_retries itself. Five providers files read the owned proxy's spend rows or response cache through the worker's database or Redis and opt into sharing. |