Router.make_call now increments the deployment TPM/RPM counter before set_response_headers reads remaining usage, so the headers carry post-increment values directly and the in-flight subtraction workaround from LIT-2719 is removed. deployment_callback_on_success adds only the tokens not yet counted (streams) and never a second request. The counter key uses the resolved deployment name so wildcard routes are read back correctly, and the proxy strips the router-owned counted-tokens marker from client metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Budget is enforced once during auth, against the requested model group.
`_is_model_cost_zero` waives every budget check for a zero-cost group, and the
router then picks a fallback target afterwards, inside `run_async_fallback`,
where nothing re-checks budget. A free model with a paid fallback therefore
bills with no budget gate at all.
Add `fallback_budget_check`, the budget sibling of the existing
`fallback_access_check`: a predicate awaited per fallback target that skips
targets the caller cannot pay for. The primary attempt is untouched, so a
zero-cost model is never blocked by budget and only the paid fallback is
refused.
Counter reads pass `max_budget` so `get_current_spend` verifies against
authoritative recorded spend, matching the auth-time key and user checks; a
counter restored from an older snapshot reads as a hit rather than a clean
miss, so without it a stale-low value would keep admitting paid fallbacks.
A zero-cost fallback target is always allowed, and a team key does not inherit
the key owner's personal budget unless `apply_user_budget_to_team_keys` is set,
matching `_PROXY_MaxBudgetLimiter`.
Scope is key and user budgets. Team, team-member, end-user, org, global and
per-model budgets are not covered yet: those auth-path functions enforce rather
than report, so reusing them would fire threshold alerts and take spend
reservations for a target that is then skipped. Two limitations of that scope
are documented in the module docstring: the check reads the spend counter
rather than reserving against it, so concurrent fallbacks can cross a cap
together; and a request reaching the router without
`metadata["user_api_key_auth"]` is not restricted. Both are shared with
`fallback_model_access.py`.
Opt-in via `general_settings.enforce_fallback_budget`.
Relates to #41344
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failure logger skips fallback hops (has_logged_async_failure is already set), so
model_call_details.end_time still belongs to the previous hop and predates this hop's
api_call_start_time. The fallback cooldown guard measured a negative elapsed time and
cooled down deployments for caller-set timeouts.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
client_side_timeout records that the caller configured a timeout, not that
the timeout fired. A 408 the provider returns before that deadline is a
deployment failure and must still count toward cooldown.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A 408 produced by a timeout the caller set (a timeout body field or an
x-litellm-timeout header, which the proxy marks as client_side_timeout)
says nothing about the deployment's health, yet the router's primary
failure callback counted it toward allowed_fails and cooled the
deployment down. The fallback path already skipped it.
The marker never reached that callback because get_litellm_params drops
kwargs outside OPTIONAL_KWARGS_KEYS, so it is listed there now, and
deployment_callback_on_failure returns before the failure counter when
is_caller_timeout_408 holds. A 408 from a timeout the deployment or the
provider set still counts and still cools the deployment down.
RouterRateLimitError now carries the model group's deployment ids so it
can tell when every deployment is cooled down, and exposes that as
type=all_deployments_in_cooldown with an explicit message. A partial
cooldown keeps type=rate_limit_error. Either way the proxy no longer
reports type=internal_server_error next to code 429
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The docstring cited session_affinity_ttl_seconds as the keepalive bound,
but the Router-level knob feeding ttl_seconds is
deployment_affinity_ttl_seconds; session_affinity_ttl_seconds is the
separate per-request PreRoutingHookResponse override. Anyone grepping
the docstring's name to shrink the Router default finds only the
override. Name both, scoped correctly.
The sync Redis read now raises while the circuit breaker is open, and the
health state merge caught that as a generic error, skipping the local write
and logging an error on every background health check cycle. Read the shared
snapshot through a helper that treats the refused read as a miss so the merge
falls back to the pod-local copy the way a swallowed connection error already did
The encrypted_content_affinity check only read the Anthropic history from request_kwargs["messages"], so a caller that passes it through the callback's messages argument alone skipped the pin. Read the argument first and fall back to the kwargs.
When the minting deployment is not a candidate of the routed group, the base already strips the Responses input's encrypted reasoning; do the same for the bridge-tagged thinking blocks in Anthropic messages so the routed deployment gets the readable thinking text instead of ciphertext it cannot decrypt.
The encrypted_content_affinity check only read the pin from the Responses
input, which /v1/messages builds after the router has picked a deployment,
so a model group spread across OpenAI orgs sent follow-up turns to the
wrong org and got invalid_encrypted_content back. The check now also
decodes the pin from bridge-tagged thinking and redacted_thinking blocks
in the Anthropic messages. The bridge also keeps a deployment's own
include list next to reasoning.encrypted_content instead of replacing it.
A team-scoped auto-router is stored under an internal
model_name_{team_id}_{uuid} with the caller-facing name in
model_info.team_public_model_name, and the four pre-routing strategy
registries key on that internal name. A team key asks for the public name,
so the strategy lookup missed, the team early-resolve exit handed back the
marker deployment itself, and every call 400'd with "Unmapped LLM provider".
The strategy lookup now resolves the requested name through the same
team-first, then global, then admin-across-teams deployment resolution the
deployment path uses, and looks the registries up under the model_name of
whatever that resolves to. Both exits of _common_checks_available_deployment
drop strategy markers through one helper, so a marker-only resolution is
rejected as uncallable on every path. The request team id has one reader.
Resolves LIT-7363
Claude-Session: https://claude.ai/code/session_01NU97S7d2FUDDvTk59k53Wp
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
A Responses API follow-up that replays reasoning.encrypted_content is pinned to the
deployment that minted it. Behind an auto-router the pre-routing hook rebinds the model
to the tier it picked before the candidate pool is built, so a turn that classifies into a
different tier never finds the origin and the affinity check raised its fail-fast 503,
whose text claims a cooldown that does not exist
When the deployment that minted the reasoning is not a member of the model group this turn
is routed to, strip the encrypted reasoning (keeping any readable summary, string or block
form) and dispatch to the routed group. Membership is tested by deployment id against the
candidate set the router itself resolved for the route (routing group, model_name, team,
and pattern alike), not by model-group name, so an alias, a provider-qualified spelling, a
team-public name, or a pattern route of the same group is not misread as a tier change.
An unknown origin (a removed deployment, or a forged/unauthenticated marker) is handled the
same as a cross-group one and its reasoning is stripped, so a real cross-group id and a
nonexistent id return the same response and cannot be used to enumerate deployment ids.
Unavailability within the origin's own group keeps the existing 429/503 fail-fast, so the
cooldown contract is unchanged
Resolves LIT-7195
Claude-Session: https://claude.ai/code/session_01KAumQbhzk6jdWWHFLA8Jar
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Replace the thread semaphore around HuggingFace encodes with an anyio CapacityLimiter
applied at every offloaded count site through offload_token_count, so waiting counts no
longer hold slots in the shared 40-thread pool and inline counts on the event loop never
block on the bound. Rename TOKEN_COUNTER_MAX_CONCURRENT_HF_ENCODES to
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS
The Advanced scoring editor now lists built-in and custom dimensions together. Editing any weight holds it and rescales the others proportionally so the vector totals 1.00, and Save stores those explicit values. The backend scores exactly what is stored, with no runtime normalization, so routers nobody edits keep their weights.
CustomDimension gains an opt-in scoring_mode. match_count scores 0, 0.5 or 1 by distinct matcher hits; the default stays binary. The tuning fingerprint omits a binary scoring_mode, so routers written before this change keep their recorded baseline and the upgrade does not consume the free heuristic-v1 tuning slot.
Cooldown entries rode the router-wide DualCache, which re-reads a key that is
missing from memory at most once every 10s. A deployment benched on one replica
therefore kept taking traffic on its siblings for up to 10 seconds, and the same
shared in-memory tier could evict a live cooldown once 200 unrelated router keys
crowded it out, which sent even the benching replica back to the dead deployment.
CooldownCache now owns a DualCache over the router's Redis with a 1s read
interval and an in-memory tier that only holds cooldown keys. Redis is attached
lazily because the router builds the cooldown cache before it wires Redis up.
Both Azure routes refuse it. A live call to the same deployment through
openai/deployments/gpt-6-astra/chat/completions on api-version 2025-04-01-preview
answers reasoning_effort max with a 400 unsupported_value naming none, low, medium,
high and xhigh as the values it takes, and xhigh returns 200, so azure/gpt-6-astra
and azure/us/gpt-6-astra now match the azure_ai row.
* fix(proxy): make the invalid-model 403 path cheap under a burst of rejections
Keep the wildcard pattern registry in specificity order at registration time
so route() no longer re-sorts every pattern per lookup, and reuse the
standardized failure payload across the async and threaded sync failure
handlers regardless of what a callback did to log_event_type. Rejections
are still logged and observable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(router): wrap the filtered pattern tuple the way ruff format wants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router,logging): assert registry order and callback awaits instead of patching a class
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): inject the pattern sorter so the lookup test observes that route() never sorts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
