Declare the live-verified reasoning_effort_levels on the Mistral cost-map
entries and round an undeclared request to the nearest declared level
(up to the weakest level at least as strong, down to the strongest when
the request exceeds the ceiling). Codex's default medium no longer 400s
on mistral-medium-latest, mistral-small-latest, or the vibe-cli family;
an entry that declares nothing keeps forwarding the value verbatim
Replace the per-deployment asyncio.Semaphore with MaxParallelRequestsLimit, which admits a call synchronously or raises the router's RateLimitError (429) right away. Nothing waits for a slot any more, so the max_parallel_requests_queue_size and default_max_parallel_requests_queue_size settings from the earlier commits are dropped along with their proxy validation, dashboard control and generated schema entries. The rpm/tpm derivation of the cap is unchanged. Every router endpoint family now enters the slot through one _deployment_slot context, and the provider coroutine is only created once the slot is held
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Budget is enforced once during auth, against the requested model group.
`_is_model_cost_zero` waives every budget check for a zero-cost group, and the
router then picks a fallback target afterwards, inside `run_async_fallback`,
where nothing re-checks budget. A free model with a paid fallback therefore
bills with no budget gate at all.
Add `fallback_budget_check`, the budget sibling of the existing
`fallback_access_check`: a predicate awaited per fallback target that skips
targets the caller cannot pay for. The primary attempt is untouched, so a
zero-cost model is never blocked by budget and only the paid fallback is
refused.
Counter reads pass `max_budget` so `get_current_spend` verifies against
authoritative recorded spend, matching the auth-time key and user checks; a
counter restored from an older snapshot reads as a hit rather than a clean
miss, so without it a stale-low value would keep admitting paid fallbacks.
A zero-cost fallback target is always allowed, and a team key does not inherit
the key owner's personal budget unless `apply_user_budget_to_team_keys` is set,
matching `_PROXY_MaxBudgetLimiter`.
Scope is key and user budgets. Team, team-member, end-user, org, global and
per-model budgets are not covered yet: those auth-path functions enforce rather
than report, so reusing them would fire threshold alerts and take spend
reservations for a target that is then skipped. Two limitations of that scope
are documented in the module docstring: the check reads the spend counter
rather than reserving against it, so concurrent fallbacks can cross a cap
together; and a request reaching the router without
`metadata["user_api_key_auth"]` is not restricted. Both are shared with
`fallback_model_access.py`.
Opt-in via `general_settings.enforce_fallback_budget`.
Relates to #41344
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The failure logger skips fallback hops (has_logged_async_failure is already set), so
model_call_details.end_time still belongs to the previous hop and predates this hop's
api_call_start_time. The fallback cooldown guard measured a negative elapsed time and
cooled down deployments for caller-set timeouts.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
client_side_timeout records that the caller configured a timeout, not that
the timeout fired. A 408 the provider returns before that deadline is a
deployment failure and must still count toward cooldown.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The sync Redis read now raises while the circuit breaker is open, and the
health state merge caught that as a generic error, skipping the local write
and logging an error on every background health check cycle. Read the shared
snapshot through a helper that treats the refused read as a miss so the merge
falls back to the pod-local copy the way a swallowed connection error already did
The cross-group branch of EncryptedContentAffinityCheck removed only the signature from
Anthropic-shaped thinking blocks, which left unsigned thinking blocks that Anthropic and
Bedrock reject (thinking.signature: Field required). Drop the whole block, the way #40280
drops undecryptable Responses input items, so the routed request carries the conversation
text with no reasoning item for those turns
The encrypted_content_affinity check only read the Anthropic history from request_kwargs["messages"], so a caller that passes it through the callback's messages argument alone skipped the pin. Read the argument first and fall back to the kwargs.
When the minting deployment is not a candidate of the routed group, the base already strips the Responses input's encrypted reasoning; do the same for the bridge-tagged thinking blocks in Anthropic messages so the routed deployment gets the readable thinking text instead of ciphertext it cannot decrypt.
The encrypted_content_affinity check only read the pin from the Responses
input, which /v1/messages builds after the router has picked a deployment,
so a model group spread across OpenAI orgs sent follow-up turns to the
wrong org and got invalid_encrypted_content back. The check now also
decodes the pin from bridge-tagged thinking and redacted_thinking blocks
in the Anthropic messages. The bridge also keeps a deployment's own
include list next to reasoning.encrypted_content instead of replacing it.
A Responses API follow-up that replays reasoning.encrypted_content is pinned to the
deployment that minted it. Behind an auto-router the pre-routing hook rebinds the model
to the tier it picked before the candidate pool is built, so a turn that classifies into a
different tier never finds the origin and the affinity check raised its fail-fast 503,
whose text claims a cooldown that does not exist
When the deployment that minted the reasoning is not a member of the model group this turn
is routed to, strip the encrypted reasoning (keeping any readable summary, string or block
form) and dispatch to the routed group. Membership is tested by deployment id against the
candidate set the router itself resolved for the route (routing group, model_name, team,
and pattern alike), not by model-group name, so an alias, a provider-qualified spelling, a
team-public name, or a pattern route of the same group is not misread as a tier change.
An unknown origin (a removed deployment, or a forged/unauthenticated marker) is handled the
same as a cross-group one and its reasoning is stripped, so a real cross-group id and a
nonexistent id return the same response and cannot be used to enumerate deployment ids.
Unavailability within the origin's own group keeps the existing 429/503 fail-fast, so the
cooldown contract is unchanged
Resolves LIT-7195
Claude-Session: https://claude.ai/code/session_01KAumQbhzk6jdWWHFLA8Jar
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The Advanced scoring editor now lists built-in and custom dimensions together. Editing any weight holds it and rescales the others proportionally so the vector totals 1.00, and Save stores those explicit values. The backend scores exactly what is stored, with no runtime normalization, so routers nobody edits keep their weights.
CustomDimension gains an opt-in scoring_mode. match_count scores 0, 0.5 or 1 by distinct matcher hits; the default stays binary. The tuning fingerprint omits a binary scoring_mode, so routers written before this change keep their recorded baseline and the upgrade does not consume the free heuristic-v1 tuning slot.
Cooldown entries rode the router-wide DualCache, which re-reads a key that is
missing from memory at most once every 10s. A deployment benched on one replica
therefore kept taking traffic on its siblings for up to 10 seconds, and the same
shared in-memory tier could evict a live cooldown once 200 unrelated router keys
crowded it out, which sent even the benching replica back to the dead deployment.
CooldownCache now owns a DualCache over the router's Redis with a 1s read
interval and an in-memory tier that only holds cooldown keys. Redis is attached
lazily because the router builds the cooldown cache before it wires Redis up.
Both Azure routes refuse it. A live call to the same deployment through
openai/deployments/gpt-6-astra/chat/completions on api-version 2025-04-01-preview
answers reasoning_effort max with a 400 unsupported_value naming none, low, medium,
high and xhigh as the values it takes, and xhigh returns 200, so azure/gpt-6-astra
and azure/us/gpt-6-astra now match the azure_ai row.
Foundry rejects reasoning_effort max on the gpt-6-astra deployment with a 400 that
names none, low, medium, high, and xhigh as the supported values, so the card no
longer lists max. The request path never gated max (only xhigh is opt-in), so this
only changes /model_group/info and router capability gating. The azure/ twin stays
as is because it was not verified on an Azure OpenAI host
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object]
annotations instead of bare dict, and the docstrings added to the new tests go away
since the test names already say what they cover. No behavior change
A gpt-6-astra deployment on a Foundry project reached through the
azure_ai route had no cost map entry of its own, so it resolved to the
OpenAI gpt-6-astra card: missing from the azure_ai/* wildcard listing,
flex and priority prices and /v1/batch it does not sell, and no none
reasoning effort. Add azure_ai/gpt-6-astra mirroring the
azure/gpt-6-astra Standard Global sheet the way azure_ai/gpt-5.5 mirrors
azure/gpt-5.5, and extend the cost, reasoning-effort, and wildcard
listing tests to the Foundry route.
