Without the auto_router feature in the signed enterprise license a proxy may hold
one complexity router with classifier_type heuristic_v2 across config.yaml and the
DB; with it the limit is lifted. The ceiling is derived once from LicenseCheck and
handed to the Router, which refuses the extra router at registration. config.yaml
over the limit refuses to start, and /model/new, /model/update and
PATCH /model/{id}/update refuse the write with a 403 before touching the DB.
Expiry follows the existing max_users/max_teams pattern: judged when the
license is verified, not on every call, and a verify that rejects the license
(expired or unreadable) leaves no signed payload behind. The rollback after a
failed upsert re-admits state that was already serving, so it is exempt from the
ceiling: an edit that fails, including one refused by a ceiling that has since
tightened, leaves the router serving its previous configuration.
A write that leaves a row on heuristic_v2 under a limited license runs in one
transaction that takes a Postgres advisory lock before counting the DB rows plus
this proxy's config.yaml routers, so concurrent writes on any pod cannot both
claim the sole slot and no surplus row is ever persisted.
Only the row insert runs under that lock: the team model bookkeeping, which
needs a second pool connection, runs after the transaction has committed.
PATCH /model/{id}/update follows the same order as create: the row is written
through the slot first and the team's model list is updated only afterwards, so
a refused write leaves the team as it was.
The slot transaction bypasses the repository's publish-on-write, so it
publishes the config change once after commit, as delete_team_models does.
OpenAI documents low, medium, high, xhigh, and max for gpt-6-astra, with no none level, so the entry stops advertising none and starts advertising max.
* fix(proxy): keep SpendLogs and callback session ids in sync when the request has none
Add general_settings.missing_session_id (generate | reject). In generate mode one id is
stamped into litellm_session_id, litellm_trace_id and metadata.session_id before callbacks
run, so LiteLLM_SpendLogs.session_id and the Langfuse session id match. In reject mode such
requests get a 400. Unset keeps the legacy behavior. MCP routes are not affected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate schema.d.ts and shorten mutable-ok comment for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark generated session ids so affinity consumers do not pin on them
Fireworks x-session-affinity, the router session_affinity pre-call check and the
complexity router session pin all read metadata.session_id as a caller-chosen
stable key. A missing_session_id: generate id is fresh per request, so it now
carries metadata.litellm_session_id_generated and those consumers skip it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): resolve fallbacks against the tier a pre-routing hook selected
A complexity or auto router picks a tier behind the router group name, but
fallback lookup kept using kwargs["model"], which is still the router name. The
tier's configured chain never ran, so a provider failure on its first hop went
straight back to the client with "No fallback model group found for original
model_group=smart-router".
The hook assigns the selected model to a local only, and fallback resolution runs
on an outer kwargs dict that **kwargs already copied, so writing it there is not
visible. Record the selection in the metadata bucket instead, which is a nested
dict shared by reference across those copies and is how the router already
carries values back up, then key fallback lookup off it when present.
Applies to the generic, context-window, content-policy and weighted-failover
lookups. Reporting keeps using the router name, since that is what the caller
asked for.
Fixes#38832
* fix(router): annotate the recorded-selection helper with a read-only mapping
record_pre_routing_selection only reads the request kwargs, writing into the
nested metadata bucket it finds there, so Mapping states what it actually needs
and clears the LIT001 mutable-annotation budget without a suppression.
* test(router): assert the no-kwargs path leaks nothing
The tolerated-None case called the helper without checking anything, which the
test-quality gate counts as a test with no assertion. Assert that a fresh mapping
still reads back empty, so the case proves the call is a no-op rather than only
that it does not raise.
* fix(router): stop declaring loop-assigned locals Final in the selection helpers
Both helpers annotated a loop-assigned local as Final, which reassigns a Final on
every iteration and cost three basedpyright errors. Read the buckets through a
generator instead, so the write path iterates a for-target and the read path
resolves in one shot with next(), which also matches the functional style the
type-discipline rules ask for.
* style(router): apply ruff format to the selection helpers
* fix(router): derive the pre-routing tier fresh on every fallback hop
The metadata buckets also carry whatever the caller sent, so an inbound
pre_routing_selected_model let a client pick which fallback chain its
request fell into. A fallback hop also inherited the previous hop's tier,
so the second hop keyed its own failure off the tier that already failed
and never ran its own chain.
Clear the key at the top of async_function_with_fallbacks. Every hop
re-enters there, so only the hook that routed that hop can set it.
* fix(router): drop the cast at the fallback-hop clear call site
* feat(router): fall back on anthropic safeguard refusals on /v1/messages
---------
Co-authored-by: Priyansh Nandwana <nandwana.priyansh103@gmail.com>
Resolving github_copilot/chatgpt names through get_llm_provider runs the
provider's OAuth device flow synchronously on the event loop. Adopt the
declared provider in PatternMatchRouter.get_pattern, which the auth
layer's zero-cost budget check walks on every request against wildcard
routers, and in /utils/supported_openai_params.
When a pre-call filter left no order-2 deployments, target_order matching
fell through to the remaining healthy list and reselected the failed
primary. Prompt-cache and deployment affinity also pinned that hop back
to order 1. Match the requested order strictly, skip those pins while
target_order is set, and keep target_order across retries of that hop.
* fix(anthropic): handle per-level reasoning_effort flags without supports_reasoning
When a model has only per-level flags (e.g. supports_minimal_reasoning_effort: true)
but no explicit supports_reasoning flag, treat it as implicitly reasoning-capable.
This fixes gpt-5-search-api which declares minimal support but was incorrectly
degraded to low/minimal floor due to missing explicit supports_reasoning flag.
Test: verify per-level flag enables resolution path even without supports_reasoning.
Note: This change indirectly causes 20 azure deployments to forward max/xhigh
instead of degrading to high when requested, as these models now correctly
resolve their supported efforts through declared capability flags. This is
intended behavior (avoiding unnecessary degradation) but silent; operators
seeing increased latency/cost should check reasoning effort changes in logs.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(anthropic): explicit supports_reasoning=False wins over per-level flags
Greptile P1: the implicit-True branch bypassed the operator's explicit
supports_reasoning: false escape hatch when per-level flags were present
or inherited through the bare-twin lookup. Return () first on explicit
False, then apply the per-level implication only when the flag is unset.
Also drops a test comment that restated the test name (P2).
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
A complexity-router setting placed beside complexity_router_config, or inside a
tier entry's litellm_params, is read by nobody: the router loads its settings only
from litellm_params.complexity_router_config. It does not stay inert. The
alias-marker forwarding and the per-tier param spread carry every unrecognized key
onto the outbound request, and all_litellm_params only knows the outer names, so
the key reaches the provider as an unknown body field and every call through that
model group fails with an error naming an internal config key.
Guard the whole set, derived from ComplexityRouterConfig.model_fields so a field
added later is covered, and scoped to complexity-router deployments because the
names only mean this there (embedding_model is a legitimate flat param on an
s3_vectors vector store). Scope is read from the same merged field view the naming
check is judged on, so a router named only by its default model is in scope and a
field added to the required-field table is covered without another edit. The write
endpoints reject with a 400 naming the keys and where they belong, config.yaml
refuses to start for the same reason max_agentic_loops does, and a tier entry is
judged by the config model itself.
An already-stored deployment keeps loading, so an upgrade cannot take a running
gateway down over a row that was written before the gate existed.
Kimi K3 accepts exactly low, high and max, defaults to max, and always thinks.
The map could not say that: medium and high have no supports_*_reasoning_effort
flag because every other reasoning model takes them, so the ten kimi-k3 entries
carried supports_reasoning alone and resolved to unknown. The dashboard then fell
back to a capability-blind level list that deliberately omits max, which is why a
kimi-k3 tier cannot be set to max thinking today.
Add reasoning_effort_levels, an array key in the shape the map already uses for
supported_endpoints and supported_modalities. Where present it is read first and
wins whole; every other entry keeps answering through the per-level flags,
unchanged. It is deliberately a different name from the computed
ModelGroupInfo.supported_reasoning_efforts, which stays derived from a group's
deployments and is never seeded from one deployment's model_info.
The levels are per entry rather than per model, because the deployments differ:
Moonshot, Together, Fireworks and Azure Foundry all forward the level unchanged
and get the model's own low/high/max, while Perplexity documents a six-value
enum it maps down internally and gets that. The /v1/messages degradation chain
consults the same declaration, so the level the map advertises is the level that
path forwards.
A non-ProxyException from the team, project or access-group lookup used to
escape the fallback loop and replace the provider's error. Treat it as a
denial and log it. Also drop the unrelated reformatting of test_router.py
and test_fallback_event_handlers.py so both diffs are additions only.
Router fallbacks configured in router_settings were attempted without
re-checking whether the calling key could use the fallback model, so a key
limited to one access group was served by any model listed as a fallback
for something it could call. Auth only validated the requested model and
fallbacks sent in the request body.
Add a fallback_access_check predicate to Router, consulted before every
cross-model-group fallback attempt; rejected targets are skipped and the
primary's own error is raised when none remain. The proxy injects a check
that runs the same key, team and project model access checks the requested
model goes through.
* feat(health): opt-in model-group allowlist for background health checks and health-check routing
* fix(health): merge shared health states per writer scope instead of replacing
* refactor(health): drop restating comment and parameterize test scope annotations
* chore: remove stray generated prisma migration file
* fix(health): merge health states against the Redis snapshot, not the pod-local copy
* fix(health): fall back to the pod-local snapshot when the Redis read returns nothing
An auto_router deployment is a marker, not something a probe can contact, so
`_run_model_health_check` returns `{}` for it and it lands healthy whatever is
behind it. This derives its verdict from the models it actually resolves.
Rules and owners:
- `strategy_router_dependencies` is the single answer to "what does this router
call": tier, default, classifier and embedding names per router kind, aligned
with what init and the request path actually use.
- `_health_check_eligible` is the single probe-eligibility gate, applied to the
requested set and to the pool a router's dependencies are drawn from alike, so
an opted-out deployment cannot re-enter through a router that depends on it.
- `_resolved_deployment_ids` resolves names through `get_model_list`, the same
composition of alias, routing-group and wildcard channels a request uses.
- A dependency reds its router only when *every* deployment behind the name is
known unhealthy. A replica this run never judged, hidden from the caller or
opted out of health checks, can still serve what the dead one drops, so
partial evidence leaves the verdict green. Absent information never reds.
- Verdicts settle over rounds, because a marker never fails a probe of its own
and a parent whose tier is a red router must inherit that fault. Both sweeps
are bounded loops, so a router cycle terminates green.
- Dependency probes are added only on the targeted `/health?model_id=` path the
dashboard uses per deployment, and are dropped from the response.
Resolves LIT-6073
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A reasoning model whose map entry names no effort flag now resolves to None, so
the API omits the field and the dashboard keeps its six-level fallback, and a
deployment counts as catalog-known only when the map supplied its mode, so an
operator writing model_info on an off-map deployment no longer empties the
levels its mapped siblings agree on.
Also drops the ultra level nothing asked for, forwards every level the public
literal names across the chat to Responses bridge, and removes the unreachable
supported_reasoning_efforts validator.
get_model_info answers supports_reasoning None both for a model absent from
the map, which the router registers under a synthesized entry, and for a
mapped model that simply is not a reasoning model. Reading both as "adds no
levels" let one custom deployment wipe every level its mapped siblings agreed
on.
The synthesized entry carries no mode, which every real map entry for a
routable model does, so an unset flag with no mode now resolves to unknown and
never narrows its group. A group that genuinely shares no level still
advertises none, and the dashboard drops the effort control for it instead of
offering levels routing would refuse.
AzureOpenAIGPT5Config raises UnsupportedParamsError on reasoning_effort='none'
unless the model map flags it, while OpenAI never refuses the level, so a single
opt-out polarity advertised none for 61 azure deployments that reject it. Defer
to _supports_factory when the azure flag is absent so the advertised list and the
request gate agree on every azure gpt-5 model in the map.
Also wrap the widened reasoning_effort Literal in main.py, which ruff format
flagged over the line limit.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
A file uploaded through Router.acreate_file lands in the account of the
deployment that stored it, so a cross-group fallback silently stores the
file with the wrong provider and every later batch or fine-tuning call
against the returned id permanently fails. Extend the provider-scoped
fallback pin that already covers input_file_id and training_file to file
creation, so the original provider error surfaces instead.
* feat(router): make routing groups callable as virtual models and list them in /v1/models
* fix(router): traffic-scoped cooldown exemption, live model_names on delete, group-info cache invalidation
* fix(router): share one recognized-model predicate across proxy gates, resolve aliases in group cooldown, read metadata via the dual-bucket owner
* fix(router): close the gate and cache families for callable groups, strip member access_groups from group rows, prove cooldown wiring end to end
* refactor(router): cache materialized group rows under the model-group cache owner and drop the redundant wiring test
* fix(router): warn-and-shadow on group name collisions, name-level test coverage for group helpers, faithful router doubles in a2a and cursor tests
* test(router): pin group cooldown metadata across the retry path
A deployment that carries one provider's credentials while resolving to
another is silently broken: litellm ignores the credentials and sends the
request to the resolved provider. The common shape is a Bedrock model
group where one entry lost its route prefix, so `model: claude-sonnet-5`
with aws_region_name set resolves to the first-party Anthropic API and
returns "x-api-key header is required". Because the router load balances
across the group, only the fraction of requests routed to that entry
fails, which reads as an intermittent provider outage rather than a
config error, and nothing at startup says otherwise.
Warn at deployment registration when provider-scoped credential params
(aws_*, vertex_*) sit on a model that resolves elsewhere, naming the
params, the resolved provider, and the likely missing prefix. Warn only:
an operator may be overriding a route deliberately, so this must not
block startup. Deployments litellm cannot classify are left alone.
Resolves LIT-5391
* fix(router): cool down failed fallback deployments and correct cooldown TTL after Redis backfill
A deployment that failed partway through a fallback chain (any attempt after
the first) was silently exempt from cooldown, because the has_logged_async_failure
dedup flag blocks the normal failure callback for every attempt past the first.
_trigger_cooldown_for_failed_deployment now explicitly evaluates cooldown for
that deployment when the dedup flag is set, using the same deployment-config >
response-header > router-default precedence as the primary failure path, and
skips advisor-orchestration failures. Deployment-ID resolution prefers the
exception's stamped failed_deployment_id, now also set from the generic-API-call
fallback path (rerank, embeddings, /v1/messages, etc.), falling back to metadata
inspection for call paths that don't stamp it yet.
CooldownCache also recomputes the remaining TTL when DualCache promotes a Redis
entry into the in-memory layer: before this, a cooldown entry restored from Redis
kept the in-memory layer's default 600s TTL regardless of the deployment's real
cooldown_time, so a deployment could stay excluded from routing for up to 10
minutes after a much shorter cooldown had already expired.
* fix(router): address Greptile review on the fallback-cooldown trigger
Two P1 findings on PR #35104:
- _trigger_cooldown_for_failed_deployment never incremented the deployment's
per-minute failure counter before evaluating cooldown, so a fallback
deployment's repeated retryable failures never accumulated toward the
default percent-fail-rate threshold that _should_cooldown_deployment checks.
- The metadata-bucket fallback (checking "metadata" before "litellm_metadata"
for a deployment_model_name marker) could be fooled by a caller with
permission to set metadata, since neither bucket's authorship can be
determined without knowing the call's function_name. Removed it entirely;
cooldown now requires the server-stamped failed_deployment_id, matching
what the primary chat-completions path and the generic-API-call path
(rerank, embeddings, /v1/messages, etc.) already set unconditionally.
* fix(router): freeze the litellm_params fallback mapping to satisfy the type-discipline gate
* fix(router): defer f-string interpolation in fallback-cooldown debug logs
* fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget
* fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks
* fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one
* fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout
* fix(router): stamp dynamic client-side-credential id in completion fallback paths too
The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential)
deployment id on exceptions, but the regular _completion/_acompletion exception handlers still
stamped the static shared deployment's id. A tenant using invalid forwarded credentials could
generate repeated failures attributed to, and eventually cooling down, the shared deployment
other tenants rely on. Extracted the stamping logic into one shared helper used by all three
call sites (generic API, sync completion, async completion) so the fix and future changes to it
stay in one place.
* test(router): add direct-reference unit tests for the new stamping helper
router_code_coverage.py's coverage gate flags _stamp_failed_deployment_id_with_effective_model_info
as untested because it only sees the function invoked indirectly through _completion/_acompletion's
exception handlers. Added two tests that call it directly, covering both the dynamic-id-present and
static-fallback branches.
* test(router): cover the timeout stamping branch and async active-cooldown append
_acompletion's litellm.Timeout handler and async_get_active_cooldowns' happy
path both lacked direct coverage despite their sibling branches (the generic
Exception handler, the sync get_active_cooldowns) being tested.
* test(router): remove duplicate cooldown-trigger and fallback-helper tests
#34416 landed its own TestTriggerCooldownForFailedDeployment/
TestRunAsyncFallbackTriggersCooldown classes and
test_ageneric_api_call_with_fallbacks_helper_stamps_failed_deployment_id
covering the exact same scenarios as this branch's earlier flat-function
tests, once its version of fallback_event_handlers.py was taken as-is
during the last merge. Dropping the redundant copies.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* feat(router): add per-deployment allowed_fails_policy and cooldown_time override support
Three bugs fixed in the router cooldown system: (1) deployment-level allowed_fails and
allowed_fails_policy in model_info now take precedence over router-level settings in
_should_cooldown_deployment; (2) failed fallback deployments now get evaluated for
cooldown via _trigger_cooldown_for_failed_deployment, bypassing the Logging dedup gate;
(3) DualCache promotes Redis cooldown entries using default 600s TTL instead of true
remaining cooldown time -- _corrected_active_cooldown now evicts expired entries and
corrects stale in-memory TTLs on backfill. Adds ServiceUnavailableError, BadGatewayError,
and NotFoundError fields to AllowedFailsPolicy and cooldown_time to LiteLLMParamsTypedDict.
* fix(router): gate fallback cooldown trigger on has_logged_async_failure; use only litellm_metadata for deployment ID
* fix(router): use X | Y union syntax to fix UP007 strict lint gate
* test(router_utils): add coverage for _trigger_cooldown_for_failed_deployment and has_logged_async_failure gate
* test(router_utils): cover deployment cooldown override and exception swallow paths
* fix(router): add InternalServerError/ServiceUnavailableError/BadGatewayError/NotFoundError to router-level get_allowed_fails_from_policy
* fix(router): format router.py and add router-level policy tests
* test(router): add CI-visible coverage for per-deployment cooldown policy
Tests for `_get_deployment_cooldown_policy`, `_resolve_allowed_fails_from_policy`,
and `_should_cooldown_based_on_deployment_policy` (cooldown_handlers.py), the
`_corrected_active_cooldown` branches in CooldownCache, and the four new
exception-type branches in `Router.get_allowed_fails_from_policy` (router.py) --
all in `tests/test_litellm/` which the enterprise-routing CI job runs.
* fix(router): use is not None guard for cooldown_time_override in should_cooldown_based_on_allowed_fails_policy
A cooldown_time_override of 0 was previously treated as falsy and silently
fell through to the router-level cooldown_time value. Switched to an explicit
is not None check so that zero is honored as a valid override.
Added a regression test covering the zero case.
* fix(router): honor has_logged_async_failure and metadata for fallback cooldown; support both model_info and litellm_params locations
Manual verification against a live proxy surfaced that the fallback-cooldown-gap
trigger never actually fired: the has_logged_async_failure check read a plain
attribute that Logging never sets (the real flag lives in model_call_details),
and the deployment_id lookup only trusted litellm_metadata, which regular chat
completions never populate (only batch/thread/file endpoints do). Router
overwrites model_info on whichever key is present before every attempt, so
metadata is equally authoritative there, not caller-controlled as previously
assumed. Also let allowed_fails/allowed_fails_policy/cooldown_time be set under
either model_info or litellm_params, each preferring its own canonical location.
* fix(router): fix ContentPolicyViolationError policy shadowing and partial-policy zero-threshold
Two bugs from Greptile review on PR #34416:
- ContentPolicyViolationError subclasses BadRequestError, so listing
BadRequestError first in _EXCEPTION_POLICY_FIELDS made the isinstance
check always match BadRequestError for content-policy errors, using the
wrong allowed_fails threshold. Reordered so the subclass is checked first.
- A deployment with a partial allowed_fails_policy and no deployment-wide
allowed_fails forced allowed_fails_override=0 for any exception type its
policy didn't cover, cooling the deployment down on the first unrelated
failure. Now defers to router-level behavior for uncovered exception
types instead of forcing an immediate cooldown.
* fix(router): only trust a metadata/litellm_metadata bucket the router itself wrote deployment info into
veria-ai flagged that preferring litellm_metadata whenever present could pick up a
caller-supplied litellm_metadata.model_info.id (preserved via allow_client_pricing_override)
instead of the metadata bucket the router actually populated for a regular completion's
fallback attempt, naming an arbitrary "victim" deployment for cooldown.
Router._update_kwargs_with_deployment() always writes model_info and
deployment_model_name into the same bucket together. Only trust a bucket that
carries deployment_model_name alongside model_info, since that marker is only
ever set by the router itself, not by request-body metadata.
* test(router): add regression coverage for ContentPolicyViolationError policy shadowing
The subclass-ordering fix in commit 38fe4e4490 had no regression test.
Verified the new test fails on the pre-fix ordering (asserts 2, got 10)
before restoring the fix, and confirmed the same behavior through the full
_should_cooldown_deployment call path against a real Router instance.
* fix(router): let explicit allowed_fails_policy entries override the generic 4XX cooldown exclusion
_is_cooldown_required skips cooldown evaluation for any 4XX status outside
{429, 401, 408, 404} by default, since a generic client error is usually not
the deployment's fault. BadRequestError and ContentPolicyViolationError both
carry status 400, so their AllowedFailsPolicy fields (BadRequestErrorAllowedFails,
ContentPolicyViolationErrorAllowedFails, both router-level pre-existing and the
new deployment-level ones) were silently unreachable: an operator could set
them to any value with no effect, since _is_cooldown_required blocked cooldown
evaluation before that policy was ever consulted.
_should_run_cooldown_logic now also checks whether an explicit allowed_fails_policy
entry (deployment-level or router-level) covers the exception's type, and if so,
proceeds with cooldown evaluation regardless of the generic status-code exclusion.
The exclusion remains the default for exception types with no explicit policy.
Verified live against a mock-triggered ContentPolicyViolationError (config-level
mock_response, azure/gpt-4.1-mini deployment) with BadRequestErrorAllowedFails=100
and ContentPolicyViolationErrorAllowedFails=0 on the same deployment: it now cools
down after exactly one ContentPolicyViolationError instead of never cooling down.
* fix(router): use the router-stamped failed_deployment_id for fallback cooldown targeting
Greptile flagged a real gap in the metadata-bucket-based deployment lookup:
for a generic-API-call fallback, the router writes the current attempt into
litellm_metadata, but a stale "metadata" bucket carrying the same
deployment_model_name marker (from an earlier point) would be picked first,
cooling the wrong deployment.
Router already has a more robust, pre-existing mechanism for this exact
problem: _set_failed_deployment_id_on_exception stamps the failing
deployment's id directly onto the exception at the point of failure,
immune to metadata-bucket ambiguity since a caller can't influence it and
it doesn't depend on which bucket the current call type happens to use.
It just wasn't called from _ageneric_api_call_with_fallbacks_helper's
except block, unlike _completion/_acompletion.
Added the missing call there (matching the existing pattern exactly), and
changed _trigger_cooldown_for_failed_deployment to prefer
exception.failed_deployment_id when present, falling back to metadata-bucket
inspection only for call paths that don't stamp it yet.
Verified live: the standard fallback-cooldown-gap scenario (two bad-key
deployments in a fallback chain) still correctly cools down both the
originally-called and fallback deployment.
* fix(router): address human review on per-deployment cooldown overrides
Scope allowed_fails_policy override to deployment-level only (a router-level
policy predates this feature and must keep its existing behavior), exempt
advisor-orchestration failures from the fallback cooldown trigger, keep the
single-deployment model group protection intact against a generic
deployment-level allowed_fails, make cooldown_time precedence consistent
across resolution paths, fix a falsy-zero swallowing bug in the router-level
allowed_fails fallback, and make allowed_fails_policy resolution fall through
to the next matching exception type instead of stopping at the first unset
field.
Also restrict allowed_fails/allowed_fails_policy/cooldown_time to model_info:
litellm_params gets copied into the actual provider request, so a router-only
setting placed there would leak into that request.
* test(router): update test_cooldown_handlers.py for the deployment-policy signature change
Surfaced by the rebase: this mirrored test file (tests/test_litellm/ mirrors
litellm/) predates the router_unit_tests/ coverage added earlier in this PR and
was still calling _should_cooldown_based_on_deployment_policy with its old
4-argument signature and asserting the now-removed litellm_params cooldown_time
location.
* test(router): update test_fallback_event_handlers.py for model_info-only cooldown_time
Another mirrored test file surfaced by the rebase that still asserted the
now-removed litellm_params.cooldown_time location.
* fix(router): match cooldown-duration precedence in the fallback path to the primary path
_trigger_cooldown_for_failed_deployment only checked deployment config before
falling back to the router default, skipping the response Retry-After header
step that Router.deployment_callback_on_failure applies on the primary path.
* fix(router): restore litellm_params.cooldown_time as a pre-existing fallback
cooldown_time already had litellm_params support on Router.deployment_callback_on_failure
before this PR; the earlier model_info-only restriction (aimed at the leak concern
for the genuinely new allowed_fails/allowed_fails_policy fields) incorrectly dropped
that pre-existing capability too. model_info still takes priority when both are set.
* fix(router): keep the fallback-cooldown trigger in sync with #35104's review fixes
Applies the same two fixes landed on the split-out PR #35104 (which #34416
still duplicates until it's rebased onto the merged base): increment the
deployment's per-minute failure counter before evaluating cooldown, and
require the server-stamped failed_deployment_id instead of trusting a
metadata bucket, since neither "metadata" nor "litellm_metadata" can be told
apart from a caller-supplied one without knowing the call's function_name.
* fix(router): freeze the model_info fallback mapping to satisfy the type-discipline gate
* fix(router): defer f-string interpolation in fallback-cooldown debug logs
* fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget
* fix(router): suppress reportPrivateUsage for cross-module cooldown helpers
* fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks
* fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one
* fix(router): keep up with upstream typing modernization and Final-annotation ratchet
* fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout
* fix(router): stamp dynamic client-side-credential id in completion fallback paths too
The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential)
deployment id on exceptions, but the regular _completion/_acompletion exception handlers still
stamped the static shared deployment's id. A tenant using invalid forwarded credentials could
generate repeated failures attributed to, and eventually cooling down, the shared deployment
other tenants rely on. Extracted the stamping logic into one shared helper used by all three
call sites (generic API, sync completion, async completion) so the fix and future changes to it
stay in one place.
* fix(proxy): recognize body-supplied timeout/request_timeout/stream_timeout as caller-controlled
client_side_timeout was only set when the caller used the x-litellm-timeout header, but
Router._get_timeout also resolves the effective timeout from kwargs["timeout"],
kwargs["request_timeout"], and kwargs["stream_timeout"], all settable directly in the
request body (and x-litellm-stream-timeout wasn't marked either). A caller could set any
of those to a near-zero value, force a 408 on every deployment in a fallback chain, and
cool down deployments other tenants rely on without the guard in
_trigger_cooldown_for_failed_deployment recognizing it as caller-controlled. Also strip
any client-forged client_side_timeout from the request body so the marker is always
server-computed.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
A batch or fine-tuning job is created from a file the caller already uploaded,
and that file only exists under the credentials of the deployment that stored
it. When the router fell back to a different model group it handed that file id
to a provider that has never seen it, so the caller got the second provider's
complaint about the file id instead of the error that explains what was actually
wrong with their request.
run_async_fallback now skips fallback targets outside the original model group
whenever the request carries input_file_id or training_file. Order-based
fallbacks stay inside the group, so retrying across deployments still works.
The same handler also crashed with "'NoneType' object has no attribute 'update'"
whenever a fallback fired on a request with metadata set to None, which
/v1/batches always does when the caller sends no metadata, turning the provider's
400 into a 500. Record the model group with a merge instead of setdefault, and
write it to litellm_metadata on the endpoints that use it so the router's
bookkeeping no longer lands in the metadata stored on the provider's batch.
* feat(complexity_router): let operators rename the four complexity tiers
Adds an optional tier_labels map to complexity_router_config so a deployment can
put its own vocabulary on the four tiers, e.g. Cheap / Standard / Premium / Deep,
instead of reading SIMPLE / MEDIUM / COMPLEX / REASONING in its dashboard, its
spend logs, and the rubric the LLM classifier reasons with.
Labels are display-only. Every config key stays canonical, so tiers,
keyword_tier_rules[].tier, and tier_boundaries are written exactly as they are
without labels, and partial maps are fine with unlisted tiers keeping their
default name. A validator rejects blank labels, two tiers sharing a label, and a
label that is another tier's canonical name, since any of those would make a log
row or a rubric line ambiguous. That validator runs on the /model/new and
/model/update write path already, so an ambiguous config gets a 400 rather than
being stored for the router to refuse later.
Under the default heuristic scorer the names are cosmetic: the scorer maps a
weighted score to a rung and never reads a tier name, verified by running the
eval corpus with and without a rename and getting identical tier and identical
score on all 29 cases. Under classifier_type: llm the labels are the names in the
rubric and the values the classifier must return, so the response format's enum
is now built from the configured labels and a reply is resolved back to its tier
against labels first, then canonical names, case-insensitively. An unresolvable
reply degrades to the heuristic on the existing fallback path. A test pins the
generated schema for an unrenamed deployment as equal to the shipped
TierClassification schema, so the wire shape can't drift.
Spend logs keep routing_decision.tier canonical so rows from before and after a
rename stay comparable, and gain routing_decision.tier_label on the tiers that
were renamed.
* refactor(complexity_router): drop added comments and the Counter construction
Review feedback: the repository guide bans new comments, so the explanatory
comments and the appended docstring paragraphs this branch added come back out.
One-line docstrings stay in complexity_router.py, matching that file's own
convention.
The duplicate-label check no longer builds a Counter, which the mutable-collection
budget counts, and the error text drops its list() reprs for joined strings. The
labels are stripped in tier_label() now rather than by rewriting the field in the
validator, so the stored config keeps exactly what the operator wrote.
schema.d.ts is regenerated: ComplexityRouterConfig is exposed in the OpenAPI spec,
so tier_labels surfaces there.
* fix(ui): carry tier_labels through the auto-router preset prefill
buildPresetPrefill maps every payload key onto form state, but the tier_labels
key added by this branch had no line, so a preset shipping labels would apply
its tiers and silently drop its names.
"Add keyword rule" seeds a row with no keywords, and the only check that a
rule carried one lived inside getSemanticConfigError, which returns early
when semantic keyword matching is off. Off is the default, so an unfilled
row fell through to serializeKeywordTierRules and was discarded on the way
to the payload; the create reported success and the rule was gone.
The row now reports the gap itself and the submit is withheld while one is
outstanding, on the create form and the edit modal alike, both reading
emptyKeywordTierRuleIndexes so the row named and the row marked cannot
differ. Enter commits a typed keyword: the dropdown is kept closed, which
left antd nothing for Enter to select, and submitting was what used to
supply the blur that saved the word.
The backend already refused such a rule, but only when the router built the
deployment, so a caller that sent one anyway got the row written, dropped on
reload, and a 500. The management write paths now parse the incoming
complexity_router_config with the router's own ComplexityRouterConfig, judged
on the config alone so a patch that writes one without naming a model is
covered too, and reject it with a 400 having persisted nothing.
Key and team `router_settings.model_group_alias` was accepted, persisted and
echoed back by `/key/info`, but never applied at request time, so the request
ran on the group the caller asked for. `route_request` forwards only the
settings the Router accepts as per-request kwargs, and `model_group_alias` is
not one of them: the Router resolves aliases from its own instance attribute,
which holds the global config map and is shared across requests.
Resolve the alias in the proxy instead, alongside the existing model-alias
rewrites and ahead of the pre-call hooks, so per-model limits and guardrails
key off the group that actually serves the request. Authorize the alias target
before the rewrite; model access was checked against the requested group, so a
key whose alias points at a group it cannot call gets the usual 403 rather than
being quietly served it.
Resolves LIT-4879
An auto-router deployment's litellm_params.model (auto_router/...) is the
discriminator the router loads it by, but the model management endpoints
accepted any client-supplied value verbatim; a doubled or stripped prefix
made router init fail on the next load and ignore_invalid_deployments
silently dropped the deployment. Validate writes that supply
litellm_params.model at all three endpoints against the merged params and
reject incoherent values with an actionable 400. Classification is
extracted to router_utils/auto_router_model_naming.py so the Router
predicates and the validation share one source
The read gate cannot cause a wrong pin. A deployment is only pinned when the cache
already holds an entry for the prefix, and async_log_success_event writes entries
against the deployment's real model rather than the group alias, so a model that
will not cache a prefix never records one and there is nothing to pin it to
That makes this gate purely a cheap short-circuit deciding whether the cache lookup
is worth doing, so the threshold must be the lowest minimum in the group. Taking the
highest skipped the lookup for a prefix a lower-minimum member had genuinely cached,
losing a hit it earned, and protected against nothing. It also broke the Fable 5
direction this ticket is meant to fix: its real minimum is 512, so a group gate stuck
at a higher value would skip the lookup for a prefix Fable 5 had actually cached
get_model_info is lru_cached, so swapping litellm.model_cost is not enough on its
own. An earlier test that resolved these models against the remote map, which does
not carry prompt_cache_min_tokens yet, leaves cached entries without it, and the
stale hit resolves to the default. The assertions would then pass for the wrong
reason or fail depending on execution order
Clear on teardown as well, so entries these tests warm against the local map do not
leak into later tests, matching the fixture already used in test_utils.py
Also pin that a wildcard route resolves the underlying model's minimum. That works
only because pattern_match_deployments substitutes the real model name into
litellm_params before the deployment reaches the check; without the assertion that
claim is unpinned and the threshold would silently fall back to the default
MINIMUM_PROMPT_CACHE_TOKEN_COUNT was a flat 1024 described as "minimum number of
tokens to cache a prompt by Anthropic". Anthropic's minimum cacheable prefix is
per-model and ranges from 512 to 4096, and it can differ per platform for the same
model, so one constant is wrong in both directions
is_prompt_caching_valid_prompt gates PromptCachingDeploymentCheck, which is what
optional_pre_call_checks: ["prompt_caching"] turns on. When it believes a prompt is
cacheable, async_filter_deployments pins routing to whichever deployment previously
served that prefix. For a prompt between 1024 and 4096 tokens on Opus 4.6, Opus 4.5
or Haiku 4.5, litellm judged it cacheable and constrained routing while the provider
never cached it, so the pin cost load balancing for nothing. In the other direction
Fable 5 caches from 512 tokens, so a 512 to 1024 token prefix was refused a pin it
had earned
The minimum now resolves from prompt_cache_min_tokens in the model cost map, which
keeps it current with new models and lets the Bedrock override for Fable 5 fall out
of the existing per-entry keys with no special casing. MINIMUM_PROMPT_CACHE_TOKEN_COUNT
stays as a global escape hatch when explicitly set, and as the fallback for models the
cost map has no entry for
async_filter_deployments only ever receives the model group alias, never a model name,
so it resolves the threshold from healthy_deployments instead. A group may mix models
with different minimums, so it takes the max: a prompt is only treated as cacheable when
it clears every member's minimum, because an unnecessary pin is the defect being fixed
while a missed pin only forfeits an optimization
Gemini context caching shares this gate and has the same defect; its entries are left
unset so they keep today's behavior, tracked separately in LIT-4525
* feat(router): add separate ITPM/OTPM deployment rate limits
Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(router): keep ITPM/OTPM diff minimal in router.py
Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): make ITPM/OTPM limits separate and atomic
Address Greptile review on separate ITPM/OTPM deployment rate limits.
- OTPM is now reserved atomically pre-call with rollback, matching the ITPM
path, so concurrent requests can no longer overshoot the configured output
limit before reconciliation
- ITPM counts input tokens only; it no longer accumulates completion tokens,
so the input-token limit and x-ratelimit-limit-input-tokens header describe
input usage as their names imply
- _read_reservation_from_kwargs only falls back to litellm_params.metadata when
the top-level metadata channel is absent, so production requests carrying a
litellm_params.metadata dict still reconcile and refund their reservation
Adds regression tests for OTPM atomicity under concurrency, input-only ITPM
enforcement, and reservation lookup when litellm_params.metadata is present.
* fix(router): subtract input tokens only from remaining-input-tokens header
The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total
tokens (input + output) instead of input tokens only, so clients saw remaining
input quota understated by the completion token count on every response. Now
consistent with the input-only ITPM counter.
* fix(router): make itpm/otpm vs tpm/rpm precedence explicit
When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path
takes over and the tpm/rpm limits are not enforced. Log a warning the first
time such a conflicting deployment is seen so the supersession is not silent,
and document the mutual exclusivity.
Post-call reconciliation now only trues up a counter that was actually
reserved against, so the itpm/otpm keys are no longer incremented for
deployments that never configured that limit.
* fix(router): track actual io-token usage on the reservation-minute key
Post-call reconciliation now keys off the exact cache key stashed at pre-call
time rather than one recomputed from the response-time minute. This fixes two
issues: a request whose pre-call estimate was 0 now still writes its actual
billable input to the ITPM counter (previously it was skipped, leaving the
limit unenforceable for that request), and a call that finishes in a later
minute reconciles against the minute it reserved against instead of pushing a
negative delta into the next minute. Counters are only touched when their
limit is configured.
* fix(router): run io-token reconciliation before the model_id guard
async_log_success_event gated IO reconciliation behind the model_id guard that
only the TPM tracking path needs. Since reconciliation works entirely from the
cache keys stashed in kwargs, a success event whose standard_logging_object
lacks model_id would skip reconciliation and leave the reservation on the
counter until the TTL expired, wasting quota. Route the IO path first.
* fix(router): don't replay in-flight delta for itpm/otpm headers
For ITPM/OTPM model groups the counter is incremented at reservation time
(pre-call), so the remaining values returned by get_remaining_model_group_usage
already account for the current request. Replaying the in-flight delta on top
double-counted it and understated x-ratelimit-remaining-input/output-tokens by
up to max_tokens on every response. Skip the delta for io-token groups; the
legacy TPM/RPM replay path is unchanged.
* fix(router): clear io-token reservation after reconcile/refund
async_io_token_refund_failure and async_io_token_reconcile_success now clear
the stashed reservation keys from the request metadata once done. Otherwise, on
a model group mixing IO-limited and non-IO deployments, a failed IO call that
retries on a non-IO fallback left the stale sentinel in the shared request
metadata; the fallback's success handler would divert into IO reconciliation
against the already-refunded key, driving the ITPM counter negative and
skipping the non-IO deployment's TPM tracking.
* fix(router): tidy reservation channel lookup and header guard
Consolidate the reservation channel lookup into a single ordered helper shared
by read and clear, so top-level metadata always wins over litellm_params
metadata without the tangled per-iteration fallback.
Also stop gating the router rate-limit header block on the presence of
x-ratelimit-remaining-input/output-tokens. That block only emits those headers
for ITPM/OTPM groups; for a non-IO group backed by a provider that natively
returns input/output token headers, the extra conditions suppressed the
router's own remaining-tokens/requests headers.
* fix(router): strip client-supplied io-token reservation keys
The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key,
and the otpm equivalents) are server-only, but metadata is caller-controlled on
proxy requests. An authenticated caller could forge these fields with an
arbitrary cache key so the post-call reconcile/refund path would decrement any
deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip
the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs,
which runs before the router stashes its own reservation, so only a genuine
server-side reservation is ever read post-call.
* fix(router): track TPM routing load for io-limited deployments
deployment_callback_on_success early-returned for any deployment with itpm/otpm
set, so its total-token usage never landed in the router's TPM routing counter.
TPM-aware routing strategies then saw 0 load for IO deployments and over-routed
to them in mixed model groups. Only skip tracking when neither tpm/rpm nor
itpm/otpm are configured; itpm/otpm enforcement still runs separately in
ModelRateLimitingCheck, so the routing counter and the enforcement counters
stay independent.
* fix(router): expose standard tpm/rpm headers for io-limited groups
get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group
that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests;
clients and prometheus gauges reading those saw no data. Build both header sets
instead of returning early.
Also simplify the in-flight header replay: only the tpm/rpm counters are
incremented post-response, so the delta now adjusts just those. The itpm/otpm
counters are incremented at reservation time (pre-call), so the input/output
token headers already reflect the request and are left untouched - which
removes the need for the separate io-group special case.
* fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance
Two follow-ups from review. The pre-call OTPM reservation only rolled back the
ITPM reservation on a RateLimitError, so a transient cache error while reserving
OTPM left the ITPM counter inflated until the TTL expired; catch any exception,
release the ITPM reservation, then re-raise.
Replace the module-level lru_cache warn-once (caching a logging side effect,
which never re-warns in a long-lived process) with an instance-scoped set of
already-warned deployment ids on ModelRateLimitingCheck.
* fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup
Clear the reservation in a finally block so a mid-reconciliation cache error
still removes the stash and a duplicate success event can't re-process it.
Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a
deployment with no id no longer collapses every id-less deployment onto the
str(None) key (which would suppress all but the first warning).
* fix(router): skip io reservation when deployment can't be keyed
_get_cache_keys returned a shared 'global_router:None:None:...' key when a
deployment was missing model_info.id or litellm_params.model, so misconfigured
deployments could share one rate-limit bucket. Return None in that case and
skip io reservation for the request.
* fix(router): honor explicit max_tokens=0 in io reservation
_resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit
max_tokens=0 fell through to the model default. Only fall back to
max_completion_tokens when max_tokens is absent.
* fix(ci): satisfy lint budget, router coverage, and dashboard schema sync
- Modernize the new itpm/otpm module's type hints to PEP 585 lowercase
generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006
violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match.
- Replace three try/except Exception blocks that must stay broad by design
(token_counter and litellm.get_model_info raise untyped exceptions, and an
io-token refund failure must never break the logging pipeline) with
contextlib.suppress(Exception), matching the codebase's existing resolution
for this exact BLE001 pattern.
- Add direct unit tests for get_model_group_io_token_usage (multi-deployment
aggregation and the empty-model-list case) in test_router_helper_utils.py,
satisfying the router function-coverage check.
- Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on
GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types.
* fix: enforce io token rate limits consistently
* fix: honor zero max tokens in otpm reservation
* fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base
Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10
floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span
alias.
The previously committed ruff-strict-budget.json ratcheted UP006 down from a
stale base; litellm_internal_staging has since tightened that same ceiling
further on its own. Reset the file to the current base's committed values and
re-ratchet from there so the budget only ever moves down relative to the
actual merge-base, never against a stale snapshot.
* fix(router): attach ITPM/OTPM headers on dict responses and harden reservation
Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM
estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit
headers through /v1/messages dict responses via _hidden_params.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses
Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so
set_response_headers can attach rate-limit headers to streaming Anthropic
messages responses that lack a _hidden_params slot.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: ruff format add_retry_fallback_headers.py
Fix CI ruff format check failure on get_hidden_params_dict call site.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(router): extract set_response_headers helpers to fix C901 budget
Move header-attachment logic into add_retry_fallback_headers helpers so
set_response_headers stays under the strict complexity ceiling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: keep IO token reservation when response usage is missing
Missing usage was reconciled as zero and fully refunded the pre-call
reservation, allowing limit bypass on repeated successful calls. Only
adjust counters when usage is resolved from the response or standard
logging fields; otherwise keep the reservation until TTL expires.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: enforce RPM/TPM alongside IO-token limits on mixed deployments
Deployments with both itpm/otpm and tpm/rpm previously returned after the
IO reservation and skipped RPM/TPM checks. Run both paths and refund the
IO reservation only when RPM/TPM rejects after a successful reservation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: track TPM usage on success for mixed IO+TPM deployments
The early return after IO-token reconciliation in log_success_event and
async_log_success_event skipped the TPM counter increment, so the tpm_key
the pre-call check reads was never written and tpm_limit was never
actually enforced on deployments that also configure itpm/otpm.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat total-only usage as unresolved in IO-token reconcile
usage/standard_logging_object entries carrying only total_tokens (no
prompt/completion or input/output breakdown) were treated as resolved
usage, resolving to (0, 0) and refunding the full reservation. Both
_usage_is_present and the standard_logging_object fallback now require an
actual input/output breakdown before reconciling, keeping the reservation
otherwise.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: reserve minimal token when input/output estimation fails
_reservation_value(0, limit) reserved the entire limit whenever token
estimation failed (empty/unsupported input, tokenizer error), letting one
such request claim the whole bucket and 429 every concurrent request to
the deployment until it completed. Reserve 1 token instead so estimation
failures no longer serialize traffic.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: refund IO reservation synchronously before retry deployment pick
On retry, set_io_token_rate_limit_request_kwargs clears reservation
sentinels from the shared kwargs dict before a background failure handler
can refund them, stranding the counter until TTL. Refund and clear any
stale reservation in _update_kwargs_with_deployment before stripping
sentinels for the next attempt.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling
Pass the deployment litellm_params.model to token_counter so it uses the
model's native tokenizer instead of the generic fallback, narrowing the
reservation over/under-estimate window between pre-call and post-call
reconcile.
Add a ponytail: comment to refund_stale_reservation_before_retry explaining
the known ceiling: the synchronous DualCache.increment_cache issues a
blocking Redis INCR when a Redis backend is configured. This only fires on
streaming mid-stream retries (non-streaming failures await their failure
handler before the retry picks a new deployment, leaving no sentinels to
refund). Upgrade path: make _update_kwargs_with_deployment async.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
