* fix(router): fall back from unhealthy auto-router tier
Co-Authored-By: Claude Code <noreply@anthropic.com>
(cherry picked from commit 00c7fd8376)
* fix(router): treat budget and tag exhaustion as a no-capacity verdict
The eligibility probe only read typed router errors as "nothing here can
serve this". Provider and deployment budget exhaustion, and tag routing
with no matching deployment, report it as a bare ValueError carrying a
RouterErrors marker, so the probe read a spent tier as live, skipped the
peer and default recovery, and failed the request.
---------
Co-authored-by: Tin Chi Lo <tin@berri.ai>
Co-authored-by: Claude Code <noreply@anthropic.com>
The provider budget push runs inside the request success callback, so
awaiting the Redis pipeline there made every request wait for the round
trip. Hand it back to a task whose failure is logged through the breaker
aware logger, so an open breaker stays a debug line and a real Redis error
is one error line instead of an unretrieved task traceback
While the Redis circuit breaker is open every guarded call was refused with a bare
Exception that each swallowing catch site logged as an ERROR traceback, so a
sub-second latency blip turned into thousands of tracebacks per minute and pinned
every replica's CPU. The sync guard also recorded socket timeouts as hard
connectivity failures, so with least-busy routing the breaker opened on the first
slow replies and the timeout-only min-duration guard never applied.
Refusals now raise RedisCircuitBreakerOpenError, and the catch sites route it
through log_redis_failure, which logs a refusal at DEBUG and everything else at
the caller's level. The sync guard passes is_timeout like the async one.
A team-scoped auto-router is stored under an internal
model_name_{team_id}_{uuid} with the caller-facing name in
model_info.team_public_model_name, and the four pre-routing strategy
registries key on that internal name. A team key asks for the public name,
so the strategy lookup missed, the team early-resolve exit handed back the
marker deployment itself, and every call 400'd with "Unmapped LLM provider".
The strategy lookup now resolves the requested name through the same
team-first, then global, then admin-across-teams deployment resolution the
deployment path uses, and looks the registries up under the model_name of
whatever that resolves to. Both exits of _common_checks_available_deployment
drop strategy markers through one helper, so a marker-only resolution is
rejected as uncallable on every path. The request team id has one reader.
Resolves LIT-7363
Claude-Session: https://claude.ai/code/session_01NU97S7d2FUDDvTk59k53Wp
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(router): support percentile-based TTFT routing
* fix(router): apply routing_strategy_args updates to the live selector
Runtime routing_strategy_args updates (config reload, update_settings)
only rebuilt the strategy selector when routing_strategy itself changed,
so a newly added ttft_percentile sat unused until the proxy restarted.
Also drops a comment that only restated the code it sat above.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* refactor(router): drop unreachable empty-samples guard in percentile latency
_percentile_latency is only called behind use_ttft, which already requires
a non-empty ttft sample list, so the early return was dead code and the one
line Codecov flagged as uncovered on this patch.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): cover the no-selector path of a routing_strategy_args update
simple-shuffle has no selector attribute to re-link, so the early return
guards a setattr with a None attribute name. Dropping the guard makes the
new test fail with "attribute name must be string, not 'NoneType'".
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(test): assert ValidationError on out-of-range ttft_percentile
pytest.raises(ValueError) tripped PT011 for being too broad. Pydantic
raises ValidationError for the gt/le constraint, so naming it satisfies
the rule and pins the assertion to the constraint under test.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(router): drop Final from a per-deployment loop variable
basedpyright rejects "A Final variable cannot be assigned within a loop",
which pushed reportGeneralTypeIssues one over its budget. selected_latency
is rebound each iteration, so it matches its unannotated neighbours in the
same loop.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): exempt _apply_updated_routing_strategy_args from the name scan
The scan only reads test files with "router" in the filename, so it cannot
see the update_settings tests in router_strategy/test_lowest_latency.py.
Calling the private helper directly would test structure rather than
behaviour, so it joins the existing entries ignored for the same reason.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(router): keep deployment tags out of retry and fallback tag routing
Deployment-level tags merged into request metadata for spend attribution were also read as caller tag constraints on later attempts, so a tag-filtered group re-narrowed to the deployment that just failed. Snapshot the caller's routing tags before the merge and have tag routing read that snapshot.
Resolves LIT-7113
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): drop redundant comment in tag routing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): make tag retry regression deterministic and cover routing snapshot helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): give cost-based routing its own cache key so it stops overwriting latency samples
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): prefix the cost routing cache key so it cannot alias another group's latency key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
simple_shuffle only looked at healthy_deployments[0] to decide whether a
metric was configured, so a weight, rpm, or tpm on a later deployment was
ignored and the pick fell back to uniform random. Decide the metric from
all healthy deployments and keep the total_weight <= 0 fall-through.
Resolves LIT-7112
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep per-request routing_strategy override selectors out of global callbacks
Override selectors built by _get_override_strategy_selector were registered in litellm.callbacks / litellm.input_callback and cached for the router's lifetime, so one request with router_settings_override made every later request on the worker run that selector's pre-call check (rpm enforcement for usage-based-routing-v2). Build them with register_callbacks=False and run the selector's own pre_call_check / async_pre_call_check only for the request that carried the override.
Resolves LIT-7064
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): run override selector pre-call check on specific-deployment early returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover pass-through and affinity-pinned override pre-call paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A client behind an auto-router sends one max_tokens for every tier, so a value
sized for the smallest tier starves a bigger tier's thinking budget and a value
sized for the biggest is rejected by the smallest. After the complexity router
picks a tier, its per-tier litellm_params now carry max_tokens set to the
smallest max_output_tokens across that tier model's deployments (model_info,
then the cost map), applied the same way a per-tier reasoning_effort already
is, on every routing exit including plan mode, the empty-ask default and the
classifier fallback. The router seam collapses whichever ceiling alias a tier
carries onto the surface's own name, so one tier max_tokens reaches chat,
/v1/messages and /v1/responses alike, drops the caller's other carriers of the
same setting before the merge, and stamps the caller's original once so a
fallback into a group no tier owns gets it back instead of a ceiling sized for
the tier that failed.
Proxy-level reservations were sized from the caller's cap before routing, so a
raised cap left them short. Both owners now re-validate at the deployment hook:
the v3 limiter tops up its combined-TPM and project-OTPM reservations to the
final cap or writes the admitted cap back, and the budget limiter re-estimates
on the chosen deployment and grows the reservation or writes the admitted cap
back. An auto-router alias also reserves budget at its priciest tier model now
instead of pricing to zero.
An explicit per-tier max_tokens, max_completion_tokens or max_output_tokens
still wins, and max_tokens_from_tier_model: false forwards the caller's value
unchanged.
The Advanced scoring editor now lists built-in and custom dimensions together. Editing any weight holds it and rescales the others proportionally so the vector totals 1.00, and Save stores those explicit values. The backend scores exactly what is stored, with no runtime normalization, so routers nobody edits keep their weights.
CustomDimension gains an opt-in scoring_mode. match_count scores 0, 0.5 or 1 by distinct matcher hits; the default stays binary. The tuning fingerprint omits a binary scoring_mode, so routers written before this change keep their recorded baseline and the upgrade does not consume the free heuristic-v1 tuning slot.
Cuts the explanatory comments and docstrings added here down to one line each, or
removes them where the code already says it. Restores the four pre-existing
docstrings this PR had reworded to their original text; the one remaining edit to
existing text is TierDefinition.description, whose hardcoded tier list would
otherwise misstate that a tier named NON_REASONING may also omit its description.
Agent harnesses send a lot of operational turns that relay or reformat tool
output rather than reason about it, and the cheapest built-in tier was SIMPLE.
NON_REASONING adds a rung below it, behind enable_non_reasoning_tier so an
already-deployed router cannot move.
The toggle is what keeps it safe. The tier set feeds the classifier rubric, the
response-format enum, the escalation ladder and the savings baseline, so a
default-on fifth tier would have changed what every existing router sends and
where its traffic lands. Off, the ladder, rubric, wire labels and baseline are
byte-identical to before. On, the rung is added at index 0, escalation walks up
out of it, and it can never win the savings baseline.
It requires an llm or custom classifier and a model of its own: the v1 score
ladder has no rung below simple_medium and the v2 artifact is trained on four
classes, so the heuristic scorers cannot produce the tier and a router that
enabled it there would pay for a bullet nothing reaches.
The dashboard follows the same flag, and the edit modal now reads the tier back
from the stored config rather than assuming four keys, since it rewrites tiers
wholesale on save and would otherwise delete a hand-written tier on any edit.
* fix(router): rank streaming latency routing by raw TTFT, not TTFT per token
Latency-based routing divided time-to-first-token by completion_tokens
before storing it, so a deployment that streamed a long answer looked
faster to first token than one that answered briefly. TTFT is now stored
as plain seconds (first token time minus request start) in both the sync
and async success handlers, which is what the routing decision compares.
Non-streaming latency normalization per output token is unchanged.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* fix(router): store streaming TTFT under a seconds-only cache key
Workers on the previous release keep writing seconds-per-token samples
under "time_to_first_token" in the shared router cache during a rolling
deploy, so mixing the new raw-seconds samples into the same list averaged
incompatible units. Raw TTFT now lives under "time_to_first_token_seconds"
and routing reads only that key.
Also fix the regression test's token counts: with 50 tokens on the fast
deployment and 500 on the slow one the old per-token formula picks the
slow deployment, so the routing assertion now catches the bug.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* test(router): cover the TTFT sliding window from the unit-test shard
Move the TTFT list trimming checks from the CircleCI-only suite into the
mapped unit test file as one sync/async parametrized test, so the changed
lines in lowest_latency.py are exercised by the GitHub unit-test shard
that reports patch coverage.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
The prompt-management factory picks its deployment with a placeholder message. That
was inert while the pick ran on the synchronous path, which never runs the routing
plugin pipeline. Now that the pick runs the pipeline, a plugin classifying request
content would score the placeholder instead of the conversation, and the narrowing
it writes decides which deployments the real call may use.
Two routers in one process shared a single handler, because the callback
manager dedupes on the class name plus the handler's public attributes and the
handler had none. The second router's requests were never counted. The handler
now carries the id of the cache it was built on, so routers with different
caches both register while the two selectors one router builds for its routing
groups still collapse into one.
Clamping a negative count back to zero used SET, which drops the key's TTL, so
the next write started the hour over. It uses INCRBY by the negative amount now,
which leaves the expiry alone.
The Lua script had no test that ran it, so tests/local_testing covers both the
sync and async paths against a real Redis, and the file is wired into the
CircleCI job that provides one.
A Redis outage read as "every deployment is idle", because batch_get_cache
swallows the failure and answers with an empty dict. batch_get_counts and its
async twin raise instead, so a worker that cannot reach Redis falls back to its
own numbers rather than routing on zeros.
The counter's TTL is now set only on a key that has none, so a +1 left behind by
a worker that died mid-request ages out an hour after the key was created. It
used to be refreshed on every touch, which kept that stuck count alive for as
long as the group took traffic.
Two least-busy groups counted the same request twice, since the pre-call list
kept a selector per group while the success list deduped by class. The selector
now goes on through add_litellm_input_callback, which dedupes the same way.
A prompt-management model picked its deployment on the synchronous path, so the
new Redis read landed on the event loop and configured routing plugins never
ran. It awaits the async selector now.
A decrement whose matching increment is gone, because the counter key
expired while the request was still in flight, used to recreate the key
at -1, and a deployment with a negative count looks permanently idle, so
it collects every pick from then on. The repair write that followed the
decrement could also land after another pod's increment and erase it.
The increment, the clamp at zero and the TTL refresh now run as a single
Lua call, so nothing can interleave between them.
Keep each worker's own in-flight counter up to date alongside the shared one,
so a Redis outage routes on that worker's counts the way it did before this
branch instead of treating every deployment as idle.
Floor a counter at zero when a decrement finds the key gone, which happens when
a request outlives the 1 hour TTL, so an expired counter cannot settle at -1 and
win every pick.
Least-busy kept one dict of in-flight counts per model group in the
router cache, which reads in-memory first, so every worker and replica
routed on its own stale copy and each write overwrote the shared value.
Counts now live in one key per deployment, incremented and read through
Redis when the router has a Redis cache, and in the process-local cache
otherwise.
_average_latency skipped integer samples in the sum while counting them in the denominator, which contradicted its own
Sequence[float | int] signature; it now averages every sample. Both success loggers in cost-based routing computed
response_ms / completion_tokens and threw the result away, so a chat response with zero completion tokens raised
ZeroDivisionError inside the handler. The proxy swallows and logs it, but the handler then skips that request's tpm and
rpm update, so cost-based routing undercounts the deployment's usage. The QA run for the latency fix hit it on real
gpt-5.5 traffic through /v1/chat/completions and /v1/messages
Latency-based routing averaged a deployment's cached samples with total / len(samples) and raised ZeroDivisionError
once an entry held none, which the proxy answered as a 500 for every later request on that model group. Cost-based
routing writes the same {model_group}_map entry with minute counters only, so a group used by both strategies hit this
on every latency-routed request. A deployment with no samples now counts as 0 latency, the same as one the router has
never seen
Resolves LIT-7053
The previous cleanup only ran inside a caller's own except handler, so a
build that failed after its only caller had already been cancelled left
the failed task cached with nothing left to clear it. Move the cleanup
onto the task itself as a done-callback, which fires whether or not
anyone is still awaiting it, so the next request always gets a fresh
attempt instead of replaying the stale failure.
Adds a regression test for exactly that ordering (cancel the only
caller, let the build fail unobserved, then confirm the next request
builds successfully); it fails against the previous except-based
cleanup, which left the task cached.
test_load_state_from_db_handles_unknown_request_type compared the
WRITING cell after load against a cold-start value captured for
GENERAL. They happened to be equal for this fixture (the fast model's
empty strengths list makes every request type's prior identical), which
hid that the assertion was comparing the wrong baseline. Capture each
request type's own cold-start value instead.
_ensure_routelayer previously awaited asyncio.to_thread(...) directly
inside the lock. Under cancel_on_disconnect, cancelling that await
released the lock while the worker thread kept running, so a second
concurrent request would see no lock held and start a duplicate billed
build. Store the build as a task on self and have every caller await it
through asyncio.shield: cancelling one caller's wait no longer cancels
the build or lets another caller start a second one. A real build
failure (not merely a cancelled caller) clears the slot so the next
call retries fresh instead of replaying the same failure forever.
Also replaces the two tests that monkeypatched _build_routelayer (an
anti-pattern per this repo's conventions) with ones that instrument the
already-injected embedding router dependency instead, and adds a third
proving the cancellation race is actually closed.
