The LLM classifier capped every prior turn at 200 characters independently, so a
785 character turn was cut even when the whole block it belonged to was 353
characters. A character budget now bounds the block: turns are taken newest first
and quoted whole while they fit, older turns are dropped whole once it runs out,
and only the turn straddling the boundary is cut. The per-turn cap stays as an
optional clamp for operators who set it deliberately, defaulting to unset.
The LLM classifier's conversation context cut each prior turn head-only, so a turn
opening with an incident report and closing with the actual request reached the
classifier as the incident report alone. Keeping head and tail costs the same
budget and is what the truncation literature measures as best for classifying
long text.
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
* feat(complexity_router): add business classification rubric preset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate api schema for business rubric
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): suppress preexisting antd import violations in touched files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: tin <tin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The forwarded-keys record moves off the shared metadata dict onto the
per-request kwargs, where _update_kwargs_with_deployment consumes it, so
sibling requests that share a metadata dict (abatch_completion) can no
longer clear it mid-routing. Per-tier litellm_params from the hook
response are never treated as forwarded marker params, and a deployment
only beats a forwarded value when it sets its own, not when it carries a
LiteLLM_Params default such as merge_reasoning_content_in_choices=False.
An `auto_router/<alias>` marker entry's litellm_params (for example
`aws_region_name: eu-west-3`) were forwarded onto every routed call with
setdefault and then won the `{**litellm_params, **kwargs}` merge against
the selected tier's own values, so a Bedrock tier pinned to us-east-1 was
called in eu-west-3 and failed with 400.
The hook now records which keys it actually forwarded on the request's
metadata bucket, and `_update_kwargs_with_deployment` drops every
forwarded key the selected deployment defines itself, so marker params
only fill gaps a tier leaves open. Request-supplied values still win over
both. The stamp is stripped from logged metadata like its siblings.
Fixes#37613
The reasoning override's floor was pinned to tier_boundaries.simple_medium,
so an operator could not restore the unconditional promotion nor raise the bar
independently of the SIMPLE/MEDIUM cut. Setting reasoning_override_min_score
was accepted and echoed back by /model/info, because the config model allows
extra keys, while routing ignored it.
Resolve the floor through one accessor that falls back to simple_medium when
the field is unset, so moving that boundary still moves the floor with it, and
an explicit 0 is a real floor rather than an absent one. Record the resolved
value on the routing decision so a logged row states the floor that applied,
which is also what lets the Admin UI stop hardcoding the copy PR #37500 added.
Two or more reasoning keyword matches promoted a request straight to the
REASONING tier no matter what the weighted score said, so "hi, step by step,
pros and cons" scored 0.100 and still bought the most expensive tier.
Require the score to clear the simple_medium boundary before the override
applies. Promotion from MEDIUM or COMPLEX is unchanged; only prompts the
scorer already placed in the cheapest band stay there.
* feat(complexity_router): custom classifier plugins via classifier_type 'plugin'
Adds a third classification mode where an operator-supplied hook decides the
tier instead of the heuristic scorer or the LLM classifier. The hook implements
an async classify(context) returning a tier name (built-in value, tier_labels
label, or tier_definitions name) or None to decline; failures, timeouts, and
unknown tiers fall back exactly like a failed LLM classifier. The context
carries the request messages and metadata, including caller identity, so a
plugin can route by team, spend, or any business rule.
The plugin resolves from a dotted path at proxy startup with a load-time check
that classify is a coroutine function, and is closed off over HTTP like the
routing plugins list. Routing decisions record the new classifier_plugin cause.
tier_definitions now accepts classifier_type 'plugin' alongside 'llm'.
* fix(proxy): resolve plugin dotted paths in _delete_deployment before hashing ids
The db-sync reconcile re-reads the raw config and hashes litellm_params to
compute which ids the config wants served, but the router's ids were hashed
from the resolved params where plugin dotted paths are live instances. The
mismatched ids made the reconcile evict every plugin-bearing auto-router one
sync after startup, on any proxy with a database connected. This also affected
the existing routing plugins list, not just the new classifier plugin.
Resolving the plugins in _delete_deployment the same way load_config does makes
both sides hash the same canonical form. A plugin module broken on disk at
reconcile time skips cleanup instead of evicting valid deployments, matching
how a get_config failure is handled
* fix(complexity_router): treat non-string plugin verdicts as declines, centralize the empty-mapping sentinel
A hook returning a non-string raised inside resolve_classified_tier outside the
plugin exception boundary, failing the request instead of falling back. Also
moves the read-only empty mapping to constants.py per repo convention and moves
the classifier plugin product docs out of the package README for the docs repo
* refactor(complexity_router): rename the plugin classifier mode to classifier_type 'custom'
The mode value now names the operator's intent while classifier_plugin keeps
naming the mechanism; routing decisions keep the classifier_plugin cause
* refactor(proxy): pin plugin-bearing deployment ids from the raw params instead of resolving in the reconcile
Replaces the previous approach of re-running plugin resolution inside
_delete_deployment, which imported operator modules on every reconcile cycle
and skipped the whole cleanup pass when any one module was broken on disk.
load_config now stamps model_info.id from the raw litellm_params before
resolution swaps dotted paths for live instances, so the reconcile's raw-config
hash matches by construction and needs no resolution at all: a broken module
cannot stall cleanup for unrelated models, and any future param-transforming
resolution is covered by the same pin. _generate_model_id becomes a staticmethod
so the pin can run before the Router exists; its statically dead non-string key
branches are removed. Also documents candidate_models as an informational
snapshot for classifier plugins, unlike the narrowing surface RoutingPlugin
filters
* fix(router): restore _generate_model_id key handling, align classifier context with the routing-plugin pattern
The staticmethod conversion accidentally dropped the non-string-key branches
from _generate_model_id, a silent hash change for any params with non-string
keys; they are restored verbatim. The classifier plugin context now follows
the Router-level routing-plugin recipe exactly: structured messages come from
resolve_structured_messages over the raw messages, and the metadata key comes
from the shared get_metadata_variable_name_from_kwargs helper, which also
replaces the duplicated inline sniff in _pick_model_for_tier. This removes the
raw-or-resolved fallback where a plugin could silently receive resolved
messages when a call site forgot to pass the raw ones
* refactor(router): make generate_model_id public, guard classifier context construction
Two modules legitimately hash deployment ids with the same helper now (Router
and the proxy's config-load pin), so the private name was lying about its
audience and the cross-module call needed a pyright suppression; renaming it
public restores the static safety net. The classifier plugin's RoutingContext
construction moves inside the failure boundary, matching the LLM path where
litellm-side prompt building also falls back rather than failing the request,
and a prompt-only call with no message list is now covered by a test
* feat(complexity_router): plan-mode tier floor for coding-agent clients
Claude Code and Copilot signal plan mode only through client-injected prompt
text, which the ask-extraction path deliberately strips, so the router could
never see it. Detect the sentinels on the raw wire body and route those
requests to at least plan_mode_min_tier.
The floor is raise-only and transient: classifier results above it still win,
it overrides a session-affinity pin only on turns carrying the sentinel
without rewriting the pin, and plan_mode decisions are not pinnable, so the
first turn after plan mode exits routes as if plan mode had never happened.
Classification is skipped when the floor is the top configured tier. On
adaptive routers the floor rides _soft_floor_pick as a hard_floor that
excludes below-floor candidates, closing the adaptive_eligible=all gap where
a request classified at or above the floor could still route below it.
Detection is staleness-aware: only leading system content and the newest-ask
tail count, so sentinels surviving in history after plan mode exits, built-in
or operator-supplied, never fire. Custom tier sets are supported with
severity from the tier_definitions list order, same as keyword_tier_rules.
Off by default; decisions are recorded with the new plan_mode cause and the
matched sentinel in matched_keyword
* fix(complexity_router): gate pin writes and the failure exit on sentinel presence, not the floor binding
A plan-mode turn classified at or above the floor keeps its ordinary cause,
but pinning it would carry a plan-mode-shaped choice past plan mode's exit
(on adaptive routers the hard floor constrained that pick), so no
sentinel-carrying turn writes the session pin. The default_model failure
exit is skipped for sentinel turns for the same reason: default_model's
placeholder tier can equal the floor while default_model itself sits in no
pool the floor can vouch for
* feat(complexity_router): calibrate the classifier rubric with worked examples
The built-in rubric stated its tier boundaries as prose alone, and prose
calibrated to consumer chat puts "non-trivial code, multi-step technical work"
at the top of the scale. That is the median request in developer and agent
traffic, so ordinary engineering read as top-tier and the router paid for the
most expensive model on it.
Adds calibration examples to the rubric, selected by a new
classifier_llm_config.rubric preset. The agentic preset (now the default)
anchors routine installs, builds, multi-file edits, and standard debugging at
MEDIUM; the chat preset omits those anchors for deployments serving only
conversational traffic. Both share the same tier criteria, the trust-boundary
paragraph, and the context-window closing line, so this moves where the
boundary sits without changing the taxonomy.
Both presets render byte-identical to the strings a prompt sweep scored, and a
test pins that, so the measured accuracy describes what a router sends.
* feat(ui): pick the classifier rubric preset on an auto-router
Adds a Rubric dropdown to the auto-router's classification panel, so the
agentic and chat presets are selectable rather than config-file only. The
prompt editor prefills from the selected preset, since prefilling agentic text
for a router on chat would show examples its classifier never receives.
The picker is disabled while a custom prompt is set, and the payload builder
drops the preset in that case: a custom prompt is the classifier's whole system
role, so the backend rejects the two together. The builder records the default
preset explicitly, so a later change to which preset is default cannot silently
move an existing router.
* fix(complexity_router): mark an unchosen rubric preset with None, not model_fields_set
The mutual-exclusion check read model_fields_set to tell an explicit preset
from the default. That flag does not survive serialization, and this config is
dumped and handed straight back to ComplexityRouter by /auto_router/test_routing,
where a dump re-states every field. So a custom-prompt classifier saved fine and
then failed validation on preview, rejecting on the second pass what it accepted
on the first.
The preset is now optional, with None meaning the default, matching how None
already means the built-in rubric for system_prompt on the same model. The
default lives in one place, DEFAULT_RUBRIC_PRESET, resolved where the prompt is
assembled. The dashboard stops sending a copy of the default it displays, so a
router nobody configured follows the default rather than pinning today's value,
and UI-built routers behave the same as hand-written config.
Regenerates schema.d.ts, which was left stale by an earlier description edit.
* feat(complexity_router): grandfather existing routers onto the uncalibrated rubric
An unset preset now means LEGACY, the rubric exactly as it shipped before
calibration examples existed, so upgrading cannot move the tier decisions or the
bill of a router that is already running. Config-file routers get this for free
since they name no preset, and a stored config that never had one reads the same
way.
New routers still get the calibrated rubric: switching a classifier to LLM
stamps the agentic preset, because a classifier being configured for the first
time has no prior tier behaviour to preserve. The picker offers legacy so an
existing router's state is representable and opening the form cannot silently
upgrade it.
Each preset is pinned byte-identical to the text the prompt sweep scored,
legacy included, which is what proves an existing router's prompt did not move.
Also collapses the preset data from a NamedTuple with group wrappers and
per-preset frozensets into plain text blocks in a MappingProxyType, matching how
the tier criteria next to it are already stored: 21 lines of prompt text no
longer cost 190 lines of constructors. Tiers are format placeholders so
tier_labels still reach the examples.
* refactor(complexity_router): name the field classification_rubric
`rubric` alone did not say what it selects, and the field sits beside
`system_prompt`, which genuinely is the whole classification prompt. The name
now says which of the two an operator is reaching for: the rubric the built-in
prompt is assembled from, not the prompt itself.
Renames the config field, the query param, the enum, and the dashboard label to
match, and moves the preset text to classification_rubrics.py.
* test(ui): set the preset the mutual-exclusion case is meant to drop
The rename left classification_classification_rubric in the custom-prompt case,
so its input never carried a preset and the assertion held for the wrong reason:
it proved an absent preset stays absent, not that a set one is dropped. A
normalizer that forwards the preset whenever one is set passed with the typo and
fails without it.
tsc reports the typo as TS2353; the earlier sweep grepped for the source file
and not the test, so it went unseen.
* test(ui): scope the role-gate assertions to each page's own endpoint
The memory, workflows, and guardrails-monitor page tests asserted that a denied
role fires no request at all. Their names, and the assertion on the very next
line, say the intent is narrower: the page must not fetch its own data.
Resolving whether a caller is an org admin goes through /organization/list for
every role, since deciding org-admin-for-any-org needs the list, and the route
scopes rows per caller. That legitimate request fails a blanket no-fetch
assertion, so all three files went red on staging for a reason unrelated to
what they test.
Drops the blanket assertion and keeps the scoped one. Bypassing the gate in
memory/page.tsx still fails five tests, so the narrower assertion continues to
catch a genuinely broken gate.
* fix(complexity_router): document that an unset rubric keeps the legacy prompt
The field said 'Leave unset for agentic' while an omitted rubric resolves to
LEGACY, so the OpenAPI schema an operator reads promised calibrated routing
where they got the uncalibrated one.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
System prompts (harnesses, tools, framework boilerplate) are session-wide
constants identical across all requests. Scoring them saturates keyword-match
signals and produces false-positive high-complexity classifications on
trivial utterances like 'hi', routing them to expensive models (sonnet/opus)
instead of tier-1 haiku. A real ~1.6KB CLI-agent harness alone supplied
5 codePresence + 2 technicalTerms matches, overshadowing user signal.
Rescope four scoring dimensions (codePresence, technicalTerms, simpleIndicators,
multiStepPatterns) from full_text (system + user) to user_text (user only).
reasoningMarkers was already scoped this way. This returns 0.63 of the weight
budget to text that actually varies per-request.
Now that every dimension scores user_text only, _score_keyword_match's
disclosable_text param is redundant -- it existed solely to let the signal
name terms matched in the caller's own message while withholding terms
matched only in the (invisible-to-the-caller) system prompt. With no more
system-prompt text in scope, text and disclosable_text were identical at
every call site, so the param is dropped and the function collapses to a
single text argument.
Add mutation-proven regression test: trivial 'hi' message with realistic
Claude Code agent system prompt now routes to haiku tier-1 (not sonnet).
- Unfixed: haiku -> sonnet (bug)
- Fixed: haiku -> haiku (correct)
Invert three pre-existing assertions in TestSignalsNeverQuoteTheSystemPrompt
to capture the corrected behavior: system-prompt-only terms produce no signal.
Co-authored-by: Claude <noreply@anthropic.com>
* fix: never price a strategy-router alias
A strategy-router alias (auto_router/complexity_router/<name>) is never the
deployment that gets called or billed, but custom pricing configured on it was
being treated as real pricing in two places:
- registered in litellm.model_cost under the alias deployment id, so an
explicit zero made _is_cost_explicitly_configured() report the group as a
genuinely free model and every budget check was skipped, while the request
routed to a paid deployment and accrued real spend
- copied onto request_kwargs by the alias-params merge, so the routed
deployment got re-registered at the alias price and the request billed 0.0
Both are fixed at the writer, so config, /model/new and price-map reload all
take the same path
Co-Authored-By: Claude <noreply@anthropic.com>
* chore: annotate filtered cost-map copy for the mutable-collection gate
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(auto-router): track turns per complexity tier (LIT-5302)
Stamps complexity tier at decision time (rollup never re-derives from routed
model, since tier->model mapping is mutable config). Records per-tier turn
counts in LiteLLM_AutoRouterSession.tier_turns (jsonb), rolls up per router
in benchmarks SQL via jsonb_object_agg, returns on AutoRouterBenchmarkGroup
for dashboard turns/share metrics.
Addresses Greptile/Bugbot findings:
- Missing _SessionAggRow.tier_turns field: added with field_validator to
parse jsonb text cast and handle NULL. Would 500 every benchmarks read.
- Missing ::text cast on tier parameter: Postgres fails type inference on
parameterized CASE/IS NULL without explicit cast. Added to all usages.
- Docstring false claim (only complexity routers produce tiers): quality
router stamps numeric tier '1'/'2'/'3'. Per-type grouping in SQL prevents
cross-contamination. Rewrote docstring to clarify isolation.
- Comment convention violations: stripped per CLAUDE.md rule.
- Test gaps: 8 unit tests for extraction/validation/aggregation, 7 behavior
tests for SQL semantics against real Postgres. 12 mutations killed.
Fixed fragile complexity_router test that broke on nested function calls.
No API change; extends existing GET /auto_router/benchmarks response only.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): address review findings on tier turns tracking
- Guard router_type update so a mid-session reconfigure can't pool
foreign tier names into tier_turns
- Keep pinned turns attributed to the tier that actually serves them
- Drop stray -- AlterTable comment from hand-written migration
- Drop the now-unnecessary ::text/json.loads round-trip; prisma
already returns tier_turns as a parsed dict
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): satisfy type-discipline lint gate
- tier_turns fields: dict[str, int] -> Mapping[str, int] (LIT001,
mutable collection in annotation); these are read-only after
construction
- _summed_agg_row: {} -> MappingProxyType({}) (LIT002, mutable dict
literal)
- default-fallback branch: replace the reassigned-without-Final
fallback_tier with a Final default_model_first flag and a single
ternary assignment (LIT010)
Verified locally: type_discipline_gate.py, ruff_strict_gate.py, and
type_check_gate.py all pass against the litellm_internal_staging
merge-base; full test_complexity_router.py (374), auto_router
management-endpoint tests (26), db-layer rollup tests (31), and the
live-Postgres proxy_behavior rollup suite (17) all pass.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(auto-router): accept every reminder marker pair a harness emits
reminder_markers held one (open, close) pair, so a harness that wraps
injected context differently per agent type only got the slice of traffic
using the configured envelope stripped. Every other agent type kept hitting
the original bug: its reminder-only turn never stripped to empty, won
"newest human ask", and the harness blob got classified in place of the
real question, choosing the tier and therefore the spend.
The field now takes a list of ReminderMarkerPair, following the
KeywordTierRule pattern already in this file so each pair validates itself
and errors point at reminder_markers.N.close rather than a bare index.
Blocks from different pairs can nest, which the gap construction could not
handle: resuming the kept text at an inner block's end walks back inside
the enclosing block and leaks its remainder. Running the block ends through
a maximum collapses nested and overlapping spans without a separate merge
pass, and stays linear in block count, which a fold over a growing tuple
of merged spans would not.
A single pair's ends already increase, so the maximum is the identity and
the default path is byte-identical: verified against the shipped function
over 200k generated inputs, and every existing reminder test passes
unchanged. The prior single-pair config shape is rejected loudly at
startup and at /model/new rather than silently stripping nothing.
* docs(auto-router): document reminder_markers in the complexity router README
* chore(ui): regenerate dashboard API types for the reminder_markers shape
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
* fix(autorouter): match CJK keyword_tier_rules that regex word boundaries miss
Single-word keywords were matched with a \b...\b regex. Every CJK character is a
regex word character and CJK is written without spaces, so \b never fires between
two of them and a rule like 发票 silently missed 我需要开发票, falling through to
complexity scoring instead of the configured tier.
Keywords containing CJK now match as plain substrings, the same way multi-word
phrases already did. The gate reads the keyword rather than the prompt, so a
keyword with no CJK in it keeps word boundary matching regardless of the script
the prompt is written in.
* fix(autorouter): cover Han extensions in planes 2 and 3, not just up to U+2FA1F
The supplementary range stopped at U+2FA1F, so Extension G and H ideographs kept
the word boundary path and stayed unmatchable. Both planes are dedicated to CJK
ideographs, so covering them whole also handles later extensions without chasing
each new block.
* feat(auto-router): let operators replace the LLM classifier's system prompt
The complexity router's LLM classifier has always sent one built-in rubric, so the
router could only ever grade difficulty. Operators can now supply their own system
prompt, which replaces the rubric outright and repurposes the same tier machinery for
whatever taxonomy the prompt defines, data sensitivity being the obvious case.
Replacement is total: neither the rubric nor its closing line is appended, since both
describe grading difficulty over a "current message" and a prompt grading something
else is entitled to contradict them. That closing paragraph is also the classifier's
prompt-injection defense, so the config field and the dashboard editor both warn that
a replacement omitting it lets a caller ask for a tier and get it.
The heuristic fallback still scores complexity, which is meaningless for a repurposed
taxonomy, so classifier_fallback now chooses between the heuristic scorer and routing
straight to default_model. The default_model path bypasses tier pools, the adaptive
bandit, and escalation, because no tier was decided and the point of that fallback is
a known destination. It reports itself as default_model_fallback in the spend logs.
The dashboard's prompt editor prefills from a new
/auto_router/classifier/default_prompt endpoint rather than a copy of the rubric in
the frontend, and stores no override when the draft matches the default, so later
rubric improvements still reach every router that never customized it.
Tier names stay SIMPLE/MEDIUM/COMPLEX/REASONING; a custom prompt redefines what they
mean, not what they are called.
* fix(complexity-router): don't let the default_model classifier fallback bypass routing plugins
* fix(complexity-router): don't pin a session to the default model after a classifier failure
* fix(complexity-router): omit the tier from a default-model-fallback routing decision
The classifier never answered, so no tier was decided. The record reported the
tier whose pool happens to hold default_model, which reads in the spend log and
the UI as if the request was classified. Matches how default_fallback already
records a route that no tier produced.
* fix(proxy): allowlist /auto_router/ on the UI backend component
The new GET /auto_router/classifier/default_prompt is a UI-consumed management
route, so it belongs on the control plane. Without the prefix it was exposed by
neither component and test_gateway_plus_backend_covers_full_app failed.
* docs(ui): reword the classifier prompt disclaimer
Frames the closing paragraph as a strong recommendation rather than a
description of what gets dropped, names prompt injection explicitly, and
notes the tier names stay fixed regardless of their display names.
* fix(complexity-router): stop logging a fabricated tier on the plugin fallback path
The classifier-failed fallback resolves a tier so the routing-plugin pipeline has a
pool to filter, but nothing about the request produced that tier. The non-plugin
short-circuit already dropped it from the logged decision; the plugin path still
reported it, so a spend log claimed a classification the request never received.
Record the pool as a plugin-filtered-pool signal instead.
Also name the real problem when the resolved tier has no models at all: that raised
"No candidate models left after routing-plugin filtering" and sent operators hunting
for a policy plugin that never narrowed anything.
* feat(complexity_router): let operators rename the four complexity tiers
Adds an optional tier_labels map to complexity_router_config so a deployment can
put its own vocabulary on the four tiers, e.g. Cheap / Standard / Premium / Deep,
instead of reading SIMPLE / MEDIUM / COMPLEX / REASONING in its dashboard, its
spend logs, and the rubric the LLM classifier reasons with.
Labels are display-only. Every config key stays canonical, so tiers,
keyword_tier_rules[].tier, and tier_boundaries are written exactly as they are
without labels, and partial maps are fine with unlisted tiers keeping their
default name. A validator rejects blank labels, two tiers sharing a label, and a
label that is another tier's canonical name, since any of those would make a log
row or a rubric line ambiguous. That validator runs on the /model/new and
/model/update write path already, so an ambiguous config gets a 400 rather than
being stored for the router to refuse later.
Under the default heuristic scorer the names are cosmetic: the scorer maps a
weighted score to a rung and never reads a tier name, verified by running the
eval corpus with and without a rename and getting identical tier and identical
score on all 29 cases. Under classifier_type: llm the labels are the names in the
rubric and the values the classifier must return, so the response format's enum
is now built from the configured labels and a reply is resolved back to its tier
against labels first, then canonical names, case-insensitively. An unresolvable
reply degrades to the heuristic on the existing fallback path. A test pins the
generated schema for an unrenamed deployment as equal to the shipped
TierClassification schema, so the wire shape can't drift.
Spend logs keep routing_decision.tier canonical so rows from before and after a
rename stay comparable, and gain routing_decision.tier_label on the tiers that
were renamed.
* refactor(complexity_router): drop added comments and the Counter construction
Review feedback: the repository guide bans new comments, so the explanatory
comments and the appended docstring paragraphs this branch added come back out.
One-line docstrings stay in complexity_router.py, matching that file's own
convention.
The duplicate-label check no longer builds a Counter, which the mutable-collection
budget counts, and the error text drops its list() reprs for joined strings. The
labels are stripped in tier_label() now rather than by rewriting the field in the
validator, so the stored config keeps exactly what the operator wrote.
schema.d.ts is regenerated: ComplexityRouterConfig is exposed in the OpenAPI spec,
so tier_labels surfaces there.
* fix(ui): carry tier_labels through the auto-router preset prefill
buildPresetPrefill maps every payload key onto form state, but the tier_labels
key added by this branch had no line, so a preset shipping labels would apply
its tiers and silently drop its names.
* feat(spend): derive a default auto-router savings baseline from the hardest tier
The savings driver shipped off by default: unless an operator names
litellm_settings.autorouter_savings_baseline_model, every auto-routed request
records $0.00 and the dashboard card never populates. Nobody discovers a knob
whose feature they have never seen work, so the default has to come from
somewhere the proxy already knows.
The router's own tier ladder is that place. Without a router a deployment runs
one model that can carry the hardest request it will see, so the derived
baseline is the priciest model in the hardest configured tier, REASONING when
present, otherwise the most severe tier the router actually defines. A cheap
tier is a choice the router made, not a ceiling it was bounded by.
An earlier draft of #35521 derived this per request and was deleted for it:
ranking candidates against the request that ran meant reading the request, and
every input shape it could take produced its own review finding. This
derivation is ranked against one fixed reference request instead, a cache-heavy
shape matching real auto-routed traffic, so it never reads the request at all.
Candidates still resolve through the router's deployments, so Azure base_model
and per-deployment pricing overrides rank correctly.
The deciding router records the result on its routing_decision, because one
model name can carry several tag-scoped routers with different tier ladders and
only the deciding instance knows which of them routed the request. The spend
writer's precedence is: configured baseline, then the recorded one, then off.
When the setting is present the router skips deriving entirely rather than
pricing candidates per decision only to be ignored.
Resolution never raises; an unresolvable baseline zeroes the driver instead of
failing a live request. Rows queued by a pod on the previous release carry no
recorded baseline and fall back to the configured setting, exactly as today.
The schema.d.ts regeneration also picks up the reminder_markers field that
UI-19232 (#35874) added without regenerating, so one hunk there is inherited
staleness rather than part of this change.
* fix(spend): cache the derived baseline, price it by deployment, keep it out of the routing preview
Three review findings on the derived baseline, addressed together because they
all sit on the same value's path from derivation to consumer.
Derivation walked and priced the hardest tier's whole pool inside a property
read on every routing decision, unbounded by pool size. The router now caches
the result per instance with a 30 second TTL, None results included, so the
hot path is a clock compare and a deployment edit still lands within a window
no operator watches closer than.
Ranking used each deployment's effective pricing but recorded only the model
name, so the spend writer priced the winning baseline at its public rate: a
hardest tier whose deployment carries a negotiated rate produced materially
wrong savings. The decision now also records savings_baseline_deployment_id
and the writer resolves it through Router.get_deployment_model_info, exactly
as the selected arm already does. The id is ignored whenever the configured
setting overrides the recorded baseline, since the setting names a model, not
a deployment.
/auto_router/test_routing returns the routing decision verbatim to team admins
while only authorizing the classifier and embedding models, so a derived
baseline would resolve another team's model-group alias into its backend
provider/model mapping and hand it to a caller never authorized for it. The
preview's throwaway router is built with derive_savings_baseline=False; its
decisions are never spend-tracked, so nothing is lost, and a source-pinning
test keeps the flag on the endpoint.
Also strips the explanatory comments this PR had added.
* refactor(spend): pin the derived baseline per router instance instead of a TTL
Creating or editing a router already rebuilds its ComplexityRouter instance,
through unregister and re-add on upsert and through the registry reset on a
full model_list load, so a value derived once per instance refreshes on
exactly the flows that can change it. That makes the TTL a solution to a
problem the rebuild lifecycle already solves, and it goes.
Derivation stays deferred to first use rather than running in __init__: during
a config load this router can be constructed before the deployments its tiers
name, and a baseline pinned at that moment would be empty for the process
lifetime.
The one behavior the TTL had that the pin does not: editing a tier deployment
without touching the router itself refreshed the baseline within a window.
That edit path rebuilds only the edited deployment's own strategies, so the
pin holds the old answer until the router is next saved or the config next
loads. A stale deployment id degrades to public-rate pricing rather than
failing, which is where every other unresolvable baseline already lands.
* feat(auto-router): make reminder marker pair configurable
Some harnesses inject internal context using their own marker pair
instead of Claude Code's <system-reminder>/</system-reminder>
convention, and some send it as a separate follow-up user message
rather than inline with the ask. Both cases fall out of the same root
cause: the router's marker-matching is hardcoded, so foreign markers
never strip to empty and the reminder-only turn wins "newest human
ask" selection instead of being skipped.
Add an optional reminder_markers field to ComplexityRouterConfig so
operators can override the (open, close) pair via proxy config, with
the existing skip-when-empty selection logic handling both cases once
the markers match.
* test(auto-router): drop unsolicited comments from the reminder-markers regression test
Per Greptile review on #35874: no comments unless explicitly requested.
* feat(spend): add net auto-router savings to the cost-optimization dashboard
The dashboard credited compression and prompt caching but said nothing about the
optimization that picks the model, so the driver with the largest lever on a bill
was the one an operator could not see.
Savings are the counterfactual: without a router a deployment runs one model, and
it has to be one that can carry the hardest request, so the baseline is the
priciest model in the router's hardest configured tier. A cheap tier is a choice
the router made, not a ceiling it was bounded by. `auto_router_savings_baseline_model`
overrides it for operators who would genuinely have run something else. Both are
provider-qualified before pricing, because a bare name can resolve to a different
vendor's rates or to nothing at all, and a deployment is priced by its `base_model`
where it has one, which is how Azure deployments are priced everywhere else.
Both arms price the request's real usage through `generic_cost_per_token` rather
than re-deriving per-token arithmetic, so tiered rates, ephemeral cache-write tiers
and regional uplifts stay consistent with what was actually billed. `prompt_tokens`
already includes the cache buckets, so charging them again at the input rate would
price the same tokens twice.
Cache state is what makes this hard. The baseline serves every turn, so whether it
had the prompt cached is whether the conversation was already underway. On a
continuing conversation it wrote the prompt earlier and would only read it now, so
this request's write is what switching cost and counts against the saving. On a
first turn nothing was cached for any model, the baseline would have written the
same prompt, and both arms carry the write at their own rates. Charging the write
to both cases understates a first turn to a few percent of its value, and because
the write premium is fixed by prompt size while the saving grows with completion
length, it can render a profitable route as a loss.
That shape is read off the conversation rather than remembered: a second human ask
means an earlier turn was served. No cache, no session id, and no dependence on a
caller sending a session header. It cannot see a switch on a turn the router did
not classify, and it reads a few-shot prompt's synthetic turns as prior
conversation; both err toward charging the write, which under-claims.
The baseline and the shape ride on the existing `routing_decision` record, which is
already carried from the router to the spend log, already classified for redaction,
and already written-or-cleared per attempt. A fallback that re-enters the hook
therefore cannot leave either fact behind to be attributed to a deployment that
never routed, and no new metadata key crosses the trust boundary.
The result is signed. Whether a switch pays off is a race between the rate gap and
the cache-write cost, and a narrow gap loses; flooring at zero would hide exactly
the routing behaviour an operator needs to see. The donut plots only drivers that
saved, while the card and range total keep the sign.
Savings accrue into a new `autorouter_savings_spend` column on the six daily rollup
tables, declared `NotRequired` because rows queued by a pod on the previous release
carry no such key. It is summed by the rollup merge the cross-pod Redis drain also
runs, and carried through the aggregation query, the per-row accumulation and the
response model, so the dashboard reads a value the API actually sends. Tests
enumerate the drivers from the response model itself and assert each is summed,
accumulated, carried and totalled, so one added later cannot be half-wired.
* fix(spend): let the baseline pay for a continuing turn's own growth
`_baseline_usage` moved every cache-creation token into the baseline's read bucket
whenever the conversation was underway. That is right for a switch, where the
baseline never left the model it was on and really would only read, but wrong for a
turn that stayed put: the prompt grew, and the tokens written are that growth. They
are new to every model, so the baseline would have paid to write them too. Forgiving
it that write made the counterfactual cheaper than it was and shrank the reported
saving on ordinary steady-state traffic, by about 2% per turn.
The selected arm was never involved; it has always been priced on the real usage.
The error sat entirely on the baseline.
The condition is that the request read more than it wrote, not that it read anything.
A switch onto a model already holding a small prefix of this prompt still writes most
of it, and that write is the switch's own cost; keying off a nonzero read would have
handed such a request the full rate gap, turning +$0.0056 into +$0.1177. Comparing
the two buckets separates a warm continuation, which reads far more than it writes,
from a cold arrival, which does the reverse, and it leaves the existing invariant
intact: a request reading 0 and one reading 1 both still land in the same place.
* fix(spend): price each arm under the key litellm billed it, and see agent turns
Two ways the savings number read the wrong thing, both from identifying a model by
its name when the name is not what it costs.
The counterfactual was ranked and priced on the public rate for the model a
deployment names. A deployment may not be charged that rate: the router registers
its configured prices under the deployment's own id and deliberately keeps them off
the shared model-name key so deployments sharing a backend model do not pollute each
other. So a hardest-tier deployment configured above its public rate lost the
ranking to a cheaper candidate, and once chosen was priced at a rate nobody pays.
Which key prices a deployment is now `_select_model_name_for_cost_calc`'s decision,
the resolver the real request is billed through, rather than a second rule here that
would have to re-learn that per-second and tiered overrides count, that a partial
override still counts, and that a deployment configured at zero is priced at zero
rather than treated as unpriced.
The arm being subtracted had the same fault and a sharper edge. It priced the spend
log's `model`, which on Azure is the deployment name, absent from the cost map, so
the whole driver silently read zero for that traffic. It no longer re-derives
anything: `model_map_information.model_map_key` is what litellm actually billed the
request under, recorded at request time by that same resolver with `base_model` and
custom pricing already applied.
Separately, the conversation-shape discriminator counted human asks, and an agent
loop can run twenty turns on one of them. Its tool traffic rides `tool_result`
blocks on user turns that flatten to empty text, and `tool` roles that are never
read, so a long agentic conversation looked like its own first turn and was handed
the arithmetic that leaves the cache write on both arms. That is the one direction
this must never fail in, because it inflates. An assistant turn is the direct
evidence that something answered earlier, and it is blind to how the tool plumbing
is spelled on either surface.
* fix(spend): give the cost-key resolver both inputs the selected arm needs
The served model was resolved through one input at a time, and each choice broke the
half the other fixed.
`model_map_key` is the served model already resolved through `base_model`, which is
the only way an Azure deployment name reaches the cost map at all; without it the
selected arm priced a name absent from the map, returned nothing, and the whole
driver silently read zero for that traffic. But it is built without
`router_model_id`, so it never carries a deployment's own price overrides, and a
custom-priced deployment was compared at its public rate while the baseline used the
real override. On a deployment configured well above its public rate that inverted
the answer outright: a route that lost $21.88 reported saving $0.10.
`_select_model_name_for_cost_calc` takes both, so it gets both. Which key prices a
deployment stays its decision rather than a rule restated here.
* fix(spend): same model is only the same cost when it is the same deployment
The short-circuit compared resolved model identity, so two deployments of one model
collapsed to "no switch" and reported zero. They are not the same cost: a deployment
can carry a negotiated rate, and routing from the dear one to the list-price one is a
real saving the dashboard reported as $0.00 against a true $21.93.
Both arms now carry the key litellm prices them under, so the comparison is between
deployments rather than between names.
* refactor(spend): price from resolved rates, not from a name we keep re-resolving
Four review rounds landed on one mechanism: which identifier prices a deployment.
base_model, then the deployment id, then cache-only overrides. Each round added a
clause to a resolution rule that should not exist, and a wrong primitive fails once
per input shape, so each shape arrived as its own finding.
`Router.get_deployment_model_info` already owns this. It merges a deployment's
configured prices over the built-in map, folds in `base_model` defaults for
deployments whose name is not a model, and falls back to the model name when nothing
is overridden. Every shape hand-rolled here (cache-only, partial, per-second, Azure)
was that function re-implemented badly.
`generic_cost_per_token` now accepts already-resolved rates instead of demanding a
name it looks up itself, which is what forced the name-bending in the first place.
Both arms resolve through the owner and pass what they got: the counterfactual by the
deployment the router would have used, the served request by the deployment that
served it. The invented cost-key resolver is gone, and `Baseline` carries a
deployment id rather than a key we chose on litellm's behalf.
Net 64 insertions against 79 deletions.
* test(spend): follow _most_expensive onto the router that prices its candidates
Ranking moved through `Router.get_deployment_model_info`, since what a deployment
costs is the router's answer to give; these four cases were still calling the old
free-function signature.
* fix(spend): rank baseline candidates by what a request costs, not by two rates
"Most expensive" was decided by comparing output rate then input rate. That is a
property of a rate, not of a request: a deployment dearer per output token can be
cheaper per cached token, so the comparison ordered cache-heavy traffic backwards and
recorded the wrong counterfactual.
Candidates are now costed on one reference request through the same engine the
savings themselves use, which leaves cache read and write rates, tiered tables and
every other billing dimension to that engine rather than to another rule restated
here. The reference request is cache-heavy because auto-routed traffic is.
* fix(spend): pick the baseline against the request that ran, not a stand-in for one
Ranking happened in the pre-routing hook, where the request has not executed yet, so
candidates were costed against a hard-coded reference workload: 20k prompt, 19k of it
cached, 1k out. Which candidate is dearest depends on that mix, so a pooled hardest
tier holding a deployment with non-proportional configured rates could be ranked for
a request nothing like the one served.
The mix is known on the spend path, so the ranking belongs there. The routing
decision now carries the tier's candidates rather than a winner already chosen, and
the baseline is resolved against the usage that actually happened. The reference
workload is gone; nothing here assumes a traffic shape any more.
The router is passed in rather than imported from `proxy_server` inside the
computation, so the savings stay a pure function of their arguments and the caller
owns where the router comes from. That also makes the spend path testable without a
running proxy, which the previous shape was not.
* refactor(spend): measure savings against one configured model, not a derived one
The counterfactual was derived per request: enumerate the hardest tier's
deployments, resolve each one's effective pricing, price them all, take the dearest.
That machinery produced a review finding per input shape it had not anticipated,
and every answer it gave was one an operator could have stated in a line of config.
So they state it. `litellm_settings.autorouter_savings_baseline_model` names the
model the traffic would have run on without a router, for every auto-router on the
proxy, and unset means the driver is off rather than a model nobody named being
guessed at. `savings_baseline.py` and its tests are deleted outright, along with the
tier enumeration, the candidate list on the routing decision, and the per-deployment
override that shadowed it.
Cache-state handling is untouched: the baseline is still priced on this request's own
read and write split, so a switch still pays for re-warming the cache and a first
turn still charges the write to both arms.
45 insertions against 482 deletions.
* refactor(router): compute the conversation shape once and pass it down
`_classify_and_route` re-derived it from the messages the hook had already resolved,
so an ordinary routed request walked the turn list twice for one boolean. The hook
computes it and hands it over, which is also where the affinity-hit path already got
it from.
Also moves `_get_llm_router` below the imports it sat among.
* fix(router): drop the dead conversation_continuing parameter off the hook
It was added to `async_pre_routing_hook` by mistake and immediately overwritten by
the value the hook computes, so it never did anything. It also widened a signature
every pre-routing strategy shares with the protocol in `types/router.py`, leaving
this one router diverged from `AutoRouter` and the interface for no reason.
Also records why an unreadable request counts as continuing: no messages is no
evidence a turn was served, so it pays the cache write and under-claims rather than
being handed a first turn's larger saving on nothing.
* fix(spend): charge a baseline its input rate for cache buckets it cannot price
A model with no cache_creation_input_token_cost, which is every OpenAI, Azure and Gemini entry, resolved that rate to 0.0 and carried the whole written prompt for free, so a first turn routed onto a cheaper model reported a loss. Same hole on cache reads. Those tokens are plain input on such a model, so they move into the text bucket.
* refactor(spend): build the daily upsert payloads in one shot
`common_data` and `update_data` were constructed and then appended to: `request_id`
conditionally for tag rows, `endpoint` unconditionally a few lines later. A dict that
grows after its literal cannot be reasoned about by reading the literal, which is the
whole point of building it at once.
The conditional key resolves to a spreadable value before either payload, so both are
single expressions and the tag branch appears once instead of twice.
Not wrapped in MappingProxyType, though it was suggested: these go straight to
prisma, whose query builder branches on `isinstance(value, dict)` to tell a nested
node from a scalar. A mappingproxy is a Mapping but not a dict, so it falls through
to the serializer and raises `TypeError: Type <class 'mappingproxy'> not
serializable` inside the batch upsert, where the surrounding except would log it and
leave the rollups silently unwritten.
* fix(spend): keep the one-shot upsert payloads under the type-discipline budget
Building both payloads as single literals traded a mutation for two dict literals,
and LIT002 counts construction rather than mutation, so the change the review asked
for is the one the gate charges for.
The empty branch is the avoidable half: it is the same value every time, so it moves
to a module constant built once instead of a literal per transaction, and it is a
read-only mapping so none of the call sites that spread it can fill it in later.
* feat(ui): expose an Auto-Router session affinity toggle
session_affinity on ComplexityRouterConfig defaults to True, and neither the
create form nor the edit modal ever emitted the key, so every auto-router built
in the UI silently pinned each session to its first turn's model for an hour
with no way to see or change that.
Adds an "Advanced: Session Affinity" switch to both surfaces, defaulted on to
match the backend field. Both paths now write the key explicitly instead of
falling through to the backend default, so a stored config states what the
router actually does. A stored config with the key absent hydrates as on, since
those routers are running with affinity enabled today; showing them as off would
report the opposite of reality and persist it on the next save.
* feat(complexity_router): default session affinity off and expose it in the UI
session_affinity defaulted to True and the Auto-Router UI never emitted the
key, so every router built there silently pinned each session to whatever model
its first turn classified into for an hour, refreshed on every hit. There was
no way to see that from the UI and no way to change it without hand-editing
config.yaml.
The default flips to False, so every turn is classified on its own merits and
lands on the cheapest adequate tier. Pinning is now opt-in.
The toggle added in the previous commit follows the field: it renders off, and
both the create tab and the edit modal keep writing the key explicitly, so a
stored config states what the router does instead of inheriting a default that
can move under it.
Behavior change for existing routers: those created before this have no
session_affinity key stored, so they pick up the new default and start
reclassifying every turn. That gives up the provider prompt cache the pin was
preserving, and a multi-turn session can now change model between turns. Set
session_affinity: true to keep the old behavior.
Two changes to the classifier's system role, both narrowing it rather than adding to it
classifier_tier_rubric let an operator replace the tier definitions. It shipped in
#35471 alongside the assistant-turn context window, but the two answer different halves
of the same report and only the context window was asked for. The override carried a
composed prompt, an overridable and a non-overridable half, a blank-is-unset rule, a
length-warning validator and a pair of dashboard controls. All of it goes
The rubric then closes on one of two lines, chosen by classifier_context_window_size.
At 0 no conversation is quoted, so the line is the original one, byte for byte: a
deployment that sends no context is told to classify the current message and nothing
else, which is what it could see all along. Above 0 the turns are quoted, and the
original line told the model to disregard them, which is how a request whose difficulty
was established in an earlier turn came back SIMPLE on the word "yes". There the line
instead says to classify the current message using the quoted turns as context, and to
rate what a short reply approves rather than the reply
The choice keys on the window and not on classifier_context_include_assistant_turns.
Whether the quoted turns are the user's alone or include the assistant's replies does
not change what the model needs told, and whose turn is whose is already on the turns.
Keying it on the assistant toggle would put the default deployment back on the original
line, which is the configuration the report was raised against
Folds in #35508, which built the window-dependent framing on top of the override this
removes; that PR is closed in favour of this one
The LLM classifier's context window carried user turns only, so a conversation
whose difficulty was stated by the model rather than by the user was classified
without it. Asked to find events, the assistant answers "here is the plan, it is
complex, should I execute?", the user answers "yes", and the router rates the
word "yes" and picks the cheapest tier
Two independent causes, so two changes that are each provable on their own
classifier_context_include_assistant_turns adds assistant turns to the window.
It is off by default because turning it on shifts tier decisions, and therefore
spend, for an already-deployed router, and because assistant text is net-new
egress to the classifier deployment. With it on, classifier_context_window_size
counts the last N turns across both roles, which is what makes the assistant's
own statement of difficulty land in the window
Assistant text reaches the classifier payload and nothing else. The window is
read only by _build_classifier_user_payload, while keyword_tier_rules, escalation
matching, the heuristic scorer and the semantic embedding all read the human ask
through _iter_human_asks_newest_first. Those are substring and vector matchers,
so an assistant echoing an escalation keyword back to a user would choose the
model, and the spend, with nobody having asked. Rather than widen the shared
iterator, _iter_context_turns_newest_first is separate and feeds the window
alone, which makes the boundary structural instead of a rule to remember
The rubric ended "Classify only the current message", and the classifier applied
it literally: a request whose difficulty was established earlier came back SIMPLE
because the message being rated was the word "yes". A context window the rubric
then tells the model to disregard buys nothing, so the wording now asks it to
rate the work the current message approves, judged in the conversation it
continues, while still forbidding it to rate a quoted section as if that section
were the request
classifier_tier_rubric lets an operator replace the tier definitions. The
trust-boundary paragraph is appended and cannot be replaced: it defends the
operator against their own callers, so an operator writing tiers without that
threat in mind would otherwise hand every keyholder the top tier by omission.
Blank reads as unset so an empty form field falls back rather than sending a
rubric with no tiers in it
Turns are labelled by role only when assistant turns can appear, so the prompt of
every deployment that never asked for this is unchanged byte for byte
The complexity router's classifier sub-call copies the parent request's metadata
verbatim, so its spend log row carries the caller's key, team and user and is
indistinguishable from traffic the caller actually sent. Nothing on the row says
otherwise: call_type is "acompletion" either way, model_group is overwritten to the
classifier's own model group so the row never looks auto-routed, and routing_decision
is absent exactly as it is on an ordinary request.
Record the fact the system already knows at call time. internal_call_origin is
declared on SpendLogsMetadata, which is the allowlist _get_spend_logs_metadata
projects onto, and stamped in _classifier_call_metadata; both classifier paths
already route through that one function and it feeds the metadata and
litellm_metadata buckets alike, so every request surface is covered at one site.
The key is reserved rather than caller-supplied, so it joins routing_decision in the
untrusted-metadata strip and a caller cannot label their own traffic as router
overhead.
The classifier call also inherited no session identity, so the router minted a fresh
trace id and the row landed in a session of its own. Forwarding the parent's session
puts it in the trace of the request that triggered it, which is where an operator
looks for what the routing cost.
The ComplexityRouter's LLM classifier saw only the last user message, so on a
multi-turn conversation it classified whatever happened to be last rather than
what the human actually asked, and a near-constant classifier input pinned a whole
session to one tier.
The blindness turned out to be narrower than first diagnosed, and the fix is
correspondingly smaller. Tool output was never the problem: on the Messages surface
it rides a user turn as tool_result content blocks, which are not text parts, so
flattening to `type == "text"` already dropped those turns; on chat completions it
arrives on a `tool` role the extractor never read. Both surfaces were already
handled before this change. What actually leaked through was the harness
`<system-reminder>` block, which arrives as ordinary text, survives flattening, and
became the current ask on any turn that carried one.
So reminders are stripped rather than used to reject the turn, because a harness
injects them alongside the live ask and not as a turn of their own; rejecting the
turn would lose the ask, and keeping the block would feed the classifier the
near-constant boilerplate that flattens tier selection in the first place. An
earlier revision of this change also pattern-matched serialized tool_result
payloads. That check only ever fired on a hand-serialized string neither request
surface produces, it was where every review finding in this PR lived, and it is
deleted here; the tests now pin the real shapes instead of the synthetic one they
were built on.
The classifier call is split into a system role carrying the rubric plus the
caller's own system prompt, which stays byte-stable across a session so a provider
can prompt-cache it, and a user role carrying the variable context: a bounded
window of prior user turns, a conversation-depth signal, and the current ask. The
caller's system prompt rides every turn, so task constraints are never dropped.
The depth signal measures content-parts messages too, since counting only string
content reported ~0 tokens for exactly the deep Messages-surface conversations that
most need an expensive tier, and it is omitted entirely on the prompt-only path
rather than asserting a false zero.
Prior turns are excluded by matching the current ask rather than by dropping the
newest turn positionally, because `aclassify` takes `prompt` and `messages`
separately and a caller may classify something other than the newest turn.
Truncated turns carry a marker so the classifier can tell a turn was clipped.
Only the LLM classifier's input changes. The heuristic scorer, keyword overrides,
escalation matching and semantic embedding still read the extracted current ask,
which is why that extraction has to yield one clean human-authored string: those
are substring and vector matchers, and an escalation keyword sitting inside a
reminder blob would otherwise trip a tier jump on its own.
Defaults keep single-turn classification equivalent to before. The prior-turn
window is on by default so existing LLM-classifier deployments actually get the
fix; the config field documents that those turns reach the classifier model, which
may be a different provider than the routed completion model, and that the call
already carries the current ask and the caller's system prompt in full.
Scoped to the ComplexityRouter; the semantic AutoRouter is not touched.
Auto-routed requests were indistinguishable from ordinary ones once logged:
the spend log recorded the requested model group and the resolved deployment,
but nothing about which tier was chosen or what chose it. That information
existed only inside verbose_router_logger f-strings, so answering "why did my
prompt land on the cheap model" required log access and a running proxy.
The complexity, quality, and adaptive pre-routing strategies now return a typed
StandardLoggingRoutingDecision on their PreRoutingHookResponse, and
Router.async_pre_routing_hook records it once for every attempt. Those three
previously side-channelled their own state through three different metadata
keys; the decision now travels on the hook contract itself, so the bucket is
resolved in one place, through get_or_create_metadata_bucket, which already
owns the question of which dict holds proxy-internal metadata and replaces a
non-dict value instead of skipping the write. Recording happens on every
attempt rather than only on a successful route: a fallback from an auto-router
group to a plain group re-enters the hook with the same request kwargs, and a
decision left behind there would attribute the first router's tier to the
deployment that actually served the retry. The log details drawer renders the
result as a Routing card between Request Details and Metrics; the card is
absent on rows that carry no decision, so ordinary and pre-upgrade rows are
unchanged.
Three defects surfaced while making the recorded cause truthful, each of which
would have persisted a wrong answer. The complexity router hardcoded
cause=complexity_scorer even when the LLM classifier decided, and its silent
fallback to the heuristic on classifier failure meant a row could claim an LLM
verdict the LLM never gave; the cause now reports the path that actually ran.
The keyword that triggered a tier rule was discarded before logging, as was
the escalation keyword. The 2-reasoning-marker override returned REASONING with
a score far below the REASONING boundary and no marker saying so, which reads
as a scoring bug to anyone comparing the two; it now emits a reasoning-override
signal, and the card labels those rows as an override instead of claiming the
score met a boundary. The LLM path no longer reports a synthetic score of 1.0,
and heuristic decisions carry a snapshot of the tier boundaries that mapped the
score, so a historical row stays interpretable after the boundaries change.
Signals name a matched term only when the caller's own message contains it.
Scoring still reads the system prompt, but a term matched solely there is
reported as a count, since signals reach a spend row the caller can read and
naming one would disclose a term from a prompt it cannot see.
routing_decision is stripped from caller-supplied metadata at ingress, so a
client cannot forge its own provenance.
The classifier and semantic-embedding sub-calls now capture proxy_server_request,
but neither forwarded the caller's turn_off_message_logging opt-out. A caller who
disabled message logging still had their prompt stored in the clear in these
internal sub-calls' spend-log rows, since should_redact_message_logging reads the
flag per-call and this internal call never inherited it.
The classifier read its metadata only from litellm_metadata, which the proxy
populates just for LITELLM_METADATA_ROUTES (/v1/messages, /v1/responses, ...);
/v1/chat/completions puts it under metadata, so the classifier call arrived
unattributed and _should_track_cost_callback dropped it, leaving no spend-log
row at all for the captured request body to show up in.
Also log response_format in the wire shape litellm actually sends
(type_to_response_format_param) instead of the bare pydantic JSON schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(complexity-router): user-triggered escalation keywords
Add an escalation_keywords config option to the complexity router so a user
can force a bump to the next-higher complexity tier by including a phrase in
their message (a stronger model, but not one they get to choose). Defaults to
['LITELLM ESCALATE'] when unset, case-sensitive so it only fires on the
deliberate shouted form; admins can override the list or set [] to disable.
Escalation applies across every routing path: heuristic/LLM classification,
literal and semantic keyword_tier_rules overrides, adaptive routing, and
session affinity (where it bumps relative to the pinned model and persists the
higher tier for the rest of the session). Capped at the highest configured
tier and skips unconfigured intermediate tiers.
Expose it in the Auto-Router v2 UI as an Escalation Keywords field wired into
the complexity_router_config payload.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity-router): validate escalation keywords and pin at tier ceiling
Strip blank/whitespace escalation keywords so an empty phrase can't match every message and escalate all traffic. Keep the exact pinned model when a session escalates at the highest configured tier instead of randomly hopping to a peer in a multi-model pool.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): tag-aware pre-routing strategy selection for shared model_name
Complexity/auto/adaptive/quality router registries were keyed by model_name
alone, so a second deployment sharing a model_name but carrying different tags
was rejected and every request used the first config. This made tag-based
routing to distinct provider configs behind one alias impossible, surfacing as
401 'Not allowed to access model due to tags configuration' for the second tag.
Each registry now holds a list of tag-scoped strategies and async_pre_routing_hook
selects the entry whose tags match the request before classification, falling
back to a default-tagged then first-registered entry. A repeat of the same
(model_name, tags) pair is still rejected.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover tag-scoped pre-routing strategy registry helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: re-trigger CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Pin a session's first-turn model for the rest of the session by default
instead of reclassifying every turn. Keeps multi-turn sessions on a single
model, preserving provider prompt caches and avoiding cross-model
conversation-history errors (e.g. Anthropic rejecting a thinking block
produced by a different model). Requests without a resolvable session_id are
unaffected. Set session_affinity: false to opt out.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): return empty dict from _classifier_call_metadata when metadata is absent
The LLM classifier reads request_kwargs.get("litellm_metadata"), but the proxy stores request metadata under "metadata", so this returned None. _classifier_call_metadata then passed None straight through to the classifier acompletion call, which assumes a dict and blows up with 'NoneType' object has no attribute 'update'; the router swallowed it and silently fell back to heuristic scoring, so the configured LLM classifier never ran. Returning an empty dict keeps the classifier call well-formed.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): cover complexity-router LLM classifier routes over the proxy
Add a live e2e regression for the complexity auto-router: a lexically simple but hard prompt ("Is P equal to NP?") is routed by the LLM classifier to the higher-tier anthropic backend, read back from the spend log's model. Before the metadata fix the classifier silently crashed and the router fell back to heuristic SIMPLE scoring on the openai backend, so this test fails pre-fix and passes post-fix.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): resolve auto-router routing plugins from proxy YAML config
Router(plugins=[...]) was Python-SDK constructor only, so proxy/YAML users
had no way to configure it, and the merged pipeline narrowed candidates
from the outer model alias rather than the auto-router's actual tier pool,
making it a no-op for auto_router deployments.
Add complexity_router_config.plugins (dotted-path strings resolved via
get_instance_fn, the same convention litellm_settings.callbacks uses) and
run the resolved plugins against ComplexityRouter's tier pool at every
model-pick site, so a policy plugin narrows what get_model_for_tier
actually returns instead of the outer alias list. adaptive=True with
plugins set now raises at config validation instead of silently ignoring
the plugins, since the bandit selector doesn't consume narrowed pools yet.
Also fixes a latent bug in Router._generate_model_id: it json.dumps every
litellm_params dict value to build a deployment hash id, which crashed
once a live plugin object could land inside complexity_router_config.
* fix(router): use stable class name, not object repr, in model-id json fallback
json.dumps(v, default=str) on a litellm_params dict containing a live
RoutingPlugin instance fell back to object.__repr__'s default
<module.Class object at 0x...>, embedding the instance's memory address.
_generate_model_id's hash (and therefore the deployment id) changed on
every process restart/hot-reload for any deployment with
complexity_router_config.plugins configured, defeating the function's own
"consistently generate the same id" contract and orphaning anything keyed
on that id across restarts (e.g. Redis-backed per-deployment state).
Use the plugin's fully-qualified class name instead, which is stable
across restarts.
* test(router): cover _json_default_stable_id for router_code_coverage gate
router_code_coverage.py's AST scanner requires every router.py function be
called by name somewhere in tests/, and flagged the new
_json_default_stable_id helper from the previous commit.
* fix(router): close two routing-plugin policy-bypass gaps flagged by Veria AI
Session-affinity pin shortcut: async_pre_routing_hook returned a session's
first-turn pinned model on every later turn without ever re-running it
through the plugin pipeline, so a policy plugin (e.g. a budget cap crossed
mid-session) was only enforced on turn one. Now the pin shortcut is
disabled whenever plugins are configured, so every turn re-runs
_classify_and_route (and therefore the plugins).
Plugin resolution validation: get_instance_fn accepts any dotted path and
returns whatever object it finds there, so a misconfigured
complexity_router_config.plugins entry passed proxy startup silently and
only surfaced as a confusing AttributeError on the first request that
reached the plugin pipeline. Extracted the resolution logic into
resolve_complexity_router_plugins() and added an isinstance(...,
RoutingPlugin) check that fails proxy startup immediately with a clear
error instead.
* fix(router): raise instead of falling back to default_model on empty plugin-narrowed tier
default_model was never checked against the configured plugins, so it
functioned as an unconditional escape hatch around whatever policy a
plugin enforces -- a tenant/budget plugin narrowing a tier to zero
candidates could still be bypassed by the fallback. Drop the fallback
entirely for this path; a plugin narrowing to zero is a policy decision,
not something to route around, matching the fail-closed behavior the
Router-level plugin pipeline already uses for the same situation.
Flagged by Veria AI on PR #33251.
* style: ruff format complexity_router.py
* style(proxy): use modern str | None instead of Optional[str] in resolve_complexity_router_plugins
* fix(router): stop default_model short-circuit from skipping plugins on no-user-message path
self.config.default_model or await self._pick_model_for_tier(...) -- Python's
`or` short-circuits on a truthy default_model, so _pick_model_for_tier (and
therefore the plugin pipeline) never ran at all for the no-user-message path
whenever default_model was configured. A tenant/budget plugin's decision was
silently bypassable this way even after the other two policy-bypass fixes,
since this call site had a different shape from the other three pick sites.
Removed the short-circuit; falls through to _pick_model_for_tier ->
get_model_for_tier, which already checks the MEDIUM tier before default_model
-- the same priority every other call site uses.
Flagged by Veria AI on PR #33251.
* fix(router): address Greptile findings on the plugin-bypass fixes
Preserve default_model-first priority in the no-user-message path when no
plugins are configured, instead of unconditionally flipping to the MEDIUM
tier -- the plugin-bypass fix must not silently change model selection for
the (much larger) population of users who don't use plugins at all. Gated
on self.config.plugins, matching the pattern already used elsewhere in
this PR, per CLAUDE.md's guidance against backwards-compat flags when a
plain conditional does the job.
Also close a gap in the plugin validation added earlier:
@runtime_checkable only checks that `run` exists as an attribute, not that
it's a coroutine function, so a synchronous `def run(self, context)`
passed isinstance(resolved_plugin, RoutingPlugin) at startup and only
failed at request time with a confusing TypeError. Added an
inspect.iscoroutinefunction check.
Both flagged by Greptile on PR #33251.
* feat(router): opt-in session affinity for complexity router
Complexity router reclassified every turn, which could flip the routed
model group mid-session and break provider-side prompt caching. Add a
session_affinity config flag: when a session_id is resolvable, pin the
model chosen on the first turn and reuse it for the rest of the session,
skipping reclassification. Pinned turns still stamp the adaptive
bandit's chosen-model metadata so reward feedback keeps working when
adaptive=True.
* fix(router): refresh session-affinity TTL on hit, scope pin by API key
Two issues from review: the TTL was only set on the first classification,
so an active session outliving session_affinity_ttl_seconds silently lost
its pin instead of refreshing as documented. And the cache key was scoped
only by session_id, which is client-supplied and unauthenticated, so two
different callers reusing the same session_id could poison each other's
routing pin. Refresh the TTL on every cache hit, and namespace the cache
key by the proxy-derived API key hash when available.
