* fix(model-management): honor an explicit null as a clear on model update
PATCH /model/{model_id}/update merged the patch with exclude_none and then
popped explicit nulls only for the mirrored pricing fields, so a null sent for
max_input_tokens, mode, supports_vision or any other key was dropped and a value
pinned by an earlier save could never be removed.
The route now follows JSON Merge Patch over both blobs: a key absent from the
body is unchanged, a key sent as null is removed from the stored row, and a key
sent with a value is set. Ownership and identity keys keep ignoring a null, as
do the fields the stored models require, since clearing one writes a row no
reload can rebuild. Mirrored pricing keys still clear from both blobs.
Clearing a price also needed the router to stop merging a deployment's cost-map
entry onto its previous registration, which left the old rate in place and kept
billing at a price the deployment no longer carried.
Adds a create, read, partial-update, clear, enforce, delete lifecycle e2e that
reads back on every replica, and a harness helper for that read-back.
* fix(router): keep a deployment id that names a real model from evicting its catalog entry
Deployments are keyed into litellm.model_cost alongside the built-in catalog, so
evicting a deployment's stale entry by id could take a real model's entry with it:
registering a deployment whose model_info.id is "gpt-4o" stripped that model's
pricing, context window and capability flags process-wide, for every other
deployment of it, until the next price-map reload.
Only evict an entry this registration owns. A colliding id keeps the previous
merge, which pollutes the catalog entry rather than emptying it.
Also pins the Admin UI round trip: the model edit form echoes the whole /model/info
row back on save, and that read reports every key the deployment never stored as an
explicit null, so the clear path has to leave those keys alone.
* fix(router): decide cost-map eviction by what this registrar created
The previous guard read a catalog entry off `litellm_provider`, so a deployment
that declares its own provider in model_info was treated as one and kept billing
at a price it no longer carried. It also only held for a single registration: a
second one under a colliding id saw the id the first merge left behind and
evicted the catalog entry anyway.
Track the cost-map keys this registrar creates instead. A key it created is
evicted before re-registration; one it did not is left to merge, which is what a
deployment id colliding with a catalog model name needs.
Also folds the required-fields comment into the docstring that already gives the
reason.
* fix(router): release a deployment's cost-map key when it is deleted
The ownership ledger only grew. A deleted deployment kept its claim, so if a
later catalog refresh started publishing a model under that same name, the next
registration would treat the catalog entry as the deployment's own and evict it.
Deleting a deployment now gives the key back, which also stops the ledger
growing for the life of the process.
* fix(router): hold a cost-map key while another live router still serves it
The claim is process-wide but the release was per-deletion, so with two routers
serving one deployment id, the first deletion put the survivor back on merging
and the price it had just cleared would keep billing.
Release the key only once no live router still serves that id.
* fix(router): register a router in the live set when it gains a deployment
_live_routers was only joined when a router was constructed with a model_list,
but a router built empty is populated through add_deployment, and the empty
branch exists for exactly that. Such a router was invisible to the live-router
scan, so deleting the deployment from another router released the shared
cost-map key while it was still serving that id.
Joining the set where a deployment enters the list covers every path, and it
also lets a price reload rebuild what a dynamically built router serves.
* fix(e2e): read the stored model row from the control plane, not each gateway
The lifecycle suite polled /model/info on every URL in PROXY_REPLICA_URLS. Those
URLs are the stack's gateways, and gateway/routes/allowlist.py trims them to the
LLM data-plane surface, so /model/info answers only on the backend and 404s on
every replica. All five tests failed at their first read-back in CI while passing
against a monolith, where one process serves both planes.
The stored row has one answer behind it, so it is read through the shared
transport, which routes control-plane paths to the backend. What every gateway
must agree on is which models it serves, so the create and delete steps poll
/v1/models per replica instead, a route the gateway does serve.
read_back_everywhere now rejects a control-plane path outright rather than
timing out on it.
Two things surfaced behind that. /public/ was missing from the transport's
control-plane prefixes, so model_cost_map() was routed to a gateway and 404'd,
and the billing steps needed a data-plane wait: a PATCH lands on the backend and
each gateway picks it up on its own config reload, measured here at 12-24s, so
they now drive calls until the new rate reaches the spend row and let the
deadline fail them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1S92J8gSxxKVe1JBzxWBF
* test(models): keep polling outcomes immutable and document shared ownership
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Turning the tier off, or switching to a classifier that cannot emit it, dropped
the flag and the pool but left plan_mode_min_tier naming a tier that is no longer
active. The backend rejects that on save, and the switch is disabled after a
classifier change, so the operator had no way to clear it.
Both paths now release the floor when it points at the cleared tier. An orphaned
keyword rule is left alone on purpose: getKeywordTierRulesError already names it
at the save gate, which is how a removed custom tier behaves.
Upstream's edit_auto_router_modal.tsx sits at 799 countable lines, one under the
800 cap, so this PR's 11 added lines put the merge result over. The built-in tier
hydration moves next to its sibling hydrators in build_complexity_router_config,
which is where hydrateCustomTierSet and hydrateTierLabels already live.
Upstream grew ClassificationMethodConfig.tsx to 783 lines, so the 14 lines this
PR added there pushed the merge result past the 800-line max-lines cap. The
helper is standalone logic with its own unit tests, so it moves out rather than
the cap moving up.
* feat(otel): add http/json export protocol for OTel v2 traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/json was accepted but routed to the protobuf
OTLP/HTTP exporter, so collectors that only decode JSON rejected every batch.
Route it to an OTLP/JSON span exporter that reuses the SDK HTTP transport and
expose the protocol as a select field on the OpenTelemetry callback in the
admin UI.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): walk the fixed OTLP shape instead of recursing when hex-encoding ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): map stored callback variables onto their form fields when editing a callback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A client behind an auto-router sends one max_tokens for every tier, so a value
sized for the smallest tier starves a bigger tier's thinking budget and a value
sized for the biggest is rejected by the smallest. After the complexity router
picks a tier, its per-tier litellm_params now carry max_tokens set to the
smallest max_output_tokens across that tier model's deployments (model_info,
then the cost map), applied the same way a per-tier reasoning_effort already
is, on every routing exit including plan mode, the empty-ask default and the
classifier fallback. The router seam collapses whichever ceiling alias a tier
carries onto the surface's own name, so one tier max_tokens reaches chat,
/v1/messages and /v1/responses alike, drops the caller's other carriers of the
same setting before the merge, and stamps the caller's original once so a
fallback into a group no tier owns gets it back instead of a ceiling sized for
the tier that failed.
Proxy-level reservations were sized from the caller's cap before routing, so a
raised cap left them short. Both owners now re-validate at the deployment hook:
the v3 limiter tops up its combined-TPM and project-OTPM reservations to the
final cap or writes the admitted cap back, and the budget limiter re-estimates
on the chosen deployment and grows the reservation or writes the admitted cap
back. An auto-router alias also reserves budget at its priciest tier model now
instead of pricing to zero.
An explicit per-tier max_tokens, max_completion_tokens or max_output_tokens
still wins, and max_tokens_from_tier_model: false forwards the caller's value
unchanged.
The Advanced scoring editor now lists built-in and custom dimensions together. Editing any weight holds it and rescales the others proportionally so the vector totals 1.00, and Save stores those explicit values. The backend scores exactly what is stored, with no runtime normalization, so routers nobody edits keep their weights.
CustomDimension gains an opt-in scoring_mode. match_count scores 0, 0.5 or 1 by distinct matcher hits; the default stays binary. The tuning fingerprint omits a binary scoring_mode, so routers written before this change keep their recorded baseline and the upgrade does not consume the free heuristic-v1 tuning slot.
Cuts the explanatory comments and docstrings added here down to one line each, or
removes them where the code already says it. Restores the four pre-existing
docstrings this PR had reworded to their original text; the one remaining edit to
existing text is TierDefinition.description, whose hardcoded tier list would
otherwise misstate that a tier named NON_REASONING may also omit its description.
Review flagged the added comments as over-explaining. Cut the call-site comment
that restated the helper's own docstring, and shortened the rest to the fact the
code cannot state itself: why the constant excludes the tier, why the flag is
cleared on a classifier change, and why the edit modal reads both keys back.
The switch adds a row at the top of the tier list, so sitting below Reasoning
put the control and the thing it changes at opposite ends of the card. It now
heads the list, with a separator between it and the first row.
Three review findings, all in the dashboard.
Switching off the LLM classifier left the toggle checked but disabled, so the
flag could not be cleared and every save was refused by the backend. The
classifier-change handler now drops the flag and the tier's pool the same way it
already drops the other classifier-specific keys.
builtInTierInfo resolved rows against the four-tier order, so the new row
rendered with no description, no examples and no rename field. It now resolves
against every built-in tier, and the duplicate BUILT_IN_TIER_ORDER constant is
gone in favour of the one in tier_rows.
The preset schema widening is reverted. It was speculative, no published catalog
carries the tier, and prefill would have discarded it while the route-wide null
exclusion changed the endpoint's passthrough contract for every other field.
Also splits NonReasoningTierToggle and TierConfigIntro into their own files to
get ComplexityRouterConfig.tsx back under the max-lines limit, and applies ruff
format to the two backend files CI flagged.
Agent harnesses send a lot of operational turns that relay or reformat tool
output rather than reason about it, and the cheapest built-in tier was SIMPLE.
NON_REASONING adds a rung below it, behind enable_non_reasoning_tier so an
already-deployed router cannot move.
The toggle is what keeps it safe. The tier set feeds the classifier rubric, the
response-format enum, the escalation ladder and the savings baseline, so a
default-on fifth tier would have changed what every existing router sends and
where its traffic lands. Off, the ladder, rubric, wire labels and baseline are
byte-identical to before. On, the rung is added at index 0, escalation walks up
out of it, and it can never win the savings baseline.
It requires an llm or custom classifier and a model of its own: the v1 score
ladder has no rung below simple_medium and the v2 artifact is trained on four
classes, so the heuristic scorers cannot produce the tier and a router that
enabled it there would pay for a bullet nothing reaches.
The dashboard follows the same flag, and the edit modal now reads the tier back
from the stored config rather than assuming four keys, since it rewrites tiers
wholesale on save and would otherwise delete a hand-written tier on any edit.
An aggregate gateway DCR authorize whose RFC 8707 resource resolves to exactly one
gateway-managed oauth2 server sealed that server into the flow and then sent the browser to
the generic connect grid anyway, so the user had to find the server the client had already
named and click Connect.
The connect URL now carries only the flow handle. GET /authorize/flow classifies the sealed
flow as unscoped, interactive, M2M, or stale, and returns the matching state to the page.
Interactive flows require a live per-user vendor credential before minting and do not burn the
flow on an early submit. M2M flows use the gateway's configured service credential and finish
without an interactive OAuth trip. Stale flows fail closed instead of becoming unscoped.
The existing explicit Finish action and a new Cancel path preserve deliberate user intent.
* fix(proxy): log disable_budget_reservation notice once at config load
The disabled-budget-reservation reminder fired as a WARNING inside request
authentication, so every authenticated request on a proxy that deliberately
set the flag produced one warning line. The notice now runs once per worker
when general_settings loads, at INFO, and the request path only skips the
reservation. Reservation skipping and read-time budget checks are unchanged
* fix(proxy): keep budget notice sentinel with constants
* fix(proxy): expose shared budget notice state
The revision an operator checks is now the git blob id of the exact bytes the process
loaded, the same id git rev-parse <commit>:model_prices_and_context_window.json prints,
so it is always present, never goes stale between bot writes, and needs no stamp in the
JSON that every PR touching the file would have to regenerate. The _metadata block, the
generated_at field, the schema and guard changes, and the bot stamping are dropped
The drop_params validator collapsed every string it did not recognize to None. A pre-fix DB row holds the flag as ciphertext, so a partial PATCH rebuilt the deployment without it and dropped the key from the stored row, and /model/new turned an os.environ/ reference into nothing before the loader could resolve it. The validator now returns the raw value when it is not a boolean flag, the field admits strings the way timeout already does, and the flag set follows pydantic's lax bool parsing instead of a hand-rolled true/false pair
The cost map JSON now carries a top-level `_metadata` block with `generated_at` and `source_revision`, written by the two bot writers only when model data changed. The loader pops it before the map becomes `litellm.model_cost`, records it next to the fetch ETag, and `/reload/model_cost_map`, `/model/cost_map/source`, and the reload schedule status return it. The Price Data Reload card shows the stamp, the ETag, and when the pod loaded the map. The schema and the cost map guard treat `_metadata` as a non-model root key
The rate fields reported base cost-map prices while the cost lines were billed at the token tier, off-peak window and regional multipliers the calculator picks for the request, so a line did not always equal tokens times its reported rate. get_billed_token_rates now resolves the rates once, the token-type breakdown and the endpoint both read from it, and a tiered-model test asserts every line equals its token count times the rate reported next to it
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
Take staging's test_bedrock_knowledgebase_hook.py, which drops the duplicate
embedding_executor parameter that turned the lint check red, and make the two
cross-module helpers this branch added public (raise_denied_scoped_mcp_access
and routes_through_gateway) so the private-usage budget stays at its base count
