OpenAI rejects every reasoning.effort on chat-latest except medium. With supports_reasoning set and no declared levels the entry resolved to None, so /model_group/info and the dashboard effort pickers had nothing to narrow the offered levels with
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.
The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.
migratedHref is renamed uiHref, the /ui base-path helper it always was.
Every retrieve of a batch through the proxy shares one spend row, the batch id
plus the batch cost suffix, and spend log inserts skip duplicates. A poll that
landed while the batch was still validating or in progress wrote that row at
$0 and no later retrieve could overwrite it, and every completed retrieve after
the first added the cost to the key, team, and user counters again with no new
row to show for it.
The cost callback now writes nothing for a batch retrieve until the batch is
final, releasing the poll's budget reservation instead, and once it is final it
charges only when no spend row for that batch is queued for flush or already
stored. Batch cost rows are flushed to the database right away so a second
instance sees them, and the logger prices a batch only once it is final, which
also covers a failed batch that never produced an output file.
A one-character value in the provider secret bundle was masked too, which
turned every 1 in the run log into ***, including the pass numbers and the
gateway addresses, so the only public diagnostics were unreadable
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
A live cost map older than this release, or a proxy whose map fetch lags, could
strip `reasoning` from a model this release knows accepts it. The bundled map is
now the floor: any OpenAI entry it flags as reasoning keeps the param whatever
the live map says. Fine-tuned ids with an empty suffix (`ft:gpt-4o-2024-08-06:org::id`)
now resolve to their base entry instead of failing open, `chat-latest` carries
the flag, and the schema test keeps every codex, deep-research, and chat-latest
entry flagged. The none-effort check goes through a public wrapper so the
responses config stops importing a private helper.
_average_latency skipped integer samples in the sum while counting them in the denominator, which contradicted its own
Sequence[float | int] signature; it now averages every sample. Both success loggers in cost-based routing computed
response_ms / completion_tokens and threw the result away, so a chat response with zero completion tokens raised
ZeroDivisionError inside the handler. The proxy swallows and logs it, but the handler then skips that request's tpm and
rpm update, so cost-based routing undercounts the deployment's usage. The QA run for the latency fix hit it on real
gpt-5.5 traffic through /v1/chat/completions and /v1/messages
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
The revert restored the dict annotation that #39121 had loosened to Mapping,
and the LIT001 ceiling has been ratcheted down since, so the gate rejected the
one reintroduced hit. Annotation only, no behavior change.
The emulated file_search handler searched whatever vector_store_id the
model returned, so a model steered to an id outside the request's
file_search tool reached a store the per-key vector store permission
check never saw. An id outside the request's stores now falls back to
those stores; an id that is one of them still narrows the search to it.
The changed-tests workflow overrode the suite's `--reruns 1` with `--reruns 0`, so a
transport blip failed a pass that pytest.ini already scopes to network errors and
5xx responses. Pass 2 of run 33692484803 also went red 15s after a model write with
"no healthy deployments": the barrier only polled /v1/models through nginx, which
proves one gateway converged, and the next request rolled the other. The stack now
exports LITELLM_PROXY_REPLICA_URLS, the barrier polls every replica with the full
budget before settling, and up.sh refuses to boot without DD_API_KEY, since the
gateway config enables the datadog callback on every run
Latency-based routing averaged a deployment's cached samples with total / len(samples) and raised ZeroDivisionError
once an entry held none, which the proxy answered as a 500 for every later request on that model group. Cost-based
routing writes the same {model_group}_map entry with minute counters only, so a group used by both strategies hit this
on every latency-routed request. A deployment with no samples now counts as 0 latency, the same as one the router has
never seen
Resolves LIT-7053
The Public Model Hub dialog in ModelHubTable was never opened (its open setter had no callers), but its See Page button navigated to /model_hub_table?key=<session key>. Delete the dialog, its state, the handler and the unused router import so the path cannot be revived.
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
OCI Cohere restates the whole assistant text on the chunk that carries
the tool calls and again on the terminal chunk that carries chatHistory.
Only the terminal restatement was dropped, so a tool-calling turn streamed
the text twice. Treat a toolCalls-bearing chunk as a restatement too and
drop its text once deltas were already emitted.
Resolves LIT-6819
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The previous cleanup only ran inside a caller's own except handler, so a
build that failed after its only caller had already been cancelled left
the failed task cached with nothing left to clear it. Move the cleanup
onto the task itself as a done-callback, which fires whether or not
anyone is still awaiting it, so the next request always gets a fresh
attempt instead of replaying the stale failure.
Adds a regression test for exactly that ordering (cancel the only
caller, let the build fail unobserved, then confirm the next request
builds successfully); it fails against the previous except-based
cleanup, which left the task cached.
test_load_state_from_db_handles_unknown_request_type compared the
WRITING cell after load against a cold-start value captured for
GENERAL. They happened to be equal for this fixture (the fast model's
empty strengths list makes every request type's prior identical), which
hid that the assertion was comparing the wrong baseline. Capture each
request type's own cold-start value instead.
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
_ensure_routelayer previously awaited asyncio.to_thread(...) directly
inside the lock. Under cancel_on_disconnect, cancelling that await
released the lock while the worker thread kept running, so a second
concurrent request would see no lock held and start a duplicate billed
build. Store the build as a task on self and have every caller await it
through asyncio.shield: cancelling one caller's wait no longer cancels
the build or lets another caller start a second one. A real build
failure (not merely a cancelled caller) clears the slot so the next
call retries fresh instead of replaying the same failure forever.
Also replaces the two tests that monkeypatched _build_routelayer (an
anti-pattern per this repo's conventions) with ones that instrument the
already-injected embedding router dependency instead, and adds a third
proving the cancellation race is actually closed.
- Shrink the fallback comments to one line each; the fuller rationale
was redundant per repo comment policy.
- Add test_pick_model_favors_the_cheaper_model_info_priced_deployment
and its hybrid-router counterpart, which exercise pick_model's actual
Thompson-sampling/scoring output instead of only asserting the
model_to_cost dict. Both are ordered so the expensive model wins
pick_best's insertion-order tie-break on the pre-fix code (proven by
reverting the production diff and rerunning), so they fail before the
fix and pass after it.
Both places that build an adaptive router's model_to_cost (the plain
auto_router/adaptive_router path in router.py, and the hybrid
adaptive-inside-complexity_router path in complexity_router.py) read
input_cost_per_token from litellm_params only. Custom pricing is
conventionally declared under model_info everywhere else in LiteLLM
(cost_calculator.py, add_deployment's litellm.model_cost registration),
so a deployment priced that way silently costs 0.0 in adaptive-router
scoring: every candidate ties on cost, the cost term contributes
nothing, and routing runs on quality alone with no warning.
Fall back to model_info at both call sites when litellm_params does not
declare a cost, matching how quality_router.py already sources cost.
litellm_params still wins when both are set.
Fixes#31481.
load_state_from_db assigned a DB row's (alpha, beta) straight into the
bandit cell, discarding the cold-start prior _init_cold_start_cells had
already put there. AdaptiveRouterUpdateQueue.flush_state_to_db only ever
persists accumulated deltas (its upsert creates a row with the raw delta
as the initial value, then increments it), never a full posterior, so a
cell whose first flush sees only one kind of signal persists a one-sided
row: e.g. alpha=1.0, beta=0.0. Loading that row as the whole cell hands
thompson_sample() a Beta(alpha, 0), and random.betavariate raises
'gammavariate: alpha and beta must be > 0.0' on every draw from that
cell from then on, surviving restarts since the bad row stays in place.
Fix: add the row on top of a freshly computed prior instead of replacing
the cell with it. Deltas are never negative, so both parameters stay
positive.
Fixes#35590. Fixes#29397.
AutoRouter's cold-start route layer construction (SemanticRouter with
auto_sync="local") ran directly on the event loop, doing at least one
synchronous embedding HTTP call inline behind a bare "if routelayer is
None" check with no lock, so it blocked the whole worker and let
concurrent cold-start requests each build a duplicate layer.
ComplexityRouter already solved this identically for its own semantic
keyword matching (_ensure_semantic_routelayer: a lock plus
asyncio.to_thread). Give AutoRouter the same treatment: extract the
build into _build_routelayer and gate it behind _ensure_routelayer's
double-checked async lock.
Fixes#33204.
* fix(ui): show indirectly granted and name-keyed MCP servers in the tool matrix
The MCP tool permission editor was fed the direct server list only, so a server a
principal reaches through an access group or a toolset never appeared in the matrix.
That single blind spot produced two opposite bugs depending on how a save handler
filtered mcp_tool_permissions: filtering by the selected servers deletes an indirect
server's allowlist, and because a missing entry means "no restriction from this
level", the principal silently gains every tool on it; not filtering leaves a stale
entry that keeps a removed access group's server reachable, since a server named
under mcp_tool_permissions is entitled on purpose.
The editor now resolves the selected access groups and toolsets to their servers and
renders them alongside the direct ones, badged with where the grant comes from, so an
admin can see and clear an inherited server's tools like any other. Resolution reuses
the data the selector already loads: access groups resolve from each server's
mcp_access_groups, toolsets from the toolset's own tool list. When that data cannot be
loaded the editor says so instead of rendering an empty list, because an absent
inherited server reads as "there are none". Servers named only by an
mcp_tool_permissions key are listed too, which is what makes a leftover entry
visible; the opt-out sentinel still renders nothing, since it short-circuits the
backend resolver to zero servers.
Opening the editor no longer applies the delete-blocked-by-default allowlist to an
inherited server. Writing an entry for one would narrow a grant the admin never
touched just by opening the form; direct servers keep that default.
Both components also matched on server_id alone, while the backend accepts a server
id, name or alias interchangeably. A grant or allowlist written by API or config with
a name rendered as a selected server with no tools under it, which reads as "this
server has no tools". Matching now covers all three identifiers, and an edit writes
back to the key the entry already uses rather than forking a second id-keyed entry.
The same mismatch could also put one server under several keys at once, its id and
its name for instance. The backend unions every key's list, so reading one key
understated what was in force and writing one key left the others granting. The
resolver now reports, per server, the key an edit keeps, the equivalent keys it
supersedes, and the union those keys allow; the card renders the union and every
write goes through one function that writes the kept key and drops the superseded
ones. A key that also names a DIFFERENT server, which happens when two servers share
a name, is never dropped, because dropping it would strip the neighbouring server's
restriction; the card names such a key and says its tools stay allowed until the
servers no longer share the name, so an admin is told rather than left to infer it
from an edit that bounces back.
A third divergence from the backend sat in the same matching. The backend resolves an
identifier with exact-id precedence: a string that is a registry server id names that
server and stops, and only a string that is no server's id falls back to name and alias,
which can name several. Matching all three fields at once meant a server merely named
after another server's id joined the matrix as if it had been selected, and because it
landed there as a directly selected server it also received the delete-blocked default
write on open. Since an mcp_tool_permissions key is itself a grant source, saving then
handed out a server nobody granted, with no admin gesture involved. Identifier
resolution now mirrors the backend's precedence, and a key is read as this server's only
when it resolves back to it, so an entry that belongs to the id's owner is neither read
into this server's allowlist nor overwritten by an edit made against it.
A toolset grant was also invisible to the tool matrix. The backend unions a toolset's
tools with whatever mcp_tool_permissions allows, so a toolset-only grant restricts the
server to that toolset's tools; the editor read the map alone, found no entry and
rendered every tool on the server as allowed. Deselecting one from that state wrote all
the others as a permission entry, and the union turned a revocation into a grant of
every tool the toolset never included. The resolved entry now carries the toolset's
tools, so the matrix opens on what is actually in force, the delete-blocked default is
withheld from a server a toolset restricts, and a write keeps out the tools only the
toolset accounts for so a grant that ends with the toolset does not become a standing
one. Those tools cannot be revoked from this screen at all, since the backend unions
them in; they render allowed and locked and the card says which of them a toolset holds
open and where to go to revoke them.
That guard originally covered only the keys an edit supersedes, on the assumption that
the key it keeps names one server. It does not when a shared key is a server's only
entry: it then becomes the key an edit writes, and writing it moves the other server's
allowlist too, which is the widening the guard exists to prevent. The key an edit writes
is now the first one naming this server and no other, falling back to the server's own
id, so a shared key is never written through and an edit against one card cannot reach
the server behind the other. Both cards say the shared key holds tools open, since
neither can revoke them.
No owner's save handler changes here. With the full effective set now available to
the editor, the key and team handlers can filter against it instead of guessing,
which makes the internal-user surface's unfiltered save redundant
Resolves LIT-4963
Resolves LIT-4958
* chore: drop tsbuildinfo churn from merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): satisfy dashboard lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep MCP tool allowlists for indirect grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep standing MCP grants on team save
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format TeamInfo and hoist inline object args
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep MCP tool allowlists for team servers granted indirectly (#35153)
* fix(ui): filter team MCP tool allowlists against the effective server set
Saving a team filtered mcp_tool_permissions down to the directly selected servers. A server reached
through an access group or a toolset is never in that list, so any save dropped its entry, including
a save that only changed the team alias. Because the resolver unions tool-permission keys into the
entitled server set and treats a missing entry as "no restriction from this level", the team kept
the server and lost the tool allowlist on it
Filtering on the direct list alone cannot get this right in either direction. Keeping every entry a
level did not directly select leaves a removed access group's server reachable through its own stale
entry, which breaks revocation. Dropping on deselection alone widens a server that an access group
still supplies
The save handler now resolves the effective server set with resolveEffectiveMcpServers and keeps an
entry only when something other than the entry itself still grants that server: a direct selection, a
selected access group, or a selected toolset. Unified access group ids are added when that selection
is untouched, since the loaded server list is then still accurate
When the server or toolset list cannot be resolved, every entry is kept and the admin is told the
allowlists were saved unchanged. Pruning on incomplete knowledge is the direction that silently
widens, so it only happens when the editor can show the server became unreachable. A failed lookup
and a changed access group selection are separate cases in a tagged union, so the notice names what
actually happened instead of describing the intentional one as a failure, and both hooks gate the
filter symmetrically so a save fired before toolsets settle cannot resolve against an empty toolset
list
Resolves LIT-4961
* fix(ui): resolve team MCP grants from access group metadata and refuse unsafe saves
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): resolve team access group grants from team info when the access group list is role-gated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): match every selected access group by id instead of by count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): reload team access group grants at save time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep frontend lint budget within limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover a standing allowlist no group grant covers at load or save
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): keep MCP grant inputs in named variables for the lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): guard MCP default write on toolset load, keep create toolsets, fix flat view
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.
The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.
Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
