GET /auto_router/benchmarks takes an optional api_key filter, applied in the
rollup aggregate on the primary key's leading column. Proxy admins get a
separate Auto-router usage tab on key detail pages with spend, baseline,
savings, tier routing, cache metrics and the existing router selector
The retrieve tool was injected whenever any hash=<24hex> string appeared in the
restored conversation, including protected rows and caller-authored text, so a
git SHA in a tool result registered a bogus hash and billed a useless retrieval
round trip on every later turn. The compression service reports the hashes it
actually stored in ccr_hashes; that field is now the only source, validated to
the service's own 12 to 24 hex grammar before it reaches the retrieve URL.
Assistant rows are no longer flattened to strings before compression: the
service protects assistant text blocks but has no gate for assistant strings,
so the model's own earlier tables came back as a schema line plus CSV.
Adds ccr_retrieval (default true) so operators on a marker-free sidecar can
turn the retrieval loop off entirely.
The chat and Responses bridges serialize tool_use blocks with model_dump(), so every
bridged /v1/messages response carried LiteLLM's internal provider_specific_fields key
(null, or a Gemini thought signature). Clients replay the block verbatim, and the next
turn that lands on a native Anthropic deployment (auto-router tier change, model swap)
is rejected with "tool_use.provider_specific_fields: Extra inputs are not permitted"
Strip the key from replayed content blocks at the single native Anthropic dispatch so
already-poisoned transcripts self-heal on every native provider, and stop emitting the
null on new responses. The bridges keep reading the signature for the Gemini round trip
Closes#19739
The API Keys route mounted the pre-App-Router UserDashboard component,
whose beforeunload handler cleared sessionStorage on every refresh of
the Virtual Keys page. That wiped the Playground chat history and model,
the logs live-tail preference, and everything else other pages keep in
session storage. The same component also re-decoded the login token,
re-fetched teams, and wrote cache entries nothing read.
ApiKeysDashboard now renders VirtualKeysTable and the Create Key button
directly, taking identity and role from useAuthorized like every other
page. Create Key is hidden for view-only roles, which the proxy already
rejects on /key/generate. The legacy component, its test, the fetch_teams
helper, and their grandfathered eslint suppressions are removed, and the
ProxySettings type moves to useProxySettings.
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks
`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:
- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.
Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.
* feat(guardrails): add inspect_embeddings toggle for AIM and Cato
* fix(guardrails): redact batched embedding input on anonymize
A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.
* fix(guardrails): reject partial embedding redactions
* fix(guardrails): avoid unnecessary batch type check
* style(tests): drop trailing blank line in cato guardrail tests
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject aim redactions with no text content
The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.
* fix(guardrails): validate all aim redaction paths
Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.
* fix(guardrails): preserve aim output redaction alignment
AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.
* test(guardrails): cover aim output anonymize alignment and malformed redactions
---------
Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
* fix(azure_sentinel): split batches under the 1MB ingestion cap and keep undelivered records queued
Azure Monitor rejects any Logs Ingestion body over 1MB with a 413. The Sentinel logger
posted the whole queue as one body and cleared it in a finally block, so an oversize
batch, a transient 5xx, or a failed token call dropped every queued record, and records
logged while a send was in flight were cleared with it. Both the standard and the audit
queue share the sender.
Move Datadog's proactive size split and 413 halving into a shared helper,
litellm/integrations/batch_utils.send_batch_with_413_split, and route Sentinel through it
with a 1MB size check. A lone record that still 413s is dropped, everything a transient
failure leaves undelivered goes back to the front of its queue, and the retry queue is
capped at max_queue_size so an unreachable workspace cannot grow memory without bound
* fix(azure_sentinel): retry undelivered records on the flush timer only
Requeued records made every later event cross the batch_size threshold, so a
down ingestion endpoint got one full-queue resend per request. Threshold sends
now go through flush_queue, so they take the flush lock instead of racing the
timer, and they stand down while records are awaiting retry.
A record that cannot be serialized raised out of the size probe and killed the
periodic flush task. The probe now runs inside the failure handling, so the
batch is split and only the record that cannot be serialized is dropped.
* fix(azure_sentinel): decide threshold sends under the flush lock
Concurrent callbacks all read logs_awaiting_retry before the first send
finished, so each one resent the whole queue once that send failed. The
flag and the batch_size threshold are now rechecked while holding the
flush lock, and each queue sends only itself instead of going through
flush_queue, which was retrying the other queue too.
* test(azure_sentinel): cover successful threshold waiters
* fix(azure_sentinel): preserve cancelled batches for retry
* fix(azure_sentinel): requeue only the undelivered part of a cancelled split
A batch over the ingestion cap goes out in pieces, so a cancellation partway
through requeued pieces the destination had already accepted and sent them a
second time on the next flush
The split helper now raises a cancellation carrying the records it never
delivered, and Azure Sentinel requeues those instead of the whole batch
* fix(azure_sentinel): drop batches a permanent rejection will never accept
A non-413 4xx from the ingestion endpoint or from the OAuth token call means the request
will fail the same way on every retry, so requeueing it held the batch, and every record
logged behind it, until the queue cap dropped them. Retryable statuses (5xx, 408, 429)
still keep the whole batch, and a shared classifier gives Datadog the same rule
The serialization probe now catches any exception, not just TypeError and ValueError,
because safe_dumps hands pydantic models to model_dump and can raise anything. It also
splits on record count, so a recovery flush sends batch_size records per request instead
of serializing the whole requeued queue to measure it
Both integrations re-raise a cancelled send as exactly asyncio.CancelledError. Python
3.12's asyncio.wait_for only translates the exact class into TimeoutError, so the
BatchSendCancelled subclass escaped the logging worker as an unhandled error
The awaiting-retry flag now follows the queue that survived the max_queue_size trim, so
a deployment with the cap at zero is not left waiting for a timer flush with nothing
queued to retry
* chore(logging): document mutable queue ownership
Annotate the queue detach and requeue constructions required by the logger's appendable queue contract so the type-discipline budget stays clean
* fix(datadog): preserve non-413 retry behavior
Keep Datadog's existing contract of requeuing every non-413 HTTP failure while Azure Sentinel applies its permanent-client-error policy through the shared splitter
* fix(batch_utils): requeue by default and let Sentinel opt into dropping
The shared splitter's default non-success handler is now requeue_after_http_error, the behavior Datadog had before the extraction, so a caller that omits the argument keeps its records. Azure Sentinel passes undelivered_after_http_error explicitly to drop permanent 4xx rejections
Also drops an explicit return None the strict ruff gate flags in the test helper
Format the model-picked id with %r so control characters in it cannot
break the log line. The regression test for the unlisted id keeps to
generic scoping wording
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.
The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.
migratedHref is renamed uiHref, the /ui base-path helper it always was.
Emulated file_search now warns when the model returns a vector_store_id that is
not one of the request's stores, naming the dropped id and the stores that were
searched instead. H16 asserts the warning is emitted exactly once.
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
The revert restored the dict annotation that #39121 had loosened to Mapping,
and the LIT001 ceiling has been ratcheted down since, so the gate rejected the
one reintroduced hit. Annotation only, no behavior change.
The emulated file_search handler searched whatever vector_store_id the
model returned, so a model steered to an id outside the request's
file_search tool reached a store the per-key vector store permission
check never saw. An id outside the request's stores now falls back to
those stores; an id that is one of them still narrows the search to it.
The Public Model Hub dialog in ModelHubTable was never opened (its open setter had no callers), but its See Page button navigated to /model_hub_table?key=<session key>. Delete the dialog, its state, the handler and the unused router import so the path cannot be revived.
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The previous cleanup only ran inside a caller's own except handler, so a
build that failed after its only caller had already been cancelled left
the failed task cached with nothing left to clear it. Move the cleanup
onto the task itself as a done-callback, which fires whether or not
anyone is still awaiting it, so the next request always gets a fresh
attempt instead of replaying the stale failure.
Adds a regression test for exactly that ordering (cancel the only
caller, let the build fail unobserved, then confirm the next request
builds successfully); it fails against the previous except-based
cleanup, which left the task cached.
test_load_state_from_db_handles_unknown_request_type compared the
WRITING cell after load against a cold-start value captured for
GENERAL. They happened to be equal for this fixture (the fast model's
empty strengths list makes every request type's prior identical), which
hid that the assertion was comparing the wrong baseline. Capture each
request type's own cold-start value instead.
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
_ensure_routelayer previously awaited asyncio.to_thread(...) directly
inside the lock. Under cancel_on_disconnect, cancelling that await
released the lock while the worker thread kept running, so a second
concurrent request would see no lock held and start a duplicate billed
build. Store the build as a task on self and have every caller await it
through asyncio.shield: cancelling one caller's wait no longer cancels
the build or lets another caller start a second one. A real build
failure (not merely a cancelled caller) clears the slot so the next
call retries fresh instead of replaying the same failure forever.
Also replaces the two tests that monkeypatched _build_routelayer (an
anti-pattern per this repo's conventions) with ones that instrument the
already-injected embedding router dependency instead, and adds a third
proving the cancellation race is actually closed.
- Shrink the fallback comments to one line each; the fuller rationale
was redundant per repo comment policy.
- Add test_pick_model_favors_the_cheaper_model_info_priced_deployment
and its hybrid-router counterpart, which exercise pick_model's actual
Thompson-sampling/scoring output instead of only asserting the
model_to_cost dict. Both are ordered so the expensive model wins
pick_best's insertion-order tie-break on the pre-fix code (proven by
reverting the production diff and rerunning), so they fail before the
fix and pass after it.
Both places that build an adaptive router's model_to_cost (the plain
auto_router/adaptive_router path in router.py, and the hybrid
adaptive-inside-complexity_router path in complexity_router.py) read
input_cost_per_token from litellm_params only. Custom pricing is
conventionally declared under model_info everywhere else in LiteLLM
(cost_calculator.py, add_deployment's litellm.model_cost registration),
so a deployment priced that way silently costs 0.0 in adaptive-router
scoring: every candidate ties on cost, the cost term contributes
nothing, and routing runs on quality alone with no warning.
Fall back to model_info at both call sites when litellm_params does not
declare a cost, matching how quality_router.py already sources cost.
litellm_params still wins when both are set.
Fixes#31481.
load_state_from_db assigned a DB row's (alpha, beta) straight into the
bandit cell, discarding the cold-start prior _init_cold_start_cells had
already put there. AdaptiveRouterUpdateQueue.flush_state_to_db only ever
persists accumulated deltas (its upsert creates a row with the raw delta
as the initial value, then increments it), never a full posterior, so a
cell whose first flush sees only one kind of signal persists a one-sided
row: e.g. alpha=1.0, beta=0.0. Loading that row as the whole cell hands
thompson_sample() a Beta(alpha, 0), and random.betavariate raises
'gammavariate: alpha and beta must be > 0.0' on every draw from that
cell from then on, surviving restarts since the bad row stays in place.
Fix: add the row on top of a freshly computed prior instead of replacing
the cell with it. Deltas are never negative, so both parameters stay
positive.
Fixes#35590. Fixes#29397.
AutoRouter's cold-start route layer construction (SemanticRouter with
auto_sync="local") ran directly on the event loop, doing at least one
synchronous embedding HTTP call inline behind a bare "if routelayer is
None" check with no lock, so it blocked the whole worker and let
concurrent cold-start requests each build a duplicate layer.
ComplexityRouter already solved this identically for its own semantic
keyword matching (_ensure_semantic_routelayer: a lock plus
asyncio.to_thread). Give AutoRouter the same treatment: extract the
build into _build_routelayer and gate it behind _ensure_routelayer's
double-checked async lock.
Fixes#33204.
* fix(ui): show indirectly granted and name-keyed MCP servers in the tool matrix
The MCP tool permission editor was fed the direct server list only, so a server a
principal reaches through an access group or a toolset never appeared in the matrix.
That single blind spot produced two opposite bugs depending on how a save handler
filtered mcp_tool_permissions: filtering by the selected servers deletes an indirect
server's allowlist, and because a missing entry means "no restriction from this
level", the principal silently gains every tool on it; not filtering leaves a stale
entry that keeps a removed access group's server reachable, since a server named
under mcp_tool_permissions is entitled on purpose.
The editor now resolves the selected access groups and toolsets to their servers and
renders them alongside the direct ones, badged with where the grant comes from, so an
admin can see and clear an inherited server's tools like any other. Resolution reuses
the data the selector already loads: access groups resolve from each server's
mcp_access_groups, toolsets from the toolset's own tool list. When that data cannot be
loaded the editor says so instead of rendering an empty list, because an absent
inherited server reads as "there are none". Servers named only by an
mcp_tool_permissions key are listed too, which is what makes a leftover entry
visible; the opt-out sentinel still renders nothing, since it short-circuits the
backend resolver to zero servers.
Opening the editor no longer applies the delete-blocked-by-default allowlist to an
inherited server. Writing an entry for one would narrow a grant the admin never
touched just by opening the form; direct servers keep that default.
Both components also matched on server_id alone, while the backend accepts a server
id, name or alias interchangeably. A grant or allowlist written by API or config with
a name rendered as a selected server with no tools under it, which reads as "this
server has no tools". Matching now covers all three identifiers, and an edit writes
back to the key the entry already uses rather than forking a second id-keyed entry.
The same mismatch could also put one server under several keys at once, its id and
its name for instance. The backend unions every key's list, so reading one key
understated what was in force and writing one key left the others granting. The
resolver now reports, per server, the key an edit keeps, the equivalent keys it
supersedes, and the union those keys allow; the card renders the union and every
write goes through one function that writes the kept key and drops the superseded
ones. A key that also names a DIFFERENT server, which happens when two servers share
a name, is never dropped, because dropping it would strip the neighbouring server's
restriction; the card names such a key and says its tools stay allowed until the
servers no longer share the name, so an admin is told rather than left to infer it
from an edit that bounces back.
A third divergence from the backend sat in the same matching. The backend resolves an
identifier with exact-id precedence: a string that is a registry server id names that
server and stops, and only a string that is no server's id falls back to name and alias,
which can name several. Matching all three fields at once meant a server merely named
after another server's id joined the matrix as if it had been selected, and because it
landed there as a directly selected server it also received the delete-blocked default
write on open. Since an mcp_tool_permissions key is itself a grant source, saving then
handed out a server nobody granted, with no admin gesture involved. Identifier
resolution now mirrors the backend's precedence, and a key is read as this server's only
when it resolves back to it, so an entry that belongs to the id's owner is neither read
into this server's allowlist nor overwritten by an edit made against it.
A toolset grant was also invisible to the tool matrix. The backend unions a toolset's
tools with whatever mcp_tool_permissions allows, so a toolset-only grant restricts the
server to that toolset's tools; the editor read the map alone, found no entry and
rendered every tool on the server as allowed. Deselecting one from that state wrote all
the others as a permission entry, and the union turned a revocation into a grant of
every tool the toolset never included. The resolved entry now carries the toolset's
tools, so the matrix opens on what is actually in force, the delete-blocked default is
withheld from a server a toolset restricts, and a write keeps out the tools only the
toolset accounts for so a grant that ends with the toolset does not become a standing
one. Those tools cannot be revoked from this screen at all, since the backend unions
them in; they render allowed and locked and the card says which of them a toolset holds
open and where to go to revoke them.
That guard originally covered only the keys an edit supersedes, on the assumption that
the key it keeps names one server. It does not when a shared key is a server's only
entry: it then becomes the key an edit writes, and writing it moves the other server's
allowlist too, which is the widening the guard exists to prevent. The key an edit writes
is now the first one naming this server and no other, falling back to the server's own
id, so a shared key is never written through and an edit against one card cannot reach
the server behind the other. Both cards say the shared key holds tools open, since
neither can revoke them.
No owner's save handler changes here. With the full effective set now available to
the editor, the key and team handlers can filter against it instead of guessing,
which makes the internal-user surface's unfiltered save redundant
Resolves LIT-4963
Resolves LIT-4958
* chore: drop tsbuildinfo churn from merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): satisfy dashboard lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep MCP tool allowlists for indirect grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep standing MCP grants on team save
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format TeamInfo and hoist inline object args
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep MCP tool allowlists for team servers granted indirectly (#35153)
* fix(ui): filter team MCP tool allowlists against the effective server set
Saving a team filtered mcp_tool_permissions down to the directly selected servers. A server reached
through an access group or a toolset is never in that list, so any save dropped its entry, including
a save that only changed the team alias. Because the resolver unions tool-permission keys into the
entitled server set and treats a missing entry as "no restriction from this level", the team kept
the server and lost the tool allowlist on it
Filtering on the direct list alone cannot get this right in either direction. Keeping every entry a
level did not directly select leaves a removed access group's server reachable through its own stale
entry, which breaks revocation. Dropping on deselection alone widens a server that an access group
still supplies
The save handler now resolves the effective server set with resolveEffectiveMcpServers and keeps an
entry only when something other than the entry itself still grants that server: a direct selection, a
selected access group, or a selected toolset. Unified access group ids are added when that selection
is untouched, since the loaded server list is then still accurate
When the server or toolset list cannot be resolved, every entry is kept and the admin is told the
allowlists were saved unchanged. Pruning on incomplete knowledge is the direction that silently
widens, so it only happens when the editor can show the server became unreachable. A failed lookup
and a changed access group selection are separate cases in a tagged union, so the notice names what
actually happened instead of describing the intentional one as a failure, and both hooks gate the
filter symmetrically so a save fired before toolsets settle cannot resolve against an empty toolset
list
Resolves LIT-4961
* fix(ui): resolve team MCP grants from access group metadata and refuse unsafe saves
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): resolve team access group grants from team info when the access group list is role-gated
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): match every selected access group by id instead of by count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): reload team access group grants at save time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep frontend lint budget within limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover a standing allowlist no group grant covers at load or save
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): keep MCP grant inputs in named variables for the lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): guard MCP default write on toolset load, keep create toolsets, fix flat view
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
