Pass-through endpoints configured with auth=false reach the cost-tracking callback with no key/user/team/end-user, so _should_track_cost_callback returned False and the spend-log write was skipped, leaving the request out of request/usage logs. Track pass-through call types even when unauthenticated so the SpendLog row is still written.
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): rename OldTeams component file to Teams
* fix: show and allow editing team model aliases after team creation
* fix(ui): mark team model_aliases as nullable to match the prisma schema
* fix(guardrails): walk custom_tool_call_output items in _content_utils
* Change _OUTPUT_ITEM_TYPES to Frozenset type
* fix(guardrails): use builtin frozenset generic for _OUTPUT_ITEM_TYPES annotation
Frozenset is not a defined name (typing exports FrozenSet, the builtin is
frozenset), so module import raised NameError and broke every proxy test
suite. The builtin generic is valid on the supported python floor (3.10)
and keeps the UP006 ruff-strict budget at its ceiling, which the typing
alias would exceed
* feat(batches): track cost for unmanaged Bedrock batches, generalize the flag
CheckBatchCost skipped Bedrock batches whose unified_object_id is a raw
model-invocation-job ARN, the same root cause previously fixed for
unmanaged Vertex batches. Bedrock batches embed the model name in their
s3:// input file name instead (litellm-bedrock-files-{model}-{uuid}.jsonl),
so the same routing mechanism now derives the model from that layout and
matches it to a configured bedrock deployment.
track_unmanaged_vertex_batch_cost is renamed to track_unmanaged_batch_cost
since two providers now share this mechanism.
* fix(batches): parse Bedrock batch output and price with deployment model name
Bedrock model-invocation-job results use modelOutput/error rows and short
internal model ids that are not in the cost map, so unmanaged batch cost
tracking logged tokens but $0 spend. Use deployment model name for pricing
and add regression tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
When a /responses request uses a hosted MCP tool (server_url: litellm_proxy/<label>)
with store=true and the model calls a tool, the gateway auto-executes the tool and
streams one logical response stitched from several upstream responses: an interim
response whose only output is the function_call, then the post-tool answer
B1 (correctness): every streamed event was pinned to the first round's response id,
i.e. the interim response that carries the function_call but no tool output. The
client then continued the next turn from that dangling response and the provider
rejected it with "No tool output found for function call <id>", which on the
streaming path surfaced as a silent empty completion. The fix adopts each
auto-execute round's own response id (the cached id is reset when a follow-up round
starts) so the client continues from the final round, whose stored input chain
includes the function_call_output
B2 (robustness): initial and follow-up call failures were swallowed; the stream
emitted the mcp_list_tools discovery events and then closed with HTTP 200 and no
output and no error. The fix stashes the failure, makes the initial call eagerly in
aresponses_api_with_mcp so a pre-stream failure re-raises as a real 4xx before any
SSE bytes are written, and emits a terminal error event when a follow-up call fails
mid-stream
Adds regression tests covering continuation exposing the final round's response id
rather than the interim tool-call id, a follow-up failure emitting a terminal error
event, and an initial-call failure being stashed for eager re-raise
* feat(router): soft-floor adaptive mode for complexity router
Let complexity_router_config.adaptive=true Thompson-sample across the
union of tier pools with a tier-distance penalty, and wire the existing
adaptive post-call bandit so mis-tiered requests can still recover.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): reattach adaptive hooks for hybrid complexity
Finalize was wiping every AdaptiveRouterPostCallHook and only
re-registering standalone auto_router/adaptive_router deployments,
so complexity adaptive=true never received bandit updates.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(router): drop unnecessary hybrid docstrings
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): attribute adaptive feedback
Credit user reactions to the model that produced the previous response while keeping current-response signals on the serving model
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): tune hybrid cold defaults
Use the cost-weighted policy that beat equal-pool complexity in the full bakeoff, and make the committed harness compare identical tier pools
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): preserve hybrid cold quality floor
Sample only unobserved models in the classified tier until feedback exists, then apply adaptive scoring without mis-penalizing models shared across tiers
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): bound feedback context cache
Cap retained session feedback so unique session IDs cannot exhaust router memory
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): preserve exhaustion signals
Include tool-result exhaustion in adaptive feedback and clear strict lint regressions blocking CI
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(router): remove stale owner cache
Remove obsolete attribution state, tighten the embedded router type, and keep the test diff focused on adaptive behavior
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(router): centralize hook cleanup
Use the callback manager to discover and remove adaptive hooks across every registered callback list
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(router): add Router(plugins=[...]) routing-plugin pipeline
Runs a sequence of user-supplied plugins before the routing decision is
made. Each plugin reads/mutates a RoutingContext (messages, candidate
models, metadata, signals); the narrowed candidate list is enforced when
picking a deployment, raising rather than silently falling back if a
plugin narrows to zero candidates.
Prototype for the routing-plugin pipeline discussed in #32168.
* fix(router): use ruff-modern typing, add raw/structured messages to RoutingContext
- Use dict/list/X|None instead of Dict/List/Optional in new code, staying
within the ruff strict-rule budget ratchet
- Extract the guardrail-translation message normalization ComplexityRouter
already had into a shared resolve_structured_messages() helper
(litellm_core_utils/prompt_templates/factory.py), reused by
ComplexityRouter and the new routing-plugin pipeline instead of
duplicating it
- RoutingContext now exposes both raw_messages (as received) and
structured_messages (normalized across chat completions / Anthropic
messages / Responses API), mirroring CustomGuardrail.apply_guardrail's
pattern, per review feedback on #32972
- Add direct unit tests for _run_routing_plugins and
_filter_by_routing_plugin_candidates (router_code_coverage gate requires
every router.py function be called by name somewhere in tests/)
* fix(test): rename to test_router_routing_plugins.py
router_code_coverage.py's AST scanner only inspects test files whose
filename contains the substring "router" -- test_routing_plugins.py
doesn't match (routing != router), so it silently skipped this file
and flagged _run_routing_plugins/_filter_by_routing_plugin_candidates
as untested despite the direct unit tests added for them.
* fix(router): fail closed when plugins are configured but the resolved
routing path can't run them
Router.completion() (and other sync entry points) resolves deployments
via the synchronous get_available_deployment(), which never runs
async_pre_routing_hook and therefore never runs the routing-plugin
pipeline. async_get_available_deployment() itself falls back to that
same synchronous method for routing strategies without an async-native
selector (e.g. legacy "usage-based-routing" v1). Both paths would let a
policy plugin (e.g. a deny-all rule) be silently bypassed.
Raise instead of silently proceeding when self.routing_plugins is
configured and the sync path is reached, since applying the pipeline to
every selector path is a larger change out of scope for this PR.
Per review: https://github.com/BerriAI/litellm/pull/32972/changes/BASE..bdfb583c2c6f8df10004fb249e11629d41ce71fa#r3565373303
* feat(router): random-pick multi-model complexity tiers
Tier pools already make sense without adaptive; stop pinning lists to
index 0 and shuffle within the classified tier instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): format complexity router config
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): use PEP 585 types for tier pools
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Selecting the LLM classifier without picking a model only surfaced a
toast on submit; the classifier model select now gets the same red
outline and helper text as the tier and embedding selects once a submit
attempt has failed.
_classify_upstream_lifetime decided "expired" from int(float(expires_in)), which truncates toward
zero, so a positive fractional lifetime in (0, 1) became 0 and was misread as already elapsed. That
rejected the mint with 502 in _finish_bridge_mint after the single-use upstream code had already been
consumed, even though the upstream reported a positive remaining lifetime.
Decide expired on the parsed numeric value rather than its truncated int, so only a genuinely
non-positive value is expired. The envelope works in whole seconds and cannot represent a sub-second
lifetime, so a positive value that truncates to 0 clamps up to the 1s floor instead of being rejected.
Values >= 1 still truncate toward zero so the envelope never claims more life than the upstream stated,
and NaN / Infinity / oversized input still read as unparseable ("unspecified").
Regression covers the classifier (0.5 and 0.001 clamp to 1, 1.9 truncates to 1, -0.5 stays expired) and
the mint (a 0.5s upstream lifetime mints a 200 envelope rather than a 502); reverting to the
truncate-then-check reddens both.
Three findings landed together, all one defect: a resolution step crushed several distinct outcomes
into a single None or a silent default, so the mint's error mapper could not tell them apart and
assigned the wrong status. Identity resolution mapped a database outage to the same None as a missing
credential, which the mint reported as 400 invalid_request, blaming the caller for a gateway outage
while admission statuses the same outage 503/500. Lifetime coercion mapped an explicit non-positive
expires_in to the same None as an absent one, so an upstream token the IdP reports as already dead was
sealed into an hour-long envelope. And the refresh_token grant was run through the upstream exchange
(which can rotate the client's upstream refresh credential) and its result then discarded, even though
a bridge server seals no refresh_token and the client never holds one to present.
Rather than add a mapping branch per finding, the fix changes the return types so a wrong status is not
representable. Each resolution step now returns a precise tagged value instead of None: identity
resolution returns a _ResolvedKey or one of no_active_key / unavailable / unresolvable, classified the
same way admission's _reload_admitted_key classifies the same conditions; upstream-lifetime
classification returns a positive number of seconds, "unspecified" (absent or unparseable, which the
envelope caps), or "expired" (a parseable non-positive value, an already-dead token); and upstream-grant
validation returns a typed grant or one of no_access_token / expired_lifetime. Thin exhaustive mappers
(match plus assert_never) lift each vocabulary into one bridge-mint taxonomy of eight named failures,
and a single _bridge_mint_error_response gives each its truthful RFC 6749 §5.2 status: 400 for the
caller's missing credential or an unsupported grant, 503 for a transient auth-DB outage, 500 for a
gateway that cannot resolve identity or is not configured, and 502 for an upstream response with no
usable token, an already-expired lifetime, or a token too large to seal. Adding a failure mode now
requires a new literal and a match arm the type checker forces, so the class of wrong-status bug cannot
recur silently.
The refresh_token grant is rejected in _prepare_bridge_mint before the exchange with
unsupported_grant_type, so it can never rotate or consume the client's upstream refresh credential;
renewal is re-running authorization_code, as the sealed refresh_token=None already intends. An absent or
unparseable expires_in still mints a capped envelope (the by-design behaviour for an upstream that omits
the field); only an explicitly-dead lifetime is rejected.
Tests cover the resolver's three failure classes (including a real connection-error outage and a missing
prisma_client), the mint statuses for each (503 before the upstream exchange, 500, 502 on an expired
upstream lifetime, and a capped mint on an unknown one), and the refresh-grant rejection before any
exchange. The three findings are mutation-checked: reverting each fix turns its regression test red.
_finish_bridge_mint floored the reported expires_in at 1. Admission expires the
envelope against the JWT's second-truncated exp, so when the mint lands in the same
second that exp falls on (a sub-second upstream lifetime, for instance), the true
remaining life is 0 and reporting 1 tells the client the bearer lives one second past
the point admission already rejects it. Floor at 0 instead so the reported lifetime
never overstates the exp; the value still cannot go negative.
The regression pins the boundary directly: minting at now=100.25 with a 1s upstream
token seals exp=101, and the reported expires_in is max(0, 101 - ceil(100.25)) = 0.
Under the old floor of 1 it reads 1, so the test fails on that mutation.
Also drops the unused mcp_server parameter from _prepare_bridge_mint; identity and
key derivation there never referenced the server.
The dcr_bridge oauth_delegate token mint validated its preconditions in two
places: a pre-exchange guard inside exchange_token_with_server (master_key set,
resolvable litellm identity) and an authoritative re-check inside the post-exchange
_mint_bridge_delegate_token_response. Keeping the two in step by hand is what kept
producing the same class of finding: a precondition guarded on one grant branch but
not the other, master_key checked after the exchange on one path, identity resolved
twice, and each failure raising an ad-hoc HTTPException with its own status and body
shape.
Model the mint as three phases whose failures are values. _prepare_bridge_mint runs
before the exchange, checks every precondition once (master_key, then identity), and
returns either a frozen _BridgeMintReady carrying the resolved key hash and the
master-key-derived envelope keys, or a _BridgeMintError literal. Because every
precondition lives in prepare, and prepare runs before the upstream POST, no failure
can burn the single-use code or rotate a refresh token, for either grant type, by
construction rather than by a guard we have to remember to keep in sync.
_finish_bridge_mint runs after the exchange and has no preconditions left that can
fail; its only failure values are properties of the upstream response itself (no
usable access_token, or a token too large to seal). One mapper,
_bridge_mint_error_response, turns each _BridgeMintError into an RFC 6749 section
5.2-shaped body with a status truthful about where the failure is (400 for the
caller, 500 for gateway config, 502 for the upstream), with an exhaustive match plus
assert_never so a new failure mode cannot be added without a matching status.
Behavior is unchanged for the client. Every failure that previously raised now
returns the same status as an OAuth error body, which is the correct token-endpoint
contract; the three tests that asserted a raised HTTPException now assert the
returned response. _exchange_for_bridge_server additionally asserts the identity
resolver is awaited exactly once for a bridge server and never for a non-bridge one.
Follow-up to the pre-exchange identity gate, which I had only added to the
authorization_code branch and which left the master_key check inside the mint
(after the upstream exchange) - so the very burn-then-fail pattern it was meant to
prevent still applied to refresh_token grants and to a misconfigured gateway.
- Hoist a single pre-exchange gate above the upstream call that covers BOTH grant
types: it fails closed (invalid_request) on an unresolvable litellm identity and
500s on an unset master_key BEFORE the single-use code or refresh token is
exchanged/rotated, so a bad key or a misconfigured gateway never burns the
upstream credential.
- Report expires_in from the envelope JWT's own second-truncated exp (rounding the
elapsed portion up) instead of the raw expires_at - now delta, so the client is
never told the bearer is valid past the ~1s point admission already expires it.
Regression tests assert the upstream exchange is never called on the no-identity
refresh grant and the master_key-unset path, and that the reported expires_in does
not overstate the JWT exp.
Findings from a full adversarial review of the mint path across security,
correctness, error-handling, concurrency, and OAuth-protocol dimensions.
- expires_in coercion is now total: int(float(...)) can raise OverflowError on
Infinity / a giant numeric string, which escaped the ValueError/TypeError catch
and 500'd the token endpoint. Unified to catch OverflowError too.
- Resolve the litellm identity BEFORE exchanging the single-use upstream code, so
a missing or transiently-unresolvable identity fails closed with invalid_request
without burning the code (the mint re-resolves via a cache hit).
- The no-identity failure is now an RFC 6749 5.2-shaped invalid_request
(JSONResponse, top-level error, no-store) instead of a detail-wrapped
HTTPException, matching the BYOK OAuth endpoint.
- EnvelopeTooLarge (upstream token too big to seal) surfaces a 502, not a 500.
- The upstream refresh_token is no longer sealed into the envelope: the edge
never consumes it, so it was dead weight embedding a long-lived upstream
credential in the client bearer and enlarging the envelope; refresh is a
follow-up (a dedicated refresh-envelope).
Security review found no exploitable defect (forgery, cross-server/user replay,
leakage, confused-deputy all closed). Regression tests cover the OverflowError,
the code-not-burned path, the RFC-shaped error, the 502, and the dropped refresh.
Two correctness gaps in the bridge mint. _bridge_grant_from_token_response only
accepted an int expires_in, dropping a float (3600.0) or numeric-string ('3600')
lifetime to None so the envelope fell back to its 1h cap and could outlive a
shorter-lived upstream token; coerce it to a positive int (bool excluded). And
_key_is_active called datetime.fromisoformat on the str|datetime expires outside
the resolver's try, so a malformed stored expiry raised an unhandled 500 instead
of the fail-closed invalid_request; it now fails closed (inactive) on an
unparseable expiry. Regression tests cover int/float/string/bool coercion, the
short-float TTL, and the malformed-expiry fail-closed path.
The keyless-user fix moved these signatures, so their pre-existing Optional[...]
annotations counted against the diff and tripped the UP045 strict-budget gate.
Modernize the four touched return annotations to the X | None form the gate
wants; runtime behavior is unchanged.
_resolve_active_litellm_key gated on _active_key_user_id, which returns None both
for blocked/expired keys AND for valid keys with no user_id, so a team-scoped or
service-account key was wrongly rejected with invalid_request at bridge token
exchange. Split the active-state gate (_key_is_active: blocked/expiry only) from
the user_id extraction; the mint seals the key hash, not the user, and admission
already handles a keyless-user key. The per-user token store still gets no user
for such a key, as there is none to key a stored credential by.
The eager access_token = token_response["access_token"] extraction ran before
the dcr_bridge branch, so a missing upstream access_token raised an unhandled
KeyError and _bridge_grant_from_token_response's nil guard (which maps to a clean
502) was dead code. Move the extraction onto the non-bridge result path so the
bridge branch reaches its 502 guard.
The mint bound only user_id/server_id into the envelope, which gave admission
no way to reload the caller's key and enforce its current restrictions. Seal the
hashed authorizing key instead (a one-way digest, not a usable credential), so
admission reloads the live UserAPIKeyAuth by it and the key's team/org/tool
permissions and revocation apply per request.
Extract the token endpoint's key resolution into a shared _resolve_active_litellm_key
so the per-user token store (user_id) and the bridge mint (key hash) derive from one
active-key-gated path, and fail the mint closed with invalid_request when no active
key accompanies the request.
Clicking Add Auto Router with the name empty returned early with only a
toast, so blank tier selects never got their inline error state. The
empty-name branch now sets showValidationErrors and triggers antd
validation on the name field, so every unfilled mandatory field is
flagged at once. Adds a regression test for the tab component.
The Add Auto Router complexity tab let chat models fill the embedding-model
slot (and vice versa) since neither dropdown filtered on ModelGroup.mode, and
submit only required at least one of the four tiers instead of all four. Adds
getMissingTiersError alongside the existing getSemanticConfigError, and
highlights unfilled tier/embedding selects inline once a submit attempt fails.
* refactor(ui): convert endpoint usage charts to shadcn/recharts
Adds a LineChart wrapper to the shared charts kit, mirroring the
BarChart/AreaChart composition with connectNulls and curveType props,
and converts EndpointUsageBarChart and EndpointUsageLineChart from
tremor to the shared wrappers. Both endpoint chart tests now assert on
real recharts SVG output instead of tremor mocks.
* refactor(ui): drop unused endpointData prop from EndpointUsageLineChart
* fix(ui): point endpoint chart test type imports at the UsagePage types alias after colocation move
* refactor(ui): colocate the usage view, keeping the shared usage components
Split for the usage (UsagePage) segment. Most of the folder is the usage page's
own view, but four pieces are reused elsewhere and stay in @/components/UsagePage:
TopKeyView (old-usage), KeyModelUsageView and value_formatters (activity_metrics),
and the shared types (activity_metrics, chartUtils). The other 21 files move into
usage/_components, preserving the folder structure.
The external consumers import only the retained files, so they are untouched. The
moved files' imports of the retained files become @/components/UsagePage paths,
other escaping relative imports are absolutized, and lint suppressions are re-keyed
for moved files only. No behavior change.
* refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface
* feat(ui): adopt openapi-react-query and convert useCustomers to $api
Add openapi-react-query and expose $api = createQueryClient(fetchClient)
alongside fetchClient. Rewrite useCustomers as
$api.useQuery("get", "/customer/list", {}, { enabled, select }), which
derives the query key from method + path (dropping the hand-written
createQueryKeys entry and the manual key) and forwards the request signal
for cancellation. The response type still flows from schema.d.ts as
CustomerResponse[]. Tests assert the path, the admin/token enabled gate,
and the empty-body select fallback.
* test(ui): read the last render's options in useCustomers helper
The lastCallOptions helper was named for the last call but read
mock.calls[0]. Harmless while each test renders once, but it would
silently assert against first-render options if a test ever re-renders.
Read the final call instead.
XecGuard's async_logging_hook wrote a bare dict to
standard_logging_object["guardrail_information"] while the typed
contract is Optional[List[StandardLoggingGuardrailInformation]].
Readers that iterated the field walked dict keys, raised on
info.get, or silently dropped the entry from guardrail usage
tracking and spend-log writes
Construct the typed entry and append it to the existing list or
create a new one, matching the shared helper pattern. Record the
configured guardrail name instead of a hardcoded "xecguard" and
pass the GuardrailEventHooks enum for guardrail_mode
