* feat: add silent CLI token refresh for apiKeyHelper support
lite auth print-token prints a valid proxy credential for use as Claude
Code's apiKeyHelper, transparently refreshing it first if the cached JWT
is stale. This unblocks MDM-managed apiKeyHelper deployments (managed
via `lite auth print-token`) that need silent mid-session credential
rotation without restarting the client.
Refresh capability is backed by a virtual key minted with an empty model
list and cli_refresh metadata, kept strictly separate from the actual
(short-lived, real-model-scoped) call credential -- so a leak of the
credential that flows through every LLM request and subprocess env var
can't also self-renew. The refresh flow is single-use: /sso/cli/refresh
mints a fresh JWT + refresh token pair and blocks the presented refresh
token immediately, so a replay can't mint a second pair from it.
Server: /sso/cli/refresh (rotate) and /sso/cli/logout (revoke) endpoints.
lite login now also stores a refresh token; lite logout revokes it
server-side instead of only clearing the local file.
* fix: allow non-admin users to hit CLI refresh routes; resolve apiKeyHelper base_url from token.json
Found via a live end-to-end test against a real proxy + real Claude Code
session: /sso/cli/refresh and /sso/cli/logout were unreachable for any
non-proxy-admin caller, since Depends(user_api_key_auth) pulls in a
route-RBAC gate that 403s any route not on an explicit allowlist. That
made the feature unusable for actual end users, who authenticate as
internal_user. Add both routes to internal_user_routes; the handlers
already do their own fine-grained check (metadata.cli_refresh) same as
/key/block does today.
Also: `lite auth print-token` required an explicit --base-url/
LITELLM_PROXY_URL matching the stored token's origin, defaulting to
localhost:4000 otherwise. But apiKeyHelper is configured bare (no
flags), so this always mismatched a real deployment. Track whether
--base-url was explicitly passed (via click's ParameterSource) and, if
not, resolve the server from token.json directly instead of the CLI
default.
* test: mock refresh-token minting in test_cli_poll_key_tolerates_missing_user_row
Landed on litellm_internal_staging after this branch's refresh-key minting
change; needs the same mock as the other cli_poll_key tests since minting
now runs unconditionally whenever a JWT is generated.
* fix(ci): update test_cli_auth.py for refresh_token contract, regenerate schema.d.ts
_poll_for_authentication now always includes "refresh_token" in its
returned dict, and _handle_team_selection_during_polling returns a dict
instead of a bare JWT string -- test_cli_auth.py predates this branch's
refresh-token work and still asserted the old shapes.
schema.d.ts regenerated via `npm run gen:api` to pick up the new
/sso/cli/refresh and /sso/cli/logout routes (plus unrelated drift from
other PRs merged since it was last generated).
* fix(ci): apply CI's own schema.d.ts diff (enterprise routes I can't generate locally)
Local `npm run gen:api` only sees OSS routes -- this machine's
litellm_enterprise editable install points at a now-deleted temp
directory, so it silently drops enterprise-only routes from the spec.
Applied the exact diff CI's own generation produced instead of
re-running the generator locally.
* fix: close refresh-token race, fail closed on DB down, fix logout base_url
Addresses Greptile review findings on the CLI refresh-token PR:
- cli_refresh_token minted a new JWT + refresh token BEFORE blocking the
presented one. Two concurrent requests bearing the same refresh token
could both pass auth and both mint fresh pairs, yielding four live
credentials from one consumed token. Now the presented token is
consumed atomically first via update_many (only succeeding if it flips
blocked from False/None to True); the loser gets count=0 and is
rejected before anything is minted.
- When prisma_client is None, refresh silently returned a new JWT
without ever being able to mark the presented token consumed, leaving
it valid indefinitely. Now fails closed with a 500 instead.
- `lite logout` sent its revocation POST to ctx.obj["base_url"], which
defaults to localhost:4000 when --base-url isn't passed -- the same
bug print_token had before the base_url_explicit fix, just missed
here. Now resolves the same way: trust the stored token's origin
unless the caller explicitly overrode --base-url.
* fix(ci): satisfy ruff format and narrow token_data type in logout
* fix(security): never trust refresh-token metadata for authorization
Addresses a real privilege-escalation path Veria flagged: cli_refresh_token
read team_id, team_alias, and max_budget straight off the presented
token's own metadata and used them to authorize the new JWT. Since any
authenticated user can self-mint a virtual key with arbitrary metadata
via the ordinary /key/generate endpoint, a self-forged key with
{"cli_refresh": true, "team_id": "<any-team>", "max_budget": 999999999}
would sail through _require_cli_refresh_token's only check
(metadata.cli_refresh == True) and get a JWT scoped to a team the
caller never belonged to, with a budget it never had -- full
cross-team / budget bypass, and a removed team member could keep
refreshing team-scoped sessions indefinitely.
Metadata's team_id is now treated as an untrusted UX hint only: honored
solely if the CALLER (identified by the authenticated key's own
user_id, not client input) is a current member per a fresh
get_user_object lookup. team_alias and max_budget are never read back
from metadata at all -- team_alias comes from a live get_team_object
lookup and max_budget is recomputed with the exact same capping logic
the initial SSO login poll uses. _mint_cli_refresh_token no longer
accepts or stores team_alias/max_budget, only the team_id hint.
Added regression tests proving: a forged/stale team_id is dropped
(falls back to no team, not silently honored), and a forged max_budget
in metadata never reaches the issued JWT.
* fix(ci): catch HTTPException specifically instead of bare Exception (BLE001)
* fix: un-consume refresh token if minting the replacement fails
Greptile flagged a real reliability gap: cli_refresh_token blocks the
presented token atomically, then does several more DB calls before
returning a replacement (user lookup, team lookup, JWT mint, new
refresh-key mint). Since this endpoint exists specifically for fully
unattended apiKeyHelper operation, a single transient failure in that
window (DB hiccup, etc.) permanently stranded the user: their old
token was already dead and no new one was issued, with no recovery
path short of a full interactive browser re-login.
Wrap that window in try/except; on any failure, best-effort revert the
consumed token back to usable (blocked=False) before re-raising, so a
retry can succeed. Standard compensating-action pattern since
generate_key_helper_fn doesn't take an injectable transaction, so
wrapping the whole thing in a real DB transaction isn't practical here.
* fix(security): refresh key had unrestricted model access, not none
Critical bug: _mint_cli_refresh_token used models=[] intending "no LLM
access", but that's backwards in this codebase. Per
_check_model_access_helper: `len(filtered_models) == 0 and len(models)
== 0` -> all_model_access = True. An empty models list on a key with no
team_id means UNRESTRICTED access to every model, not zero access. The
CLI refresh token -- meant to be usable for nothing but silently
exchanging itself for a new JWT -- was actually a fully unrestricted
API key for its entire 90-day lifetime, completely undermining the
whole point of keeping it separate from the short-lived call
credential.
Fixed with two independent layers: allowed_routes hard-restricts the
key to exactly /sso/cli/refresh and /sso/cli/logout (the real enforced
boundary, checked in the shared user_api_key_auth dependency for every
route); models is set to an unmatchable sentinel string as
defense-in-depth in case any code path only consults the models field.
Added an end-to-end regression test that exercises the actual
model-access-control function against a key shaped like the minted
refresh token, rather than only asserting on what arguments were passed
to the key-generation call -- the latter kind of test is exactly what
let the original bug ship, since asserting `models == []` is equally
consistent with "no access" and "unrestricted access" without checking
what the access-control code actually does with that shape.
Also: the compensating-rollback added for reliability un-blocked a
consumed refresh token even when the underlying user no longer exists.
That's a permanent, intentional rejection, not a transient failure --
un-blocking it would let a stale refresh token become valid again for a
different account if the user_id is ever reused/re-registered. Moved
the user-existence check outside the rollback-on-failure block so it
stays permanently blocked.
* refactor: rotate CLI refresh tokens via regenerate_key_fn instead of hand-rolled consume/rollback
The refresh token is already a plain litellm virtual key, so rotation can
delegate to the same atomic DB update /key/regenerate uses instead of a
bespoke update_many + compensating-rollback dance. This makes silent CLI
refresh an Enterprise feature, same as regular key regeneration.
* refactor: replace CLI stateless JWT + refresh-key pair with one self-rotating virtual key
The CLI previously minted two credentials on login: a stateless self-signed
JWT for LLM calls, and a separate DB-backed refresh-only key (scoped away
from ever calling an LLM) just to authorize minting a new JWT. Collapse
this into a single real virtual key, used directly as the LLM bearer token
and re-presented to /sso/cli/refresh to rotate its own secret in place.
This also means the CLI session key now shows up in the Admin UI's Keys
page and can be revoked/regenerated like any other key, rather than being
an invisible, unmanageable stateless token.
* refactor: drop silent CLI refresh, key just expires and requires re-login
/sso/cli/refresh only ever benefited Enterprise deployments (regenerate_key_fn's
gate), while everyone else already fell through to "re-run lite login" on
failure. Cut the endpoint, the rotation logic, and the client-side refresh
path entirely; print-token now just prints the cached key until it hits its
LITELLM_CLI_JWT_EXPIRATION_HOURS duration, then fails fast telling the user
to log in again. Session key itself is unaffected: still a real, revocable
virtual key visible in the Keys UI, `lite logout` still revokes it directly.
* fix(ci): regenerate schema.d.ts after removing /sso/cli/refresh route
* revert: go back to stateless JWT, keep only lite auth print-token
The virtual-key redesign (revocable, Keys-UI-visible credential) wasn't
needed just to support print-token, and cost real server-side surface
(a mint path, a logout-revoke endpoint, migrated tests/docs) for a property
this repo doesn't need yet. Reverting cli_poll_key/_types.py/schema.d.ts
back to the original stateless-JWT design; the only durable addition from
this whole effort is `lite auth print-token` (reads the cached credential,
prints it while fresh, fails with a clear message once it's past
LITELLM_CLI_JWT_EXPIRATION_HOURS) plus the base_url_explicit plumbing it
needs. `lite logout` goes back to clearing the local file only, since a
stateless JWT can't be revoked server-side.
* refactor: move CLI token freshness check to cli_token_utils, drop unnecessary renames
Addresses review: the freshness check is a pure token-shape/timestamp
util, not command logic, so it belongs alongside the other SDK-level
CLI token helpers (load_cli_token, get_litellm_gateway_api_key) rather
than in commands/auth.py. Also reverted a few incidental jwt_token/
session_key variable and string renames that weren't load-bearing.
OpenAI is winding down self-serve fine-tuning and the org can no longer
create fine-tuning jobs (403 training_not_available; the CI key surfaces
it as a 500 server_error), so test_create_fine_tune_jobs_async fails on
every batches_testing run since 2026-07-11 and reruns never clear it.
The request contract stays covered by the mocked create/list/cancel/
retrieve tests in the same file, and the deleted test's unique
standard_logging_object assertions now run inside
test_mock_openai_create_fine_tune_job.
Streaming requests return from common_request_processing before
async_post_call_success_hook runs, so response._hidden_params.additional_headers
never gets the v3 x-ratelimit-{descriptor_key}-{remaining|limit}-{rate_limit_type}
entries. Prometheus / logging callbacks that read those values from
standard_logging_object.hidden_params.additional_headers then see nothing;
combined with the pre-existing gap that Prometheus reads from that same slot
(LIT-2577 / PR #28816), per-key remaining RPM/TPM cannot be monitored for
streaming traffic at all.
Fix in three parts:
- Stash the pre-call RateLimitResponse in the metadata channels the async
success-logging callback inherits, alongside the existing top-level entry
the non-streaming path reads.
- Add async_logging_hook to the v3 handler. It fires in a distinct earlier
loop inside async_success_handler (all callbacks' async_logging_hook
complete before any async_log_success_event starts), so mirroring the
pre-call snapshot into standard_logging_object.hidden_params.additional_headers
and response._hidden_params.additional_headers here guarantees every
downstream success callback sees the values regardless of registration
order. Non-streaming keeps the existing async_post_call_success_hook write
and this hook re-populates the same values idempotently.
- Extract the shared `_merge_ratelimit_statuses_into_additional_headers`
helper the non-streaming path already had inlined so both callsites emit
the identical key shape.
Exclude vertex_ai from pipecat tool smoke; raw-ws tool_call_round_trip
remains the Vertex source of truth. Also remove the Playwright key models
dropdown suite so stage is not blocked by that UI harness
The eslint-metrics.json snapshot duplicated the violation counts already
enforced by eslint-budgets.json. Keeping it current added a CI drift check,
a pre-commit regenerate-and-flag step, and a standalone npm run lint:metrics
script, none of which caught anything the budget gate did not, yet all of
which failed noisily whenever the snapshot went stale. This drops the file
and that machinery while leaving eslint-budgets.json as the actual ratchet
gate
Replace the request.path_params read (and its defensive getattr guard) with
the route template. A real Starlette request always exposes path_params, but
common_checks runs on lightweight request doubles that don't, so reading it
directly forced a getattr workaround that only existed to tolerate those
doubles.
Instead, match the route template (/team/{team_id}) to identify the RESTful
update route and take the team id from the last path segment. This drops the
path_params dependency entirely, and because the template distinguishes the
PATCH route from its single-segment siblings (/team/new, /team/list, ...), it
also avoids a spurious team lookup those routes would otherwise trigger if we
matched the resolved path shape alone.
* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells
Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat
traffic and assert calculatedTotalCost matches StandardLogging response_cost
and proxy spend. Also assert tool calls and applied guardrails land on the
trace. Missing env or proxy is a hard failure, never a skip
* test(e2e): use langfuse_otel callback for Langfuse spend coverage
Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to
Langfuse) instead of the classic langfuse SDK. Match generations named
litellm_request by prompt marker and user_api_key_alias
* test(e2e): require Langfuse spend assert; drop AGENTS.md
Guardrail path no longer soft-gates logs_spend. Non-stream responses must
return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md
* test(e2e): fail when Langfuse spend is missing on guardrail path
Always run logs_spend assertions for tool_permission; require positive
x-litellm-response-cost on non-stream and positive /spend/logs spend
* test(e2e): do not fall back to unmatched spend log rows
poll_proxy_spend_for_key returns None when response_id or positive-spend
filters match nothing, instead of silently using rows[0]
* feat(ui): add the typed openapi-fetch client (fetchClient) as the dashboard fetch foundation
Introduces fetchClient (openapi-fetch) bound to schema.d.ts, used inside ordinary TanStack Query hooks so path/query/body types come from the proxy's OpenAPI spec. A small runtime registry feeds the client the base URL and auth header name (registered by networking) and the session token (published by AuthContext), so call sites carry no token plumbing; auth-header injection and ApiError mapping live in openapi-fetch middleware reusing deriveErrorMessage/ApiError from client.ts, and non-2xx maps to a thrown ApiError so query functions just read .data.
The base URL default resolves from NEXT_PUBLIC_BASE_URL so a request still targets the right origin if it fires before networking registers its getter. AuthContext clears accessToken alongside the token on logout so no query fires unauthenticated after the session ends.
Foundation only; callers migrate one at a time, each fully typed, in follow-up changes.
* feat(ui): migrate useCustomers to the typed fetchClient
Converts useCustomers from allEndUsersCall to fetchClient.GET("/customer/list"); the response is typed as LiteLLM_EndUserTable[] from the schema, so the hand-written Customer/CustomersResponse types are deleted. They were also inaccurate (allowed_model_region was string but is "eu"|"us", and a budget_id the table has no field for). No cast; the schema type flows to the one consumer. First caller on the new pattern.
* fix(ui): route typed-client errors through the session-expiry handler
The typed fetchClient middleware threw ApiError without invoking the
handleError side effect that the legacy createApiClient wires via
onError, so a migrated caller hitting an expired key no longer triggered
the auto-logout. Add an error-handler seam to runtime.ts, register
handleError from networking.tsx alongside the base-url/header getters,
and call it in the middleware before throwing so both clients behave the
same. Regression test asserts the handler fires with the derived message
on non-2xx and stays silent on success
* fix(ui): point the customers EndUser type at CustomerResponse
The /customer/list response model was renamed to CustomerResponse on
staging; the merged branch still aliased EndUser to LiteLLM_EndUserTable,
so the exported type and its test mock had drifted from what the schema
actually returns. CustomerResponse is also the accurate shape (it types
allowed_model_region as 'eu' | 'us' and carries budget_id)
* chore(ui): refresh eslint-metrics baseline after staging merge
The recorded baseline predated the litellm_internal_staging merge, so its
no-explicit-any and no-large-inline-object-arg counts were higher than the
merged tree actually has. Regenerate via npm run lint:metrics so the gate
reflects current reality
* refactor(ui): source the typed client token from the session cookie, not AuthContext
The typed client read its bearer from a runtime value that AuthContext pushed
via setAuthToken, but migrated hooks gate enabled on useAuthorized, which
decodes the cookie directly. Two independent derivations of the same cookie with
different timing: on first load the query fires (useAuthorized sees the token)
before AuthContext's async effect publishes it, so the first request goes out
unauthenticated and only succeeds on a React Query retry.
Make the token a registered getter like the base-url and header-name getters,
reading the same cookie useAuthorized decodes, so the client's token and the
gate can't diverge. Revert the AuthContext changes entirely; nothing is pushed
from React state anymore.
Replace lodash/debounce in TeamVirtualKeysTable with useDebouncedValue from
@tanstack/react-pacer, matching the sibling VirtualKeysTable and
PaginatedKeyAliasSelect which already debounce their key-alias search that way.
Pacer is already a dependency, so this drops the odd-one-out lodash usage and
keeps the search-debounce pattern consistent across the key tables.
The stale-redirect path logged three warnings for a single re-registration: the staleness probe plus the reuse skip in both register_client_with_server and the persist race guard. The reuse-skip message is a mechanical consequence of the probe's decision, so it now logs at debug; the actionable warning that names both bindings and the re-authentication impact is emitted once by _persisted_dcr_redirect_uri_is_stale
A dynamically registered (RFC 7591) OAuth client persisted onto the MCP server row is bound to the redirect_uri it was first registered with, but that binding was never recorded. After the proxy's public origin changed, every authorize paired the reused client with the new callback and the IdP rejected it permanently.
The DCR persist now records redirect_uris alongside the client identity. The admin register path treats a positive mismatch between the recording and the current callback as stale and re-registers a replacement client; rows without a recording (pre-existing installs and admin-configured clients) are grandfathered so upgrades never re-mint client_ids or orphan refresh tokens. The persist also writes client_secret and token_endpoint_auth_method explicitly as None when absent so the credential blob merge cannot pair a re-registered public client with the previous client's secret. Public register routes and non-admin callers keep existing behavior.
Closes#32473
The PATCH /team/{team_id} org-context wiring reads request.path_params to
resolve the team id from the path. A real Starlette Request always exposes
path_params, but common_checks is exercised with lightweight request doubles
that don't, which raised AttributeError. Read it defensively so a missing or
null path_params falls back to no path team id, matching the "not a bare team
route" outcome; real requests are unaffected
apply_json_merge_patch recurses into nested objects, which the repo's recursive_detector code-quality check flags because unbounded recursion over caller-supplied JSON has caused CPU/stack issues before. Cap the recursion at a depth far above any realistic team-metadata shape and reject deeper patches with a ValueError so a pathologically nested body fails closed instead of overflowing the stack, then register the function in the detector's ignore list alongside the other depth-bounded JSON walkers
The AI Gateway selector now always renders at the breadcrumb root, even with no plugins and Chat UI disabled, so the Chat feature stays discoverable. The Chat entry is always listed: clickable when enabled, and disabled with an "Admins can enable in Settings" hint when it is off.
Since the selector is now unconditional, the useViewSwitcherVisible hook and the section-crumb fallback added in the previous commit are removed
The AI Gateway select (ViewSwitcher) now sits at the root of the DashboardHeader breadcrumb instead of on the right, so the top bar reads [AI Gateway select] > Page to match the redesign. It keeps the same dropdown, including the Chat / Chat UI options.
When no plugins are registered and Chat UI is disabled there is nothing to switch between, so the breadcrumb falls back to the static section crumb rather than rendering a dangling leading separator
Wire the coarse route gate so PATCH /team/{team_id} is reachable by exactly the roles that can call POST /team/update: proxy admins, org admins of the team's own organization, and JWT admins. Regular internal users and view-only proxy admins stay blocked, matching the existing endpoint
Because the team id lives in the path rather than the body, the org-context resolver now also reads it from path_team_id for the bare /team/{team_id} route, so an org admin's organization is resolved and injected the same way it already is for POST /team/update. /team/{team_id} is added to management_routes rather than the role-agnostic self_managed_routes; the latter would have opened POST /team/new to any authenticated user through the shared /team/{team_id} path pattern
Exact cost-map hits resolve before fallback-generalization rules, so the
mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the
bedrock-anthropic-claude-mid-conversation-system rule and hoisted
mid-conversation system messages, invalidating the prompt cache.
* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support
AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface
(thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed
model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5)
get the effort translated to a legacy thinking budget_tokens. Models with no
reasoning support have thinking/effort dropped under drop_params. And because
adaptive thinking carries no budget while the legacy form must satisfy Anthropic's
max_tokens > budget_tokens rule, the translated budget is capped below max_tokens,
dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass
through untouched.
This matters because clients like Claude Code speak native Anthropic /v1/messages
and send the adaptive interface unconditionally, regardless of the routed model.
The native passthrough previously only capability-gated the OpenAI-style
reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so
a pre-4.6 model rejected it with "This model does not support the effort parameter"
and the request failed. Claude Code already gets drop_params auto-set, so its
requests now succeed.
* test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests
Addresses review feedback on the max_tokens-too-small branch. Previously a
thinking-capable model whose max_tokens could not fit the minimum thinking budget
had thinking silently dropped regardless of drop_params, while a residual
output_config field in the same call still raised when drop_params was off. Gate
both consistently on drop_params: raise a clear error (naming max_tokens for the
undersized case) when drop_params is off, drop otherwise. Claude Code gets
drop_params auto-set, so it still succeeds.
Adds tests for the undersized-max_tokens raise, the residual output_config raise,
and the no-adaptive-interface passthrough on a non-adaptive model.
* fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts
The previous raise-when-not-drop_params behavior broke existing bedrock and vertex
messages tests: those providers already silently strip unsupported output_config
for pre-4.6 models (issue #22797) with no drop_params required, and the shared
parent transform raising pre-empted that. It also conflicted with the goal of
keeping requests working rather than failing them.
Make the reshape silent: translate effort to legacy thinking for thinking-capable
models, drop thinking for non-reasoning models, and remove only the consumed effort
key from output_config, leaving any residual (e.g. format) for provider subclasses
(bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the
review note about inconsistent drop_params handling by making every path uniform.
Updates the tests to assert the silent behavior and residual output_config
preservation.
* fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5)
Greptile caught a real bug: the early-return guard treated supports_output_config
as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises
supports_output_config (it accepts output_config.effort) but is not adaptive, so it
rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this
model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking
block raw, reproducing the exact failure the fix is meant to prevent.
thinking:{type:adaptive} and output_config.effort are independent capabilities.
Only early-return for adaptive-thinking models. For a model that supports
output_config.effort but is not adaptive, keep the native effort and drop only the
unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code
payload now returns 200 instead of 400.
Adds regression tests for Opus 4.5 with and without adaptive thinking.
* fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models
Claude Opus 4.5 advertises supports_output_config but not adaptive thinking,
so the early-return guard forwarded thinking.type=adaptive raw and Anthropic
rejected it. The guard now only skips true adaptive models; effort-only
requests on effort-capable models still pass through untouched. The
_map_reasoning_effort call is wrapped to surface unrecognized effort values
as a clean 400, matching _translate_reasoning_effort_to_anthropic
* fix(anthropic): fall back to legacy thinking when effort level unsupported
Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code
defaults to xhigh on newer models, so preserving that level raw gets rejected
by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model
and fall through to the budget translation for unsupported levels
* fix(anthropic): keep effort-only requests untouched for provider normalization
The xhigh fall-through consumed effort-only requests on effort-capable
models, breaking bedrock invoke's own normalization which clamps xhigh to
the model's ceiling after the base transform runs
(test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict
the fall-through to requests that carry adaptive thinking; effort-only
requests pass through so provider subclasses keep owning level clamping
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
vertex_ai/claude-opus-4-8@default (and sibling @default models) were
misclassified as non-adaptive because _model_map_lookup_candidates only
stripped provider prefixes but never the @<suffix> portion. The lookup
produced candidates like ["vertex_ai/claude-opus-4-8@default",
"claude-opus-4-8@default"], neither of which exists in model_cost, so
_is_adaptive_thinking_model returned False. LiteLLM then sent
thinking.type=enabled to a @default Vertex AI endpoint that requires
thinking.type=adaptive, resulting in a 400.
_strip_version_suffix now removes @<suffix> from each candidate,
adding the bare model name (e.g. "claude-opus-4-8") to the lookup
chain. Also adds supports_adaptive_thinking: true to the three
@default model_cost entries that were missing it as belt-and-suspenders.
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
adds a top-level guard in _increment_remaining_budget_metrics that returns early
when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config),
and per-entity guards in each _set_*_budget_metrics_after_api_request helper for
partial disabling. eliminates four async DB/cache round-trips per successful LLM
request when budget metrics are disabled.
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param
- BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each
chunk and raises litellm.APIError for type=error and type=response.failed events
so callers see an exception instead of a benign stream chunk
- rate_limit* codes map to 429; client error codes (invalid_request_error,
context_length_exceeded, etc.) map to 400; all other codes default to 500;
raw integer codes are never used as-is as HTTP status codes
- ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]]
to prevent Pydantic ValidationError on dict-typed param payloads silently
dropping error events before any type inspection
* test(responses-api): add streaming iterator error event tests to CI-covered path
* test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths
* test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk
* fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting
insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees
_handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code
usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost
dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed
* fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status
Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and
5xx in-stream error events now raise MidStreamFallbackError carrying the
mapped APIError so the router's FallbackResponsesStreamWrapper triggers
mid-stream fallback and cooldown; non-retriable 4xx still raise APIError
directly. Status mapping now reads both the OpenAI error type and code
fields, so type-classified client errors (e.g. invalid_request_error with
code invalid_prompt) map to 400 instead of falling through to 500.
* fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting
MidStreamFallbackError was always raised with generated_content="", so the
router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk
and retried with the original input, streaming duplicated content to clients
that had already received partial output. The iterators now accumulate
response.output_text.delta text (mirroring chat's response_uptil_now) and pass
it as generated_content, letting the router build a continuation input via
_build_responses_continuation_input.
* test(responses-api): pin in-stream token limit error to raised APIError
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(bedrock): gate in-place system role messages on model support for Claude Invoke
* feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule
* fix(responses): preserve reasoning_tokens through chat->responses usage translation
Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever
completion_tokens_details.reasoning_tokens was None or absent. Also change
OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that
re-instantiation without explicit reasoning_tokens no longer silently zeroes out
the field, and remove the same hardcoded zero from the mock_responses_api_response
initializer.
* test(responses): update assertions to match Optional[int] reasoning_tokens default
* fix(responses): preserve explicit reasoning_tokens=0 in usage translation
Align the reasoning_tokens guard with the is-not-None guards used for
text_tokens and image_tokens: a provider-reported zero passes through
while an absent value stays omitted.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map
* test: use apac regional profile for cost-map fallback test since jp now has an entry
* feat(router): add LLM-based classifier option to complexity router
Adds classifier_type: "heuristic" | "llm" to complexity_router_config.
When set to "llm", the router calls a configured model (e.g. a small
model like haiku) via structured output to pick the complexity tier,
falling back to the existing regex/keyword scorer on any error, empty
response, or unparseable output.
* feat(ui): add classifier_type option to complexity router UI, fix edit flow
Adds an "Advanced: Classification Method" section to ComplexityRouterConfig
with a heuristic/LLM toggle, revealing a classifier model picker and timeout
when LLM is selected.
Also fixes the auto router edit modal, which never rendered the complexity
router UI at all (it only handled the semantic router), and the "Edit Auto
Router" button visibility check, which was gated on auto_router_config and
never matched complexity router deployments.
* fix(router): attribute classifier calls to caller, raise default timeout
Forwards the original request's litellm_metadata into the classifier's
acompletion call. Without it, the proxy's cost-tracking gate sees no
user_api_key/team_id/user_id and silently drops spend logging and budget
accounting for every classifier call, letting an authenticated user rack
up unaccounted provider spend via repeated requests.
Also raises the default classifier timeout from 400ms to 3000ms (400ms
undershoots real LLM latency and would silently degrade to the heuristic
scorer on most requests) and corrects the module/class docstrings, which
still claimed zero external API calls after the llm classifier path was
added.
* fix(ci): resolve ruff strict-budget and frontend-lint failures
- Use PEP 585 generics (dict/tuple/list) in the new aclassify/_classify_with_llm
signatures instead of typing.Dict/Tuple/List, and suppress BLE001 on the
intentionally broad except in aclassify's fallback path with a reason.
- Fix prettier formatting in ComplexityRouterConfig.tsx.
- Regenerate eslint-metrics.json (was stale after the classifier UI changes).
* fix(ci): regenerate stale eslint-metrics.json
* fix(router): strip parent budget reservation from classifier metadata
The classifier's internal acompletion call previously forwarded the
parent request's full litellm_metadata, including its budget
reservation (user_api_key_budget_reservation / user_api_key_auth).
That reservation belongs to the routed completion the classifier is
deciding on, not to the classifier call itself, so it's now stripped
while key/team attribution fields are still forwarded for spend
logging.
Add a RESTful PATCH /team/{team_id} that partially updates a team using RFC 7386 JSON Merge Patch. team_id comes from the path, and metadata is merged with the team's stored metadata instead of being replaced wholesale the way POST /team/update does: an omitted key is preserved, key: null deletes it, and any other value overwrites, recursing into nested objects. Every other field behaves the same as POST /team/update
The handler delegates to the existing update path, so authorization, budget checks, system-managed-key stripping, metadata encryption, cache refresh, and audit logging are shared rather than reimplemented. POST /team/update is untouched, so the change is purely additive
