Commit graph

43876 commits

Author SHA1 Message Date
Yuneng Jiang
368342cdde
refactor(auth): resolve PATCH team org-context from the route template
Replace the request.path_params read (and its defensive getattr guard) with
the route template. A real Starlette request always exposes path_params, but
common_checks runs on lightweight request doubles that don't, so reading it
directly forced a getattr workaround that only existed to tolerate those
doubles.

Instead, match the route template (/team/{team_id}) to identify the RESTful
update route and take the team id from the last path segment. This drops the
path_params dependency entirely, and because the template distinguishes the
PATCH route from its single-segment siblings (/team/new, /team/list, ...), it
also avoids a spurious team lookup those routes would otherwise trigger if we
matched the resolved path shape alone.
2026-07-11 11:24:41 -07:00
mateo-berri
4486d9fe0c
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_complexity_router_keyword_tiers
# Conflicts:
#	ui/litellm-dashboard/eslint-metrics.json
2026-07-11 18:22:44 +00:00
tin-berri
9659ae2f40
Merge pull request #32921 from BerriAI/litellm_lit4126_default_port_strip
fix(mcp): strip scheme default port from get_request_base_url netloc
2026-07-11 11:19:30 -07:00
Abhimanyu Kapur
eb0c966255 fix(complexity_router): use max aggregation for semantic keyword route scoring
SemanticRouter defaults to mean aggregation across a route's utterances. Since
each tier's route holds one utterance per configured keyword, a real semantic
match on one keyword was averaged together with the tier's other, unrelated
keywords and dragged below match_threshold — e.g. a MEDIUM tier with keywords
[beep, boop, new york] never fired for a genuine "new york" paraphrase, because
mean(sim_to_beep, sim_to_boop, sim_to_new_york) landed well under the threshold
even though sim_to_new_york alone cleared it. Pass aggregation="max" so a tier
matches if the query is close enough to any one of its keywords, not the
average of all of them.

Verified against live Voyage embeddings: raw cosine similarity for "new york"
vs a paraphrase was 0.54 (above a 0.5 threshold), but the route scored 0.28
under mean aggregation and never matched; max aggregation fixes it.

Adds a regression test with a tier holding one matching and two unrelated
keywords, asserting the tier still fires; fails without aggregation="max".
2026-07-11 11:14:59 -07:00
yuneng-jiang
5d7b7047a8
Merge pull request #32919 from BerriAI/litellm_pacer_debounce_team_keys
refactor(ui): use TanStack Pacer debounce for the team keys search
2026-07-11 11:14:42 -07:00
mubashir1osmani
1bf98c0687
test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857)
* test(e2e): cover Langfuse logging.yaml P0 logs_spend cells

Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat
traffic and assert calculatedTotalCost matches StandardLogging response_cost
and proxy spend. Also assert tool calls and applied guardrails land on the
trace. Missing env or proxy is a hard failure, never a skip

* test(e2e): use langfuse_otel callback for Langfuse spend coverage

Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to
Langfuse) instead of the classic langfuse SDK. Match generations named
litellm_request by prompt marker and user_api_key_alias

* test(e2e): require Langfuse spend assert; drop AGENTS.md

Guardrail path no longer soft-gates logs_spend. Non-stream responses must
return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md

* test(e2e): fail when Langfuse spend is missing on guardrail path

Always run logs_spend assertions for tool_permission; require positive
x-litellm-response-cost on non-stream and positive /spend/logs spend

* test(e2e): do not fall back to unmatched spend log rows

poll_proxy_spend_for_key returns None when response_id or positive-spend
filters match nothing, instead of silently using rows[0]
2026-07-11 14:11:00 -04:00
ryan-crabbe-berri
0bf81e2496
feat(ui): typed openapi-fetch foundation (fetchClient) + first typed caller (useCustomers) (#29884)
* feat(ui): add the typed openapi-fetch client (fetchClient) as the dashboard fetch foundation

Introduces fetchClient (openapi-fetch) bound to schema.d.ts, used inside ordinary TanStack Query hooks so path/query/body types come from the proxy's OpenAPI spec. A small runtime registry feeds the client the base URL and auth header name (registered by networking) and the session token (published by AuthContext), so call sites carry no token plumbing; auth-header injection and ApiError mapping live in openapi-fetch middleware reusing deriveErrorMessage/ApiError from client.ts, and non-2xx maps to a thrown ApiError so query functions just read .data.

The base URL default resolves from NEXT_PUBLIC_BASE_URL so a request still targets the right origin if it fires before networking registers its getter. AuthContext clears accessToken alongside the token on logout so no query fires unauthenticated after the session ends.

Foundation only; callers migrate one at a time, each fully typed, in follow-up changes.

* feat(ui): migrate useCustomers to the typed fetchClient

Converts useCustomers from allEndUsersCall to fetchClient.GET("/customer/list"); the response is typed as LiteLLM_EndUserTable[] from the schema, so the hand-written Customer/CustomersResponse types are deleted. They were also inaccurate (allowed_model_region was string but is "eu"|"us", and a budget_id the table has no field for). No cast; the schema type flows to the one consumer. First caller on the new pattern.

* fix(ui): route typed-client errors through the session-expiry handler

The typed fetchClient middleware threw ApiError without invoking the
handleError side effect that the legacy createApiClient wires via
onError, so a migrated caller hitting an expired key no longer triggered
the auto-logout. Add an error-handler seam to runtime.ts, register
handleError from networking.tsx alongside the base-url/header getters,
and call it in the middleware before throwing so both clients behave the
same. Regression test asserts the handler fires with the derived message
on non-2xx and stays silent on success

* fix(ui): point the customers EndUser type at CustomerResponse

The /customer/list response model was renamed to CustomerResponse on
staging; the merged branch still aliased EndUser to LiteLLM_EndUserTable,
so the exported type and its test mock had drifted from what the schema
actually returns. CustomerResponse is also the accurate shape (it types
allowed_model_region as 'eu' | 'us' and carries budget_id)

* chore(ui): refresh eslint-metrics baseline after staging merge

The recorded baseline predated the litellm_internal_staging merge, so its
no-explicit-any and no-large-inline-object-arg counts were higher than the
merged tree actually has. Regenerate via npm run lint:metrics so the gate
reflects current reality

* refactor(ui): source the typed client token from the session cookie, not AuthContext

The typed client read its bearer from a runtime value that AuthContext pushed
via setAuthToken, but migrated hooks gate enabled on useAuthorized, which
decodes the cookie directly. Two independent derivations of the same cookie with
different timing: on first load the query fires (useAuthorized sees the token)
before AuthContext's async effect publishes it, so the first request goes out
unauthenticated and only succeeds on a React Query retry.

Make the token a registered getter like the base-url and header-name getters,
reading the same cookie useAuthorized decodes, so the client's token and the
gate can't diverge. Revert the AuthContext changes entirely; nothing is pushed
from React state anymore.
2026-07-11 10:50:51 -07:00
Tin Chi Lo
168bc38d81 fix(mcp): strip scheme default port from get_request_base_url netloc 2026-07-11 10:44:23 -07:00
mateo-berri
6fa088224b fix(fallback-generalizations): cover bare Claude majors in baseline and routing, require claude- prefix in adaptive gate 2026-07-11 10:42:47 -07:00
Yuneng Jiang
18c9259d00
refactor(ui): use TanStack Pacer debounce for the team keys search
Replace lodash/debounce in TeamVirtualKeysTable with useDebouncedValue from
@tanstack/react-pacer, matching the sibling VirtualKeysTable and
PaginatedKeyAliasSelect which already debounce their key-alias search that way.
Pacer is already a dependency, so this drops the odd-one-out lodash usage and
keeps the search-debounce pattern consistent across the key tables.
2026-07-11 10:38:00 -07:00
yuneng-jiang
80c5217ddc
Merge pull request #32856 from BerriAI/litellm_/data-table-design-exploration-f3f5e9
feat(ui): add filter drawer, column visibility, and search to the shared DataTable
2026-07-11 10:27:41 -07:00
mateo-berri
55c8ca41b5 ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
tin-berri
7b9ce81543
Merge pull request #32804 from BerriAI/litellm_lit4337_dcr_bridge_ui
feat(ui): dcr_bridge toggle for client-forwarded MCP auth modes
2026-07-11 10:18:46 -07:00
Tin Chi Lo
2254ffb7f8 fix(mcp): emit one operator warning per DCR re-registration event
The stale-redirect path logged three warnings for a single re-registration: the staleness probe plus the reuse skip in both register_client_with_server and the persist race guard. The reuse-skip message is a mechanical consequence of the probe's decision, so it now logs at debug; the actionable warning that names both bindings and the re-authentication impact is emitted once by _persisted_dcr_redirect_uri_is_stale
2026-07-11 10:12:20 -07:00
Tin Chi Lo
f308bd99d9 fix(mcp): re-register DCR client when proxy origin no longer matches its registered redirect_uri
A dynamically registered (RFC 7591) OAuth client persisted onto the MCP server row is bound to the redirect_uri it was first registered with, but that binding was never recorded. After the proxy's public origin changed, every authorize paired the reused client with the new callback and the IdP rejected it permanently.

The DCR persist now records redirect_uris alongside the client identity. The admin register path treats a positive mismatch between the recording and the current callback as stale and re-registers a replacement client; rows without a recording (pre-existing installs and admin-configured clients) are grandfathered so upgrades never re-mint client_ids or orphan refresh tokens. The persist also writes client_secret and token_endpoint_auth_method explicitly as None when absent so the credential blob merge cannot pair a re-registered public client with the previous client's secret. Public register routes and non-admin callers keep existing behavior.

Closes #32473
2026-07-11 10:12:20 -07:00
Yuneng Jiang
c9259e4bf2
fix(auth): tolerate request objects without path_params in common_checks
The PATCH /team/{team_id} org-context wiring reads request.path_params to
resolve the team id from the path. A real Starlette Request always exposes
path_params, but common_checks is exercised with lightweight request doubles
that don't, which raised AttributeError. Read it defensively so a missing or
null path_params falls back to no path team id, matching the "not a bare team
route" outcome; real requests are unaffected
2026-07-11 09:44:31 -07:00
Yuneng Jiang
6875ca00d0
fix(team): bound json merge patch recursion depth
apply_json_merge_patch recurses into nested objects, which the repo's recursive_detector code-quality check flags because unbounded recursion over caller-supplied JSON has caused CPU/stack issues before. Cap the recursion at a depth far above any realistic team-metadata shape and reject deeper patches with a ValueError so a pathologically nested body fails closed instead of overflowing the stack, then register the function in the detector's ignore list alongside the other depth-bounded JSON walkers
2026-07-11 09:36:15 -07:00
Tin Chi Lo
f3bfafedc0 Merge branch 'litellm_internal_staging' into litellm_lit4337_dcr_bridge_ui 2026-07-11 09:13:15 -07:00
mateo-berri
81e639398a fix(fallback-generalizations): let exact cost-map entries beat capability rules across lookup-candidate ladders 2026-07-11 00:51:13 -07:00
yuneng-jiang
a4199d3c09
Merge pull request #32886 from BerriAI/litellm_/topnav-breadcrumbs-gateway-11e9c2
feat(ui): root the gateway breadcrumb in the AI Gateway selector
2026-07-11 00:28:14 -07:00
mateo-berri
1329036ca1 fix(fallback-generalizations): tolerate legacy remote rule schema and keep register_model cache-pricing inheritance 2026-07-11 00:27:45 -07:00
mateo-berri
1ccc3382d9 feat(fallback-generalizations): widen adaptive-thinking gate to any claude family at major 5+ 2026-07-11 00:27:45 -07:00
mateo-berri
77885779ca refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds 2026-07-11 00:27:45 -07:00
tin-berri
0c23c40627
Merge pull request #32752 from BerriAI/litellm_mcp_configured_client_for_passthrough
fix(mcp): rework the dashboard credential-field lifecycle so the OAuth app is upstream-scoped
2026-07-11 00:13:13 -07:00
Abhimanyu Kapur
02137b2d12 fix(proxy): guard delete_model router eviction on auto_router/ prefix
delete_model popped the auto_routers/complexity_routers registries by the deleted
deployment's model_name without checking it was actually an auto_router/* deployment.
Deleting a regular DB model that merely shares a name with a config-defined router
therefore evicted that router, which add_deployment never restores, leaving it
unroutable until a proxy restart. This is the same cross-tenant DoS clear_cache was
hardened against; mirror its auto_router/ prefix guard here.

Extracts _deployment_name_and_model to read model_name and litellm_params.model from
the deployment (delete_deployment returns the raw model_list dict at runtime despite
its Deployment annotation), and adds a regression test asserting a same-named config
router survives deletion of an unrelated regular model.
2026-07-11 00:08:47 -07:00
Yuneng Jiang
665c0dc508
feat(ui): always show the gateway selector with a discoverable Chat entry
The AI Gateway selector now always renders at the breadcrumb root, even with no plugins and Chat UI disabled, so the Chat feature stays discoverable. The Chat entry is always listed: clickable when enabled, and disabled with an "Admins can enable in Settings" hint when it is off.

Since the selector is now unconditional, the useViewSwitcherVisible hook and the section-crumb fallback added in the previous commit are removed
2026-07-11 00:02:00 -07:00
Abhimanyu Kapur
186d083bc7 fix(complexity_router): build semantic route index once under concurrent cold-start
Concurrent first requests each hit asyncio.to_thread to build the SemanticRouter
index, firing duplicate embedding calls for the static route utterances. Guard the
lazy build with a per-router asyncio.Lock (double-checked) so the index is
constructed exactly once regardless of how many callers race in cold.

Adds a regression test asserting ten simultaneous cold-start requests build the
index the same number of times as a single request, and reworks the fake embedding
router to count builds by how often a route utterance is embedded (robust to which
embedding path the library uses) while still recording sync-call thread ids for the
off-event-loop assertion.
2026-07-10 23:26:51 -07:00
Yuneng Jiang
ba17c4526e
feat(ui): root the gateway breadcrumb in the AI Gateway selector
The AI Gateway select (ViewSwitcher) now sits at the root of the DashboardHeader breadcrumb instead of on the right, so the top bar reads [AI Gateway select] > Page to match the redesign. It keeps the same dropdown, including the Chat / Chat UI options.

When no plugins are registered and Chat UI is disabled there is nothing to switch between, so the breadcrumb falls back to the static section crumb rather than rendering a dangling leading separator
2026-07-10 23:25:19 -07:00
Yuneng Jiang
922ed1ad30
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/data-table-design-exploration-f3f5e9
# Conflicts:
#	ui/litellm-dashboard/eslint-metrics.json
2026-07-10 23:14:14 -07:00
Yuneng Jiang
472b64cc62
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/patch-endpoint-1d2646 2026-07-10 23:14:13 -07:00
Yuneng Jiang
8f0dcb23f4
feat(team): let org admins reach PATCH /team/{team_id} like POST /team/update
Wire the coarse route gate so PATCH /team/{team_id} is reachable by exactly the roles that can call POST /team/update: proxy admins, org admins of the team's own organization, and JWT admins. Regular internal users and view-only proxy admins stay blocked, matching the existing endpoint

Because the team id lives in the path rather than the body, the org-context resolver now also reads it from path_team_id for the bare /team/{team_id} route, so an org admin's organization is resolved and injected the same way it already is for POST /team/update. /team/{team_id} is added to management_routes rather than the role-agnostic self_managed_routes; the latter would have opened POST /team/new to any authenticated user through the shared /team/{team_id} path pattern
2026-07-10 23:14:01 -07:00
Mateo Wang
c15891fc98
fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882)
Exact cost-map hits resolve before fallback-generalization rules, so the
mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the
bedrock-anthropic-claude-mid-conversation-system rule and hoisted
mid-conversation system messages, invalidating the prompt cache.
2026-07-10 22:51:41 -07:00
Abhimanyu Kapur
3c714ed7a6 feat(auto_router): keyword tier overrides and semantic keyword matching for the complexity router
Add deterministic keyword-to-tier overrides and optional embedding-based
(semantic) keyword matching to the complexity router, and surface both in the
Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]"
(complexity tiers + keyword overrides + semantic matching, the default) and
"Semantic Router [to be deprecated]" (the existing utterance-based router,
unchanged). Keyword-to-tier overrides resolve to the highest tier matched
rather than the first keyword matched, so match order no longer affects the
routing decision.

Backend:
- config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching,
  embedding_model, and match_threshold on ComplexityRouterConfig, with a validator
  requiring an embedding model and rules when semantic matching is on
- complexity_router: evaluate keyword rules before scoring; lexical matches escalate
  to the most-severe matched tier (order-independent), and semantic mode reuses
  LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity,
  falling back to the scorer when nothing matches
- model management: clear complexity_routers on cache reload so config edits take effect

Frontend:
- Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended
  by default, Semantic Router still available) and sends keyword_tier_rules plus
  the semantic settings on the recommended path, instead of flattening keywords
  into custom_technical_keywords
- client-side guard blocks submit when semantic matching is enabled without an
  embedding model or without any keyword tier rules, mirroring the backend validator
- moved the "How Classification Works" explainer below Custom Technical Keywords
  and above Keyword Tier Overrides
- remove the Test Connection action from the recommended flow, which can't build a
  valid pre-save payload for a router (leaves a TODO for a JSON preview / config
  test follow-up)

Tests cover lexical escalation, semantic matching via the real library with injected
embeddings, the semantic config guard, config validation, the reload-clear
regression, and the frontend payload builder
2026-07-10 22:05:29 -07:00
Mateo Wang
735a5bfc5b
test(models): assert capability fields on regional Azure gpt-5.6 entries (#32875) 2026-07-10 21:10:36 -07:00
Mateo Wang
3a62e5428f
fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867)
* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support

AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface
(thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed
model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5)
get the effort translated to a legacy thinking budget_tokens. Models with no
reasoning support have thinking/effort dropped under drop_params. And because
adaptive thinking carries no budget while the legacy form must satisfy Anthropic's
max_tokens > budget_tokens rule, the translated budget is capped below max_tokens,
dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass
through untouched.

This matters because clients like Claude Code speak native Anthropic /v1/messages
and send the adaptive interface unconditionally, regardless of the routed model.
The native passthrough previously only capability-gated the OpenAI-style
reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so
a pre-4.6 model rejected it with "This model does not support the effort parameter"
and the request failed. Claude Code already gets drop_params auto-set, so its
requests now succeed.

* test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests

Addresses review feedback on the max_tokens-too-small branch. Previously a
thinking-capable model whose max_tokens could not fit the minimum thinking budget
had thinking silently dropped regardless of drop_params, while a residual
output_config field in the same call still raised when drop_params was off. Gate
both consistently on drop_params: raise a clear error (naming max_tokens for the
undersized case) when drop_params is off, drop otherwise. Claude Code gets
drop_params auto-set, so it still succeeds.

Adds tests for the undersized-max_tokens raise, the residual output_config raise,
and the no-adaptive-interface passthrough on a non-adaptive model.

* fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts

The previous raise-when-not-drop_params behavior broke existing bedrock and vertex
messages tests: those providers already silently strip unsupported output_config
for pre-4.6 models (issue #22797) with no drop_params required, and the shared
parent transform raising pre-empted that. It also conflicted with the goal of
keeping requests working rather than failing them.

Make the reshape silent: translate effort to legacy thinking for thinking-capable
models, drop thinking for non-reasoning models, and remove only the consumed effort
key from output_config, leaving any residual (e.g. format) for provider subclasses
(bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the
review note about inconsistent drop_params handling by making every path uniform.

Updates the tests to assert the silent behavior and residual output_config
preservation.

* fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5)

Greptile caught a real bug: the early-return guard treated supports_output_config
as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises
supports_output_config (it accepts output_config.effort) but is not adaptive, so it
rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this
model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking
block raw, reproducing the exact failure the fix is meant to prevent.

thinking:{type:adaptive} and output_config.effort are independent capabilities.
Only early-return for adaptive-thinking models. For a model that supports
output_config.effort but is not adaptive, keep the native effort and drop only the
unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code
payload now returns 200 instead of 400.

Adds regression tests for Opus 4.5 with and without adaptive thinking.

* fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models

Claude Opus 4.5 advertises supports_output_config but not adaptive thinking,
so the early-return guard forwarded thinking.type=adaptive raw and Anthropic
rejected it. The guard now only skips true adaptive models; effort-only
requests on effort-capable models still pass through untouched. The
_map_reasoning_effort call is wrapped to surface unrecognized effort values
as a clean 400, matching _translate_reasoning_effort_to_anthropic

* fix(anthropic): fall back to legacy thinking when effort level unsupported

Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code
defaults to xhigh on newer models, so preserving that level raw gets rejected
by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model
and fall through to the budget translation for unsupported levels

* fix(anthropic): keep effort-only requests untouched for provider normalization

The xhigh fall-through consumed effort-only requests on effort-capable
models, breaking bedrock invoke's own normalization which clamps xhigh to
the model's ceiling after the base transform runs
(test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict
the fall-through to requests that carry adaptive thinking; effort-only
requests pass through so provider subclasses keep owning level clamping

---------

Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
2026-07-10 21:04:46 -07:00
yucheng-berri
60d557c930
fix(datadog): split log batches proactively under intake payload limits (#32860)
* fix(datadog): split log batches proactively under intake payload limits

* fix(datadog): size intake chunks with exact wire serialization
2026-07-10 20:54:42 -07:00
Mateo Wang
4baf326a39
fix(anthropic): strip @version suffix in _model_map_lookup_candidates (#32833)
vertex_ai/claude-opus-4-8@default (and sibling @default models) were
misclassified as non-adaptive because _model_map_lookup_candidates only
stripped provider prefixes but never the @<suffix> portion. The lookup
produced candidates like ["vertex_ai/claude-opus-4-8@default",
"claude-opus-4-8@default"], neither of which exists in model_cost, so
_is_adaptive_thinking_model returned False. LiteLLM then sent
thinking.type=enabled to a @default Vertex AI endpoint that requires
thinking.type=adaptive, resulting in a 400.

_strip_version_suffix now removes @<suffix> from each candidate,
adding the bare model name (e.g. "claude-opus-4-8") to the lookup
chain. Also adds supports_adaptive_thinking: true to the three
@default model_cost entries that were missing it as belt-and-suspenders.

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
2026-07-10 20:25:56 -07:00
Mateo Wang
1ab1515d9e
fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834)
adds a top-level guard in _increment_remaining_budget_metrics that returns early
when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config),
and per-entity guards in each _set_*_budget_metrics_after_api_request helper for
partial disabling. eliminates four async DB/cache round-trips per successful LLM
request when budget metrics are disabled.

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
2026-07-10 20:25:47 -07:00
Mateo Wang
249a999506
fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param to accept dict (#32835)
* fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param

- BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each
  chunk and raises litellm.APIError for type=error and type=response.failed events
  so callers see an exception instead of a benign stream chunk
- rate_limit* codes map to 429; client error codes (invalid_request_error,
  context_length_exceeded, etc.) map to 400; all other codes default to 500;
  raw integer codes are never used as-is as HTTP status codes
- ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]]
  to prevent Pydantic ValidationError on dict-typed param payloads silently
  dropping error events before any type inspection

* test(responses-api): add streaming iterator error event tests to CI-covered path

* test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths

* test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk

* fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting

insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees

_handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code

usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost

dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed

* fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status

Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and
5xx in-stream error events now raise MidStreamFallbackError carrying the
mapped APIError so the router's FallbackResponsesStreamWrapper triggers
mid-stream fallback and cooldown; non-retriable 4xx still raise APIError
directly. Status mapping now reads both the OpenAI error type and code
fields, so type-classified client errors (e.g. invalid_request_error with
code invalid_prompt) map to 400 instead of falling through to 500.

* fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting

MidStreamFallbackError was always raised with generated_content="", so the
router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk
and retried with the original input, streaming duplicated content to clients
that had already received partial output. The iterators now accumulate
response.output_text.delta text (mirroring chat's response_uptil_now) and pass
it as generated_content, letting the router build a continuation input via
_build_responses_continuation_input.

* test(responses-api): pin in-stream token limit error to raised APIError

---------

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
2026-07-10 20:25:38 -07:00
Mateo Wang
5e23a5ab05
fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831)
* fix(bedrock): gate in-place system role messages on model support for Claude Invoke

* feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule
2026-07-10 20:21:48 -07:00
Mateo Wang
ee6e8077ae
fix(responses): preserve reasoning_tokens through chat->responses usage translation (#32837)
* fix(responses): preserve reasoning_tokens through chat->responses usage translation

Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever
completion_tokens_details.reasoning_tokens was None or absent. Also change
OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that
re-instantiation without explicit reasoning_tokens no longer silently zeroes out
the field, and remove the same hardcoded zero from the mock_responses_api_response
initializer.

* test(responses): update assertions to match Optional[int] reasoning_tokens default

* fix(responses): preserve explicit reasoning_tokens=0 in usage translation

Align the reasoning_tokens guard with the is-not-None guards used for
text_tokens and image_tokens: a provider-reported zero passes through
while an absent value stays omitted.

---------

Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
2026-07-10 20:15:49 -07:00
Mateo Wang
4737e75c86
fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map (#32840)
* fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map

* test: use apac regional profile for cost-map fallback test since jp now has an entry
2026-07-10 20:05:37 -07:00
Krrish Dholakia
109193f26a
feat(router): add LLM-based classifier option to complexity router (#32169)
* feat(router): add LLM-based classifier option to complexity router

Adds classifier_type: "heuristic" | "llm" to complexity_router_config.
When set to "llm", the router calls a configured model (e.g. a small
model like haiku) via structured output to pick the complexity tier,
falling back to the existing regex/keyword scorer on any error, empty
response, or unparseable output.

* feat(ui): add classifier_type option to complexity router UI, fix edit flow

Adds an "Advanced: Classification Method" section to ComplexityRouterConfig
with a heuristic/LLM toggle, revealing a classifier model picker and timeout
when LLM is selected.

Also fixes the auto router edit modal, which never rendered the complexity
router UI at all (it only handled the semantic router), and the "Edit Auto
Router" button visibility check, which was gated on auto_router_config and
never matched complexity router deployments.

* fix(router): attribute classifier calls to caller, raise default timeout

Forwards the original request's litellm_metadata into the classifier's
acompletion call. Without it, the proxy's cost-tracking gate sees no
user_api_key/team_id/user_id and silently drops spend logging and budget
accounting for every classifier call, letting an authenticated user rack
up unaccounted provider spend via repeated requests.

Also raises the default classifier timeout from 400ms to 3000ms (400ms
undershoots real LLM latency and would silently degrade to the heuristic
scorer on most requests) and corrects the module/class docstrings, which
still claimed zero external API calls after the llm classifier path was
added.

* fix(ci): resolve ruff strict-budget and frontend-lint failures

- Use PEP 585 generics (dict/tuple/list) in the new aclassify/_classify_with_llm
  signatures instead of typing.Dict/Tuple/List, and suppress BLE001 on the
  intentionally broad except in aclassify's fallback path with a reason.
- Fix prettier formatting in ComplexityRouterConfig.tsx.
- Regenerate eslint-metrics.json (was stale after the classifier UI changes).

* fix(ci): regenerate stale eslint-metrics.json

* fix(router): strip parent budget reservation from classifier metadata

The classifier's internal acompletion call previously forwarded the
parent request's full litellm_metadata, including its budget
reservation (user_api_key_budget_reservation / user_api_key_auth).
That reservation belongs to the routed completion the classifier is
deciding on, not to the classifier call itself, so it's now stripped
while key/team attribution fields are still forwarded for spend
logging.
2026-07-10 18:55:34 -07:00
Yuneng Jiang
e9246e924e
feat(team): add PATCH /team/{team_id} with JSON merge patch semantics
Add a RESTful PATCH /team/{team_id} that partially updates a team using RFC 7386 JSON Merge Patch. team_id comes from the path, and metadata is merged with the team's stored metadata instead of being replaced wholesale the way POST /team/update does: an omitted key is preserved, key: null deletes it, and any other value overwrites, recursing into nested objects. Every other field behaves the same as POST /team/update

The handler delegates to the existing update path, so authorization, budget checks, system-managed-key stripping, metadata encryption, cache refresh, and audit logging are shared rather than reimplemented. POST /team/update is untouched, so the change is purely additive
2026-07-10 18:46:13 -07:00
Tin
5dbd608e17 fix(mcp): reset the remove-app checkbox on a server switch so it never deletes the next server's stored app 2026-07-10 18:24:54 -07:00
Yuneng Jiang
72cbd8a658
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/data-table-design-exploration-f3f5e9 2026-07-10 18:18:03 -07:00
Yuneng Jiang
4c4dd1f05b
refactor(ui): use shadcn Label for the DataTable filter field
DataTableFilterField rendered a bare native label element. Swap it for the
shadcn Label primitive so the filter drawer's field labels share the same
typography and disabled-state handling as the rest of the form primitives
2026-07-10 18:13:33 -07:00
Tin
86506775c9 fix(mcp): preserve a stored client on OAuth-resume restore, add the edit upstream-mismatch warning, and type the credentials field 2026-07-10 18:06:54 -07:00
Yuneng Jiang
ae55170343
fix(ui): show filter-select label and shape loading skeletons per column
The workflow-runs Status filter leaked the internal "__all__" sentinel as its
displayed value because Base UI's Select.Value renders the raw value when no
items map or children function is given. Drop the sentinel and use Base UI's
native null handling: a null "All statuses" item plus a placeholder, with an
items map so a real selection renders its capitalized label rather than the
raw status string

The shared DataTable rendered every loading-skeleton cell as one identical
half-width bar, which read as a rigid grid instead of the table beneath it.
Vary the skeleton width per column and add a per-column skeleton shape hint
(text or twoLine) on ColumnMeta so identity columns like the workflow "Run"
cell get a two-line skeleton that matches their real content
2026-07-10 18:05:49 -07:00
Krrish Dholakia
dacf1cfb26
fix: strip trailing slash from --base-url in lite CLI (#32845)
A trailing slash on --base-url (or LITELLM_PROXY_URL) produced
double-slash URLs like https://host//sso/cli/start, which 404s. Normalize
once in the CLI's top-level group callback so every subcommand benefits.
2026-07-10 18:00:00 -07:00