58 commits

Author	SHA1	Message	Date
yuneng-jiang	6372ca32c1	Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) ... This reverts commit 90f495f8dc.	2026-07-14 19:32:25 -07:00
yuneng-jiang	90f495f8dc	chore(ci): sync litellm_internal_staging into daily OSS branch (#33337) ... * feat(router): add LLM-based classifier option to complexity router (#32169) * feat(router): add LLM-based classifier option to complexity router Adds classifier_type: "heuristic" | "llm" to complexity_router_config. When set to "llm", the router calls a configured model (e.g. a small model like haiku) via structured output to pick the complexity tier, falling back to the existing regex/keyword scorer on any error, empty response, or unparseable output. * feat(ui): add classifier_type option to complexity router UI, fix edit flow Adds an "Advanced: Classification Method" section to ComplexityRouterConfig with a heuristic/LLM toggle, revealing a classifier model picker and timeout when LLM is selected. Also fixes the auto router edit modal, which never rendered the complexity router UI at all (it only handled the semantic router), and the "Edit Auto Router" button visibility check, which was gated on auto_router_config and never matched complexity router deployments. * fix(router): attribute classifier calls to caller, raise default timeout Forwards the original request's litellm_metadata into the classifier's acompletion call. Without it, the proxy's cost-tracking gate sees no user_api_key/team_id/user_id and silently drops spend logging and budget accounting for every classifier call, letting an authenticated user rack up unaccounted provider spend via repeated requests. Also raises the default classifier timeout from 400ms to 3000ms (400ms undershoots real LLM latency and would silently degrade to the heuristic scorer on most requests) and corrects the module/class docstrings, which still claimed zero external API calls after the llm classifier path was added. * fix(ci): resolve ruff strict-budget and frontend-lint failures - Use PEP 585 generics (dict/tuple/list) in the new aclassify/_classify_with_llm signatures instead of typing.Dict/Tuple/List, and suppress BLE001 on the intentionally broad except in aclassify's fallback path with a reason. - Fix prettier formatting in ComplexityRouterConfig.tsx. - Regenerate eslint-metrics.json (was stale after the classifier UI changes). * fix(ci): regenerate stale eslint-metrics.json * fix(router): strip parent budget reservation from classifier metadata The classifier's internal acompletion call previously forwarded the parent request's full litellm_metadata, including its budget reservation (user_api_key_budget_reservation / user_api_key_auth). That reservation belongs to the routed completion the classifier is deciding on, not to the classifier call itself, so it's now stripped while key/team attribution fields are still forwarded for spend logging. * fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map (#32840) * fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map * test: use apac regional profile for cost-map fallback test since jp now has an entry * fix(responses): preserve reasoning_tokens through chat->responses usage translation (#32837) * fix(responses): preserve reasoning_tokens through chat->responses usage translation Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever completion_tokens_details.reasoning_tokens was None or absent. Also change OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that re-instantiation without explicit reasoning_tokens no longer silently zeroes out the field, and remove the same hardcoded zero from the mock_responses_api_response initializer. * test(responses): update assertions to match Optional[int] reasoning_tokens default * fix(responses): preserve explicit reasoning_tokens=0 in usage translation Align the reasoning_tokens guard with the is-not-None guards used for text_tokens and image_tokens: a provider-reported zero passes through while an absent value stays omitted. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke * feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule * fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param to accept dict (#32835) * fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param - BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each chunk and raises litellm.APIError for type=error and type=response.failed events so callers see an exception instead of a benign stream chunk - rate_limit* codes map to 429; client error codes (invalid_request_error, context_length_exceeded, etc.) map to 400; all other codes default to 500; raw integer codes are never used as-is as HTTP status codes - ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]] to prevent Pydantic ValidationError on dict-typed param payloads silently dropping error events before any type inspection * test(responses-api): add streaming iterator error event tests to CI-covered path * test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths * test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk * fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees _handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed * fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and 5xx in-stream error events now raise MidStreamFallbackError carrying the mapped APIError so the router's FallbackResponsesStreamWrapper triggers mid-stream fallback and cooldown; non-retriable 4xx still raise APIError directly. Status mapping now reads both the OpenAI error type and code fields, so type-classified client errors (e.g. invalid_request_error with code invalid_prompt) map to 400 instead of falling through to 500. * fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting MidStreamFallbackError was always raised with generated_content="", so the router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk and retried with the original input, streaming duplicated content to clients that had already received partial output. The iterators now accumulate response.output_text.delta text (mirroring chat's response_uptil_now) and pass it as generated_content, letting the router build a continuation input via _build_responses_continuation_input. * test(responses-api): pin in-stream token limit error to raised APIError --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834) adds a top-level guard in _increment_remaining_budget_metrics that returns early when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config), and per-entity guards in each _set_*_budget_metrics_after_api_request helper for partial disabling. eliminates four async DB/cache round-trips per successful LLM request when budget metrics are disabled. Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(anthropic): strip @version suffix in _model_map_lookup_candidates (#32833) vertex_ai/claude-opus-4-8@default (and sibling @default models) were misclassified as non-adaptive because _model_map_lookup_candidates only stripped provider prefixes but never the @<suffix> portion. The lookup produced candidates like ["vertex_ai/claude-opus-4-8@default", "claude-opus-4-8@default"], neither of which exists in model_cost, so _is_adaptive_thinking_model returned False. LiteLLM then sent thinking.type=enabled to a @default Vertex AI endpoint that requires thinking.type=adaptive, resulting in a 400. _strip_version_suffix now removes @<suffix> from each candidate, adding the bare model name (e.g. "claude-opus-4-8") to the lookup chain. Also adds supports_adaptive_thinking: true to the three @default model_cost entries that were missing it as belt-and-suspenders. Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(datadog): split log batches proactively under intake payload limits (#32860) * fix(datadog): split log batches proactively under intake payload limits * fix(datadog): size intake chunks with exact wire serialization * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface (thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5) get the effort translated to a legacy thinking budget_tokens. Models with no reasoning support have thinking/effort dropped under drop_params. And because adaptive thinking carries no budget while the legacy form must satisfy Anthropic's max_tokens > budget_tokens rule, the translated budget is capped below max_tokens, dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass through untouched. This matters because clients like Claude Code speak native Anthropic /v1/messages and send the adaptive interface unconditionally, regardless of the routed model. The native passthrough previously only capability-gated the OpenAI-style reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so a pre-4.6 model rejected it with "This model does not support the effort parameter" and the request failed. Claude Code already gets drop_params auto-set, so its requests now succeed. * test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests Addresses review feedback on the max_tokens-too-small branch. Previously a thinking-capable model whose max_tokens could not fit the minimum thinking budget had thinking silently dropped regardless of drop_params, while a residual output_config field in the same call still raised when drop_params was off. Gate both consistently on drop_params: raise a clear error (naming max_tokens for the undersized case) when drop_params is off, drop otherwise. Claude Code gets drop_params auto-set, so it still succeeds. Adds tests for the undersized-max_tokens raise, the residual output_config raise, and the no-adaptive-interface passthrough on a non-adaptive model. * fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts The previous raise-when-not-drop_params behavior broke existing bedrock and vertex messages tests: those providers already silently strip unsupported output_config for pre-4.6 models (issue #22797) with no drop_params required, and the shared parent transform raising pre-empted that. It also conflicted with the goal of keeping requests working rather than failing them. Make the reshape silent: translate effort to legacy thinking for thinking-capable models, drop thinking for non-reasoning models, and remove only the consumed effort key from output_config, leaving any residual (e.g. format) for provider subclasses (bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the review note about inconsistent drop_params handling by making every path uniform. Updates the tests to assert the silent behavior and residual output_config preservation. * fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5) Greptile caught a real bug: the early-return guard treated supports_output_config as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises supports_output_config (it accepts output_config.effort) but is not adaptive, so it rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking block raw, reproducing the exact failure the fix is meant to prevent. thinking:{type:adaptive} and output_config.effort are independent capabilities. Only early-return for adaptive-thinking models. For a model that supports output_config.effort but is not adaptive, keep the native effort and drop only the unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code payload now returns 200 instead of 400. Adds regression tests for Opus 4.5 with and without adaptive thinking. * fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models Claude Opus 4.5 advertises supports_output_config but not adaptive thinking, so the early-return guard forwarded thinking.type=adaptive raw and Anthropic rejected it. The guard now only skips true adaptive models; effort-only requests on effort-capable models still pass through untouched. The _map_reasoning_effort call is wrapped to surface unrecognized effort values as a clean 400, matching _translate_reasoning_effort_to_anthropic * fix(anthropic): fall back to legacy thinking when effort level unsupported Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code defaults to xhigh on newer models, so preserving that level raw gets rejected by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model and fall through to the budget translation for unsupported levels * fix(anthropic): keep effort-only requests untouched for provider normalization The xhigh fall-through consumed effort-only requests on effort-capable models, breaking bedrock invoke's own normalization which clamps xhigh to the model's ceiling after the base transform runs (test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict the fall-through to requests that carry adaptive thinking; effort-only requests pass through so provider subclasses keep owning level clamping --------- Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> * test(models): assert capability fields on regional Azure gpt-5.6 entries (#32875) * feat(auto_router): keyword tier overrides and semantic keyword matching for the complexity router Add deterministic keyword-to-tier overrides and optional embedding-based (semantic) keyword matching to the complexity router, and surface both in the Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]" (complexity tiers + keyword overrides + semantic matching, the default) and "Semantic Router [to be deprecated]" (the existing utterance-based router, unchanged). Keyword-to-tier overrides resolve to the highest tier matched rather than the first keyword matched, so match order no longer affects the routing decision. Backend: - config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching, embedding_model, and match_threshold on ComplexityRouterConfig, with a validator requiring an embedding model and rules when semantic matching is on - complexity_router: evaluate keyword rules before scoring; lexical matches escalate to the most-severe matched tier (order-independent), and semantic mode reuses LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity, falling back to the scorer when nothing matches - model management: clear complexity_routers on cache reload so config edits take effect Frontend: - Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended by default, Semantic Router still available) and sends keyword_tier_rules plus the semantic settings on the recommended path, instead of flattening keywords into custom_technical_keywords - client-side guard blocks submit when semantic matching is enabled without an embedding model or without any keyword tier rules, mirroring the backend validator - moved the "How Classification Works" explainer below Custom Technical Keywords and above Keyword Tier Overrides - remove the Test Connection action from the recommended flow, which can't build a valid pre-save payload for a router (leaves a TODO for a JSON preview / config test follow-up) Tests cover lexical escalation, semantic matching via the real library with injected embeddings, the semantic config guard, config validation, the reload-clear regression, and the frontend payload builder * fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882) Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache. * feat(team): let org admins reach PATCH /team/{team_id} like POST /team/update Wire the coarse route gate so PATCH /team/{team_id} is reachable by exactly the roles that can call POST /team/update: proxy admins, org admins of the team's own organization, and JWT admins. Regular internal users and view-only proxy admins stay blocked, matching the existing endpoint Because the team id lives in the path rather than the body, the org-context resolver now also reads it from path_team_id for the bare /team/{team_id} route, so an org admin's organization is resolved and injected the same way it already is for POST /team/update. /team/{team_id} is added to management_routes rather than the role-agnostic self_managed_routes; the latter would have opened POST /team/new to any authenticated user through the shared /team/{team_id} path pattern * feat(ui): root the gateway breadcrumb in the AI Gateway selector The AI Gateway select (ViewSwitcher) now sits at the root of the DashboardHeader breadcrumb instead of on the right, so the top bar reads [AI Gateway select] > Page to match the redesign. It keeps the same dropdown, including the Chat / Chat UI options. When no plugins are registered and Chat UI is disabled there is nothing to switch between, so the breadcrumb falls back to the static section crumb rather than rendering a dangling leading separator * fix(complexity_router): build semantic route index once under concurrent cold-start Concurrent first requests each hit asyncio.to_thread to build the SemanticRouter index, firing duplicate embedding calls for the static route utterances. Guard the lazy build with a per-router asyncio.Lock (double-checked) so the index is constructed exactly once regardless of how many callers race in cold. Adds a regression test asserting ten simultaneous cold-start requests build the index the same number of times as a single request, and reworks the fake embedding router to count builds by how often a route utterance is embedded (robust to which embedding path the library uses) while still recording sync-call thread ids for the off-event-loop assertion. * feat(ui): always show the gateway selector with a discoverable Chat entry The AI Gateway selector now always renders at the breadcrumb root, even with no plugins and Chat UI disabled, so the Chat feature stays discoverable. The Chat entry is always listed: clickable when enabled, and disabled with an "Admins can enable in Settings" hint when it is off. Since the selector is now unconditional, the useViewSwitcherVisible hook and the section-crumb fallback added in the previous commit are removed * fix(proxy): guard delete_model router eviction on auto_router/ prefix delete_model popped the auto_routers/complexity_routers registries by the deleted deployment's model_name without checking it was actually an auto_router/* deployment. Deleting a regular DB model that merely shares a name with a config-defined router therefore evicted that router, which add_deployment never restores, leaving it unroutable until a proxy restart. This is the same cross-tenant DoS clear_cache was hardened against; mirror its auto_router/ prefix guard here. Extracts _deployment_name_and_model to read model_name and litellm_params.model from the deployment (delete_deployment returns the raw model_list dict at runtime despite its Deployment annotation), and adds a regression test asserting a same-named config router survives deletion of an unrelated regular model. * refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds * feat(fallback-generalizations): widen adaptive-thinking gate to any claude family at major 5+ * fix(fallback-generalizations): tolerate legacy remote rule schema and keep register_model cache-pricing inheritance * fix(fallback-generalizations): let exact cost-map entries beat capability rules across lookup-candidate ladders * fix(team): bound json merge patch recursion depth apply_json_merge_patch recurses into nested objects, which the repo's recursive_detector code-quality check flags because unbounded recursion over caller-supplied JSON has caused CPU/stack issues before. Cap the recursion at a depth far above any realistic team-metadata shape and reject deeper patches with a ValueError so a pathologically nested body fails closed instead of overflowing the stack, then register the function in the detector's ignore list alongside the other depth-bounded JSON walkers * fix(auth): tolerate request objects without path_params in common_checks The PATCH /team/{team_id} org-context wiring reads request.path_params to resolve the team id from the path. A real Starlette Request always exposes path_params, but common_checks is exercised with lightweight request doubles that don't, which raised AttributeError. Read it defensively so a missing or null path_params falls back to no path team id, matching the "not a bare team route" outcome; real requests are unaffected * fix(mcp): re-register DCR client when proxy origin no longer matches its registered redirect_uri A dynamically registered (RFC 7591) OAuth client persisted onto the MCP server row is bound to the redirect_uri it was first registered with, but that binding was never recorded. After the proxy's public origin changed, every authorize paired the reused client with the new callback and the IdP rejected it permanently. The DCR persist now records redirect_uris alongside the client identity. The admin register path treats a positive mismatch between the recording and the current callback as stale and re-registers a replacement client; rows without a recording (pre-existing installs and admin-configured clients) are grandfathered so upgrades never re-mint client_ids or orphan refresh tokens. The persist also writes client_secret and token_endpoint_auth_method explicitly as None when absent so the credential blob merge cannot pair a re-registered public client with the previous client's secret. Public register routes and non-admin callers keep existing behavior. Closes #32473 * fix(mcp): emit one operator warning per DCR re-registration event The stale-redirect path logged three warnings for a single re-registration: the staleness probe plus the reuse skip in both register_client_with_server and the persist race guard. The reuse-skip message is a mechanical consequence of the probe's decision, so it now logs at debug; the actionable warning that names both bindings and the re-authentication impact is emitted once by _persisted_dcr_redirect_uri_is_stale * ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI * refactor(ui): use TanStack Pacer debounce for the team keys search Replace lodash/debounce in TeamVirtualKeysTable with useDebouncedValue from @tanstack/react-pacer, matching the sibling VirtualKeysTable and PaginatedKeyAliasSelect which already debounce their key-alias search that way. Pacer is already a dependency, so this drops the odd-one-out lodash usage and keeps the search-debounce pattern consistent across the key tables. * fix(fallback-generalizations): cover bare Claude majors in baseline and routing, require claude- prefix in adaptive gate * fix(mcp): strip scheme default port from get_request_base_url netloc * feat(ui): typed openapi-fetch foundation (fetchClient) + first typed caller (useCustomers) (#29884) * feat(ui): add the typed openapi-fetch client (fetchClient) as the dashboard fetch foundation Introduces fetchClient (openapi-fetch) bound to schema.d.ts, used inside ordinary TanStack Query hooks so path/query/body types come from the proxy's OpenAPI spec. A small runtime registry feeds the client the base URL and auth header name (registered by networking) and the session token (published by AuthContext), so call sites carry no token plumbing; auth-header injection and ApiError mapping live in openapi-fetch middleware reusing deriveErrorMessage/ApiError from client.ts, and non-2xx maps to a thrown ApiError so query functions just read .data. The base URL default resolves from NEXT_PUBLIC_BASE_URL so a request still targets the right origin if it fires before networking registers its getter. AuthContext clears accessToken alongside the token on logout so no query fires unauthenticated after the session ends. Foundation only; callers migrate one at a time, each fully typed, in follow-up changes. * feat(ui): migrate useCustomers to the typed fetchClient Converts useCustomers from allEndUsersCall to fetchClient.GET("/customer/list"); the response is typed as LiteLLM_EndUserTable[] from the schema, so the hand-written Customer/CustomersResponse types are deleted. They were also inaccurate (allowed_model_region was string but is "eu"|"us", and a budget_id the table has no field for). No cast; the schema type flows to the one consumer. First caller on the new pattern. * fix(ui): route typed-client errors through the session-expiry handler The typed fetchClient middleware threw ApiError without invoking the handleError side effect that the legacy createApiClient wires via onError, so a migrated caller hitting an expired key no longer triggered the auto-logout. Add an error-handler seam to runtime.ts, register handleError from networking.tsx alongside the base-url/header getters, and call it in the middleware before throwing so both clients behave the same. Regression test asserts the handler fires with the derived message on non-2xx and stays silent on success * fix(ui): point the customers EndUser type at CustomerResponse The /customer/list response model was renamed to CustomerResponse on staging; the merged branch still aliased EndUser to LiteLLM_EndUserTable, so the exported type and its test mock had drifted from what the schema actually returns. CustomerResponse is also the accurate shape (it types allowed_model_region as 'eu' | 'us' and carries budget_id) * chore(ui): refresh eslint-metrics baseline after staging merge The recorded baseline predated the litellm_internal_staging merge, so its no-explicit-any and no-large-inline-object-arg counts were higher than the merged tree actually has. Regenerate via npm run lint:metrics so the gate reflects current reality * refactor(ui): source the typed client token from the session cookie, not AuthContext The typed client read its bearer from a runtime value that AuthContext pushed via setAuthToken, but migrated hooks gate enabled on useAuthorized, which decodes the cookie directly. Two independent derivations of the same cookie with different timing: on first load the query fires (useAuthorized sees the token) before AuthContext's async effect publishes it, so the first request goes out unauthenticated and only succeeds on a React Query retry. Make the token a registered getter like the base-url and header-name getters, reading the same cookie useAuthorized decodes, so the client's token and the gate can't diverge. Revert the AuthContext changes entirely; nothing is pushed from React state anymore. * test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857) * test(e2e): cover Langfuse logging.yaml P0 logs_spend cells Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat traffic and assert calculatedTotalCost matches StandardLogging response_cost and proxy spend. Also assert tool calls and applied guardrails land on the trace. Missing env or proxy is a hard failure, never a skip * test(e2e): use langfuse_otel callback for Langfuse spend coverage Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to Langfuse) instead of the classic langfuse SDK. Match generations named litellm_request by prompt marker and user_api_key_alias * test(e2e): require Langfuse spend assert; drop AGENTS.md Guardrail path no longer soft-gates logs_spend. Non-stream responses must return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md * test(e2e): fail when Langfuse spend is missing on guardrail path Always run logs_spend assertions for tool_permission; require positive x-litellm-response-cost on non-stream and positive /spend/logs spend * test(e2e): do not fall back to unmatched spend log rows poll_proxy_spend_for_key returns None when response_id or positive-spend filters match nothing, instead of silently using rows[0] * fix(complexity_router): use max aggregation for semantic keyword route scoring SemanticRouter defaults to mean aggregation across a route's utterances. Since each tier's route holds one utterance per configured keyword, a real semantic match on one keyword was averaged together with the tier's other, unrelated keywords and dragged below match_threshold — e.g. a MEDIUM tier with keywords [beep, boop, new york] never fired for a genuine "new york" paraphrase, because mean(sim_to_beep, sim_to_boop, sim_to_new_york) landed well under the threshold even though sim_to_new_york alone cleared it. Pass aggregation="max" so a tier matches if the query is close enough to any one of its keywords, not the average of all of them. Verified against live Voyage embeddings: raw cosine similarity for "new york" vs a paraphrase was 0.54 (above a 0.5 threshold), but the route scored 0.28 under mean aggregation and never matched; max aggregation fixes it. Adds a regression test with a tier holding one matching and two unrelated keywords, asserting the tier still fires; fails without aggregation="max". * refactor(auth): resolve PATCH team org-context from the route template Replace the request.path_params read (and its defensive getattr guard) with the route template. A real Starlette request always exposes path_params, but common_checks runs on lightweight request doubles that don't, so reading it directly forced a getattr workaround that only existed to tolerate those doubles. Instead, match the route template (/team/{team_id}) to identify the RESTful update route and take the team id from the last path segment. This drops the path_params dependency entirely, and because the template distinguishes the PATCH route from its single-segment siblings (/team/new, /team/list, ...), it also avoids a spurious team lookup those routes would otherwise trigger if we matched the resolved path shape alone. * chore(ui): remove eslint-metrics.json lint-count snapshot The eslint-metrics.json snapshot duplicated the violation counts already enforced by eslint-budgets.json. Keeping it current added a CI drift check, a pre-commit regenerate-and-flag step, and a standalone npm run lint:metrics script, none of which caught anything the budget gate did not, yet all of which failed noisily whenever the snapshot went stale. This drops the file and that machinery while leaving eslint-budgets.json as the actual ratchet gate * fix(complexity_router): preserve user_api_key_auth in sub-call metadata Removing user_api_key_auth entirely from classifier/embedding sub-call metadata (as _BUDGET_RESERVATION_METADATA_KEYS previously did) prevented _filter_deployments_by_model_access_groups from scoping those sub-calls to the caller's authorized access groups. An access-group-scoped caller could therefore reach embedding/classifier deployments outside their group. Only strip user_api_key_budget_reservation, which is the actual budget- reservation state that must not reach sub-calls. user_api_key_auth is now kept so access-group filtering works correctly for both the embedding path and the LLM classifier path. * test(e2e): drop vertex from pipecat tool smoke (#32925) Exclude vertex_ai from pipecat tool smoke; raw-ws tool_call_round_trip remains the Vertex source of truth. Also remove the Playwright key models dropdown suite so stage is not blocked by that UI harness * fix(complexity_router): sanitize budget reservation inside forwarded user_api_key_auth * fix(complexity_router): review hardening - blank keywords, router registry eviction, edit-modal controls - config: KeywordTierRule now strips and drops blank/whitespace keywords (a stray "" makes _keyword_matches match every prompt, silently forcing that tier for all traffic); still requires at least one real keyword to remain - frontend build_complexity_router_config: trim keywords and drop rules left empty so an unfilled "Add keyword rule" row no longer ships a rule the backend rejects with a 400 in the heuristic (non-semantic) flow, where the client-side semantic guard doesn't run - proxy clear_cache / delete_model: the auto_router/ prefix also covers quality_router/ and adaptive_router/, so pop the model_name from all four router registries (no-op where absent) instead of only auto/complexity; otherwise a DB quality_router's stale entry made reload raise "already exists" and abort, and adaptive left a leak - frontend ComplexityRouterConfig: only render the Keyword Tier Overrides and Semantic keyword matching sections when their change handlers are provided, so the edit-auto- router modal (which omits them) no longer shows interactive-but-dead controls * fix(anthropic): thread real provider through capability probes instead of pinning anthropic * docs(anthropic): note the two provider params' roles in _map_reasoning_effort * fix(anthropic): override custom_llm_provider in provider config subclasses so capability probes use the right namespace * feat(mcp): admit dcr_bridge oauth_delegate clients via a single envelope bearer * fix(mcp): admit dcr_bridge envelopes under the live key, not a frozen identity The bridge envelope sealed only user_id/server_id, and admission fabricated a UserAPIKeyAuth(user_id=...) with no object_permission, team_id, org_id, or key identity. Downstream MCP permission checks read the missing restrictions as unrestricted, so a caller holding a valid envelope for a restricted key could reach tools and servers that key was never granted, and a revoked key kept working until the envelope expired. Bind the hashed authorizing key into the envelope identity and reload the live UserAPIKeyAuth by it at admission via get_key_object, failing closed with a 401 when the key is missing, blocked, or expired. Authorization is resolved fresh per request instead of frozen at mint time, so current key/team/org and tool restrictions plus revocation are enforced. * fix(mcp): enforce team block and alias-priority token injection on bridge admission Two follow-ups on the envelope admission arm flagged in review. Team revocation bypass: _reload_admitted_key checked only the key's own blocked/expires, so blocking a key's team left every envelope minted under it live until expiry. Reload the team and reject a blocked team, mirroring common_checks, so a team block revokes its envelopes immediately. Caller-overridable upstream token: egress resolves the per-server auth header alias-first, but injection keyed under server_name, so for a server with a distinct alias a caller-forwarded x-mcp-{alias}-authorization sat at the higher-priority slot and paired the admitted identity with an attacker's upstream credential. Inject under alias-first so the sealed token owns the slot egress resolves. * fix(mcp): route bridge admission through the centralized policy gate and mirror the SCIM owner check * fix(mcp): import assert_never from typing_extensions for Python 3.10 * fix(mcp): surface real status from bridge admission policy gate instead of flattening to 401 Over-budget rendered 401 (should be 429), model-access and other typed failures collapsed to 401, and a transient DB outage was masked as an auth error. Mirror UserAPIKeyAuthExceptionHandler: budget maps to 429, a sub-check's own HTTPException/ProxyException keeps its status, a DB outage is a retryable 503, and only a genuinely unresolvable failure stays the fail-closed 401. * fix(rate-limit-v3): populate x-ratelimit-* remaining/limit values in standard_logging_object for streaming (LIT-4333) (#32711) Streaming requests return from common_request_processing before async_post_call_success_hook runs, so response._hidden_params.additional_headers never gets the v3 x-ratelimit-{descriptor_key}-{remaining|limit}-{rate_limit_type} entries. Prometheus / logging callbacks that read those values from standard_logging_object.hidden_params.additional_headers then see nothing; combined with the pre-existing gap that Prometheus reads from that same slot (LIT-2577 / PR #28816), per-key remaining RPM/TPM cannot be monitored for streaming traffic at all. Fix in three parts: - Stash the pre-call RateLimitResponse in the metadata channels the async success-logging callback inherits, alongside the existing top-level entry the non-streaming path reads. - Add async_logging_hook to the v3 handler. It fires in a distinct earlier loop inside async_success_handler (all callbacks' async_logging_hook complete before any async_log_success_event starts), so mirroring the pre-call snapshot into standard_logging_object.hidden_params.additional_headers and response._hidden_params.additional_headers here guarantees every downstream success callback sees the values regardless of registration order. Non-streaming keeps the existing async_post_call_success_hook write and this hook re-populates the same values idempotently. - Extract the shared `_merge_ratelimit_statuses_into_additional_headers` helper the non-streaming path already had inlined so both callsites emit the identical key shape. * fix(proxy): skip None model_name in clear_cache router eviction set * fix(mcp): map a DB outage during bridge key reload to a retryable 503 get_key_object's raw transport error propagated uncaught out of _reload_admitted_key as an opaque 500; classify it via the shared _raise_503_if_db_unavailable helper (also used by the live-policy gate) so a database outage is a retryable 503, while a key-not-found ProxyException stays the fail-closed 401. * test: remove live OpenAI fine-tuning job-creation test blocked by platform wind-down (#32933) OpenAI is winding down self-serve fine-tuning and the org can no longer create fine-tuning jobs (403 training_not_available; the CI key surfaces it as a 500 server_error), so test_create_fine_tune_jobs_async fails on every batches_testing run since 2026-07-11 and reruns never clear it. The request contract stays covered by the mocked create/list/cancel/ retrieve tests in the same file, and the deleted test's unique standard_logging_object assertions now run inside test_mock_openai_create_fine_tune_job. * refactor(anthropic): consolidate the provider fallback into a _resolved_provider property * feat: add lite auth print-token for Claude Code apiKeyHelper support (#32846) * feat: add silent CLI token refresh for apiKeyHelper support lite auth print-token prints a valid proxy credential for use as Claude Code's apiKeyHelper, transparently refreshing it first if the cached JWT is stale. This unblocks MDM-managed apiKeyHelper deployments (managed via `lite auth print-token`) that need silent mid-session credential rotation without restarting the client. Refresh capability is backed by a virtual key minted with an empty model list and cli_refresh metadata, kept strictly separate from the actual (short-lived, real-model-scoped) call credential -- so a leak of the credential that flows through every LLM request and subprocess env var can't also self-renew. The refresh flow is single-use: /sso/cli/refresh mints a fresh JWT + refresh token pair and blocks the presented refresh token immediately, so a replay can't mint a second pair from it. Server: /sso/cli/refresh (rotate) and /sso/cli/logout (revoke) endpoints. lite login now also stores a refresh token; lite logout revokes it server-side instead of only clearing the local file. * fix: allow non-admin users to hit CLI refresh routes; resolve apiKeyHelper base_url from token.json Found via a live end-to-end test against a real proxy + real Claude Code session: /sso/cli/refresh and /sso/cli/logout were unreachable for any non-proxy-admin caller, since Depends(user_api_key_auth) pulls in a route-RBAC gate that 403s any route not on an explicit allowlist. That made the feature unusable for actual end users, who authenticate as internal_user. Add both routes to internal_user_routes; the handlers already do their own fine-grained check (metadata.cli_refresh) same as /key/block does today. Also: `lite auth print-token` required an explicit --base-url/ LITELLM_PROXY_URL matching the stored token's origin, defaulting to localhost:4000 otherwise. But apiKeyHelper is configured bare (no flags), so this always mismatched a real deployment. Track whether --base-url was explicitly passed (via click's ParameterSource) and, if not, resolve the server from token.json directly instead of the CLI default. * test: mock refresh-token minting in test_cli_poll_key_tolerates_missing_user_row Landed on litellm_internal_staging after this branch's refresh-key minting change; needs the same mock as the other cli_poll_key tests since minting now runs unconditionally whenever a JWT is generated. * fix(ci): update test_cli_auth.py for refresh_token contract, regenerate schema.d.ts _poll_for_authentication now always includes "refresh_token" in its returned dict, and _handle_team_selection_during_polling returns a dict instead of a bare JWT string -- test_cli_auth.py predates this branch's refresh-token work and still asserted the old shapes. schema.d.ts regenerated via `npm run gen:api` to pick up the new /sso/cli/refresh and /sso/cli/logout routes (plus unrelated drift from other PRs merged since it was last generated). * fix(ci): apply CI's own schema.d.ts diff (enterprise routes I can't generate locally) Local `npm run gen:api` only sees OSS routes -- this machine's litellm_enterprise editable install points at a now-deleted temp directory, so it silently drops enterprise-only routes from the spec. Applied the exact diff CI's own generation produced instead of re-running the generator locally. * fix: close refresh-token race, fail closed on DB down, fix logout base_url Addresses Greptile review findings on the CLI refresh-token PR: - cli_refresh_token minted a new JWT + refresh token BEFORE blocking the presented one. Two concurrent requests bearing the same refresh token could both pass auth and both mint fresh pairs, yielding four live credentials from one consumed token. Now the presented token is consumed atomically first via update_many (only succeeding if it flips blocked from False/None to True); the loser gets count=0 and is rejected before anything is minted. - When prisma_client is None, refresh silently returned a new JWT without ever being able to mark the presented token consumed, leaving it valid indefinitely. Now fails closed with a 500 instead. - `lite logout` sent its revocation POST to ctx.obj["base_url"], which defaults to localhost:4000 when --base-url isn't passed -- the same bug print_token had before the base_url_explicit fix, just missed here. Now resolves the same way: trust the stored token's origin unless the caller explicitly overrode --base-url. * fix(ci): satisfy ruff format and narrow token_data type in logout * fix(security): never trust refresh-token metadata for authorization Addresses a real privilege-escalation path Veria flagged: cli_refresh_token read team_id, team_alias, and max_budget straight off the presented token's own metadata and used them to authorize the new JWT. Since any authenticated user can self-mint a virtual key with arbitrary metadata via the ordinary /key/generate endpoint, a self-forged key with {"cli_refresh": true, "team_id": "<any-team>", "max_budget": 999999999} would sail through _require_cli_refresh_token's only check (metadata.cli_refresh == True) and get a JWT scoped to a team the caller never belonged to, with a budget it never had -- full cross-team / budget bypass, and a removed team member could keep refreshing team-scoped sessions indefinitely. Metadata's team_id is now treated as an untrusted UX hint only: honored solely if the CALLER (identified by the authenticated key's own user_id, not client input) is a current member per a fresh get_user_object lookup. team_alias and max_budget are never read back from metadata at all -- team_alias comes from a live get_team_object lookup and max_budget is recomputed with the exact same capping logic the initial SSO login poll uses. _mint_cli_refresh_token no longer accepts or stores team_alias/max_budget, only the team_id hint. Added regression tests proving: a forged/stale team_id is dropped (falls back to no team, not silently honored), and a forged max_budget in metadata never reaches the issued JWT. * fix(ci): catch HTTPException specifically instead of bare Exception (BLE001) * fix: un-consume refresh token if minting the replacement fails Greptile flagged a real reliability gap: cli_refresh_token blocks the presented token atomically, then does several more DB calls before returning a replacement (user lookup, team lookup, JWT mint, new refresh-key mint). Since this endpoint exists specifically for fully unattended apiKeyHelper operation, a single transient failure in that window (DB hiccup, etc.) permanently stranded the user: their old token was already dead and no new one was issued, with no recovery path short of a full interactive browser re-login. Wrap that window in try/except; on any failure, best-effort revert the consumed token back to usable (blocked=False) before re-raising, so a retry can succeed. Standard compensating-action pattern since generate_key_helper_fn doesn't take an injectable transaction, so wrapping the whole thing in a real DB transaction isn't practical here. * fix(security): refresh key had unrestricted model access, not none Critical bug: _mint_cli_refresh_token used models=[] intending "no LLM access", but that's backwards in this codebase. Per _check_model_access_helper: `len(filtered_models) == 0 and len(models) == 0` -> all_model_access = True. An empty models list on a key with no team_id means UNRESTRICTED access to every model, not zero access. The CLI refresh token -- meant to be usable for nothing but silently exchanging itself for a new JWT -- was actually a fully unrestricted API key for its entire 90-day lifetime, completely undermining the whole point of keeping it separate from the short-lived call credential. Fixed with two independent layers: allowed_routes hard-restricts the key to exactly /sso/cli/refresh and /sso/cli/logout (the real enforced boundary, checked in the shared user_api_key_auth dependency for every route); models is set to an unmatchable sentinel string as defense-in-depth in case any code path only consults the models field. Added an end-to-end regression test that exercises the actual model-access-control function against a key shaped like the minted refresh token, rather than only asserting on what arguments were passed to the key-generation call -- the latter kind of test is exactly what let the original bug ship, since asserting `models == []` is equally consistent with "no access" and "unrestricted access" without checking what the access-control code actually does with that shape. Also: the compensating-rollback added for reliability un-blocked a consumed refresh token even when the underlying user no longer exists. That's a permanent, intentional rejection, not a transient failure -- un-blocking it would let a stale refresh token become valid again for a different account if the user_id is ever reused/re-registered. Moved the user-existence check outside the rollback-on-failure block so it stays permanently blocked. * refactor: rotate CLI refresh tokens via regenerate_key_fn instead of hand-rolled consume/rollback The refresh token is already a plain litellm virtual key, so rotation can delegate to the same atomic DB update /key/regenerate uses instead of a bespoke update_many + compensating-rollback dance. This makes silent CLI refresh an Enterprise feature, same as regular key regeneration. * refactor: replace CLI stateless JWT + refresh-key pair with one self-rotating virtual key The CLI previously minted two credentials on login: a stateless self-signed JWT for LLM calls, and a separate DB-backed refresh-only key (scoped away from ever calling an LLM) just to authorize minting a new JWT. Collapse this into a single real virtual key, used directly as the LLM bearer token and re-presented to /sso/cli/refresh to rotate its own secret in place. This also means the CLI session key now shows up in the Admin UI's Keys page and can be revoked/regenerated like any other key, rather than being an invisible, unmanageable stateless token. * refactor: drop silent CLI refresh, key just expires and requires re-login /sso/cli/refresh only ever benefited Enterprise deployments (regenerate_key_fn's gate), while everyone else already fell through to "re-run lite login" on failure. Cut the endpoint, the rotation logic, and the client-side refresh path entirely; print-token now just prints the cached key until it hits its LITELLM_CLI_JWT_EXPIRATION_HOURS duration, then fails fast telling the user to log in again. Session key itself is unaffected: still a real, revocable virtual key visible in the Keys UI, `lite logout` still revokes it directly. * fix(ci): regenerate schema.d.ts after removing /sso/cli/refresh route * revert: go back to stateless JWT, keep only lite auth print-token The virtual-key redesign (revocable, Keys-UI-visible credential) wasn't needed just to support print-token, and cost real server-side surface (a mint path, a logout-revoke endpoint, migrated tests/docs) for a property this repo doesn't need yet. Reverting cli_poll_key/_types.py/schema.d.ts back to the original stateless-JWT design; the only durable addition from this whole effort is `lite auth print-token` (reads the cached credential, prints it while fresh, fails with a clear message once it's past LITELLM_CLI_JWT_EXPIRATION_HOURS) plus the base_url_explicit plumbing it needs. `lite logout` goes back to clearing the local file only, since a stateless JWT can't be revoked server-side. * refactor: move CLI token freshness check to cli_token_utils, drop unnecessary renames Addresses review: the freshness check is a pure token-shape/timestamp util, not command logic, so it belongs alongside the other SDK-level CLI token helpers (load_cli_token, get_litellm_gateway_api_key) rather than in commands/auth.py. Also reverted a few incidental jwt_token/ session_key variable and string renames that weren't load-bearing. * fix(mcp): run the route gate on bridge admission so allowed_routes are enforced The envelope arm reloaded the identity and ran _run_centralized_common_checks but skipped RouteChecks.should_call_route, which the standard pipeline runs between the builder and common_checks. Because the centralized checks treat MCP as an inference route and never re-check allowed_routes, a key barred from MCP routes could mint an envelope at the token endpoint (not itself an MCP route) and replay it against MCP. Run the route gate before admitting, and clear the request-scoped budget_reservation, matching the wrapper's sequence; a disallowed route now surfaces the gate's own 403. * fix(proxy): reserve budget for tiered pricing Ensure tier-only models reserve their estimated request cost so concurrent requests cannot bypass exhausted budgets. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): bill tier-only deployments instead of $0 Route cost calculation to the deployment's router_model_id entry when it carries tiered_pricing but no flat per-token rate, so models like dashscope/qwen3.7-plus are billed via their tier table rather than the pricing-stripped shared alias. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(ui): convert activity metrics charts to shadcn/recharts (#32726) * refactor(ui): convert activity metrics charts to shadcn/recharts Swap the seven tremor AreaChart/BarChart sites in activity_metrics.tsx to the shared shadcn/recharts wrappers and switch CustomLegend/CustomTooltip to the ported versions in shared/charts. Chart props, colors, formatters, and legend behavior are unchanged; tests now assert on real recharts SVG output instead of tremor mocks. * fix(ui): restore tremor No data placeholder for empty AreaChart data * test(ui): scope activity metrics chart assertions to card titles instead of render order * feat(ui): extend topnav border across the sidebar header (#32920) Pin the sidebar header to the same 56px height as the dashboard topnav and give it a matching bottom border, so the two borders sit flush and read as one continuous line. Revert to auto height when the rail is collapsed so the stacked logo and toggle are not clipped. * fix(cost): coerce string tiered-pricing costs and share tier helper YAML-parsed tier costs can arrive as strings (e.g. "4e-07"), which broke arithmetic in the graduated tiered-pricing calculation. Coerce per-token costs to float in both the in-range and remaining-tokens paths. Move the tiered-cost helper out of the Dashscope module into a provider-neutral home so the proxy budget reservation no longer depends on a provider-specific module. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(anthropic): clarify the Opus 4.5 branch in adaptive-effort translation Add an inline comment explaining that the effort-capable non-adaptive branch in _translate_adaptive_effort_for_non_adaptive_model exists for models like Claude Opus 4.5 that accept output_config.effort but reject adaptive thinking, and why effort-only requests pass through while adaptive requests with an unsupported effort level fall through to the legacy translation. * fix(anthropic): translate raw adaptive thinking for chat completions on pre-4.6 models Clients that pass thinking={"type": "adaptive"} directly (not via the reasoning_effort alias) on the /chat/completions interface had it forwarded unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the translation already applied on the native /v1/messages passthrough (#32867): translate to legacy thinking={type: enabled, budget_tokens}, capped below max_tokens, dropping thinking when max_tokens can't fit even the minimum budget. Hoists the shared budget-capping helper onto AnthropicConfig so both paths use one implementation. * fix(proxy): reserve tiered budget all-or-nothing across all deployments Alibaba Model Studio (Dashscope) tiered pricing is all-or-nothing: the tier is selected by a request's total input tokens and every token, input and output, is billed at that one tier's rate. The reservation path used graduated slicing and, worse, picked the output tier from the output-token count, so a long-context request with a large output allowance reserved far less than the provider charges and could slip past a depleted budget. Select the tier from input tokens and apply its rates to all input and output tokens. Reservation also read tiered pricing from only the first deployment in a model group. A caller could hit an alias whose cheaper deployment was listed first and exceed the budget once routed to a costlier sibling. Estimate against every eligible deployment's pricing and reserve the maximum. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(complexity_router): log the cause of each routing decision The complexity router's info log didn't say what drove a routing decision. Literal and semantic keyword matches logged an identical "keyword rule fired" line (no way to tell which mechanism fired), and the scorer's line carried no consistent marker tying it to the same question. Emit one greppable line per decision naming the cause: literal_keyword_match, semantic_keyword_match, or complexity_scorer. The hook already knows which ran (the config's semantic_keyword_matching flag distinguishes lexical from semantic; the override-vs-scorer branch distinguishes keyword match from scorer), so this is label-only: no behavior change, no new types, no added latency. Adds regression tests asserting each decision path logs its cause; they fail if a label is swapped or the cause= marker is dropped. * feat(ui): add redesigned sidebar account menu (#32931) * feat(ui): add redesigned sidebar account menu Introduce SidebarAccountMenu, a sidebar-only account/logout menu built on shadcn Popover/Switch/Badge/Separator/Button, and wire it into leftnav in place of the shared UserDropdown. The panel has a LiteLLM header with the bouncing moon and a clickable version tag, Tier/Role/Email/User ID rows with copy actions, the five display toggles, and Logout. UserDropdown is left untouched so the control-plane / chat navbar keeps its existing menu. The version tag links to the same release notes page as the navbar tag, and the bouncing icon reuses the existing header animation gated by the Hide Bouncing Icon toggle. * test(ui): point account-menu e2e specs at the migrated sidebar menu The sidebar account menu moved from an antd Dropdown to a Base UI popover (SidebarAccountMenu), so the login, logout, proxy-logout-url, and internal user identity specs were still waiting on antd-era locators (.ant-dropdown, the popupRender wrapper class, the user-dropdown-panel test id, and a menuitem-role Logout). Point them at the new panel test id (sidebar-account-menu-panel) and the button-role Logout instead. The logout behavior is unchanged since both menus call the same useLogout handler. * fix(bedrock-converse): translate adaptive thinking for pre-4.6 models Follow-up to #32867 (native /v1/messages) and the /chat/completions commit earlier on this branch, extending the same adaptive-thinking translation to the Bedrock Converse path. Clients like Claude Code send thinking={type: "adaptive"} on every request. When routed via Bedrock Converse to pre-4.6 models (claude-haiku-4-5, claude-sonnet-4-5), this was forwarded as-is and rejected by the model. Mirrors the translation already applied on the /chat/completions and /v1/messages paths: map to legacy thinking={type: enabled, budget_tokens}, capped below max_tokens. Also fixes the missing custom_llm_provider arg in the chat completions path's call to AnthropicConfig._map_reasoning_effort. * fix(mcp): run proxy-wide pre-DB gates on bridge envelope admission The envelope arm bypasses user_api_key_auth, so it never ran pre_db_read_auth_checks (request-size and body-safety limits, the IP allowlist, and the general_settings route allowlist) that the normal MCP admission path runs before any key lookup. A caller blocked by IP or a disallowed proxy route could be admitted through an envelope where the same principal on the normal path is rejected. Run those gates before the envelope crypto, mirroring the pipeline's pre-DB ordering; a blocked IP or route surfaces its own 403. * fix(anthropic): pass resolved provider to adaptive-thinking check The rebase onto staging changed _is_adaptive_thinking_model to require custom_llm_provider (no default), so the one-arg call in the raw adaptive thinking branch raised TypeError at runtime for any /chat/completions caller sending thinking={type: adaptive}. Use self._resolved_provider, matching the reasoning_effort branch just below. Caught by Greptile. * test(bedrock-converse): cover adaptive-thinking drop when max_tokens too small Adds the regression test for the warning-drop branch in the Converse adaptive-thinking translation, mirroring the chat completions path's test_raw_adaptive_thinking_dropped_when_max_tokens_too_small. * fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712) * fix(guardrails): filter Add-Guardrail mode dropdown per provider The GET /guardrails/ui/add_guardrail_settings endpoint returned every GuardrailEventHooks value in one flat supported_modes list, so the Admin UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving Content Filter or Tool Permission with pre_mcp_call then failed with a 400 because those guardrails' server-side supported_event_hooks list excludes it. Expose each guardrail's supported hooks as a get_supported_event_hooks classmethod on CustomGuardrail (mirrors the existing get_config_model pattern) and have the endpoint iterate guardrail_class_registry to build a supported_modes_by_provider map. The UI Mode dropdown filters by that map when the selected provider is known and falls back to the global list otherwise. __init__ now sources its own supported_event_hooks list from the classmethod so the two sides can't drift. Also register BedrockGuardrail, ToolPermissionGuardrail, lakera, lakera_v2, and presidio in guardrail_class_registry so they participate in the map (they were previously only in guardrail_initializer_registry and had no class-registry entry). Behavior change: guardrails that previously had no supported_event_hooks declared (aim, javelin, azure/text_moderation, cato_networks, crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx, prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai, lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now validate the configured mode at instantiation. Existing configs where the mode was silently a no-op will fail at proxy startup with a clear validation error rather than running as a broken guardrail. Resolves LIT-4226 * fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form Address Greptile P1 (startup break) and P2 (edit form UX): LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported event_hook, unchanged behavior for the guardrails validated pre-PR). Setting it to false logs a warning and continues, giving deployments an opt-out while they fix configs that now surface as errors instead of silently no-op'ing. Regression test covers both modes. Edit form now surfaces the currently-saved mode even when it is not in the filtered per-provider list, so a legacy row (e.g. content_filter saved with pre_mcp_call before this fix) no longer disappears from the dropdown; the option renders with a 'not supported by <provider>' note so the user knows to pick another. * fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint Audited every get_supported_event_hooks classmethod against the hooks each guardrail's own tests exercise and its handler methods. Five were too narrow and their tests caught it in CI: rubrik gains pre_call, presidio gains during_call and pre_mcp_call, prompt_security, onyx and qualifire gain during_call. The remaining classes match either their original __init__ declarations or their exercised modes exactly. Cursor review fixes: the Add form now drops selected modes the new provider does not support when the user switches providers, so a pre_mcp_call selection cannot ride along into a provider that rejects it at save; the edit form handles list-shaped stored modes instead of treating mode as always a string. Extracted shared toModeArray and getSupportedModesForProvider helpers into guardrail_info_helpers so both forms use one implementation, typed the remaining any usages in both forms, removed nested ternaries, and committed the ratcheted-down eslint metrics and pruned suppressions * fix(proxy): reserve tiered output at the higher reasoning rate Some tiered Dashscope models price reasoning output above standard output (output_cost_per_reasoning_token > output_cost_per_token). The reservation charged all output at the standard rate, so a reasoning-heavy request reserved too little and concurrent calls could exceed the budget before reconciliation. The reasoning share is unknown before the request runs, so reserve every output token at the higher of the two configured rates, for both tiered and flat pricing. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(ui): convert user agent and per-user usage charts to shadcn/recharts (#32725) * refactor(ui): convert user agent and per-user usage charts to shadcn/recharts Swap the tremor BarChart import for the shared shadcn/recharts wrapper in user_agent_activity.tsx (DAU/WAU/MAU charts) and per_user_usage.tsx (usage distribution histogram). All chart props are unchanged; the wrapper exposes the same tremor prop surface with matching defaults. Extend user_agent_activity.test.tsx and add per_user_usage.test.tsx with parity assertions on the real recharts SVG output: bar series per category, stacked x positions, resolved fill colors, axis bucket labels, legend text, and value formatter output on axis ticks. Remove the dead ResizeObserver polyfill in user_agent_activity.test.tsx now that the scoped global mock in tests/setupTests.ts renders charts, which also lowers the no-explicit-any metric by one. * test(ui): harden bar x-position parsing against recharts path format * feat(ui): working Test Connection for the complexity auto router The consolidated auto-router tab dropped the Test Connection button because the shared prepareModelAddRequest helper returns an empty array for an auto router (it has no model_mappings), so the caller crashed destructuring result[0].litellmParamsObj. That is the crash in #31590 and the open PR #31794. #31794 only silenced the crash by pointing the test at auto_router/complexity_router, which is not a provider model, so the /health/test_connection health check (a real litellm.ahealth_check completion) would still error. Bring the button back and make it meaningful: an auto router dispatches to saved model groups, so Test Connection now probes those directly. It builds a deduped target list from the configured tiers (tiers sharing a model group collapse to one probe) plus the embedding model when semantic keyword matching is on, then runs a live /health/test_connection against each and shows per-target pass/fail. This never touches prepareModelAddRequest, so the original destructure crash cannot recur. Scope is the recommended complexity router only; the to-be-deprecated semantic router is untouched. No backend changes. Supersedes #31794. Resolves #31590. * refactor(ui): extract a shared CopyButton and fix the sidebar copy confirmation (#32945) * fix(ui): show sidebar copy confirmation only on a successful write The sidebar account menu's copy button switched to the checkmark synchronously, before the clipboard write settled, so it confirmed a copy that never happened when navigator.clipboard was undefined on non-secure origins or when writeText rejected. The handler now guards navigator.clipboard, awaits the write, and flips to the checkmark only on success Also updates the header accent emoji in the same menu * refactor(ui): extract a shared CopyButton for the sidebar account menu The copy-icon-to-checkmark pattern was hand-rolled in several places, including the sidebar account menu whose private copy button held the false-confirmation bug. Extract a single canonical CopyButton into components/shared, built on the Button primitive with a guarded and awaited clipboard write so the checkmark appears only on a real success, and have SidebarAccountMenu consume it The success and failure-mode coverage now lives in the shared component's own test; the sidebar test keeps one case asserting the email row is wired to it * fix(ui): probe auto-router tiers via real proxy routing, not /health/test_connection Live testing showed the first cut was broken: /health/test_connection merges {...configParams, ...requestParams}, so passing the public model_group name as the request model overrode the resolved provider model and every tier failed with "LLM Provider NOT provided". The frontend only has the public group name, not the underlying litellm_params, so it cannot build the request that endpoint needs. Switch to testing each model group the way production actually routes it: send a minimal request to /v1/chat/completions (or /v1/embeddings for the embedding model) by public group name through the shared apiClient. The router resolves the group, credentials, and provider itself, so a green row means the tier is genuinely reachable. Verified live: voyage embedding returns 200, a tier with a bad key returns the real provider auth error. Also address Greptile feedback: rows now update progressively as each probe settles instead of all at once, and TIER_ORDER is derived through a `satisfies Record<keyof ComplexityTiers, null>` guard so adding a tier without listing it is a compile error. * fix(completion): forward aws credential kwargs into litellm_params so the responses bridge keeps WIF auth Chat-completions requests to responses-only Bedrock Mantle models are bridged to the Responses API, but completion() forwarded only aws_bedrock_project_id into get_litellm_params, so aws_role_name, aws_web_identity_token, aws_session_name and the other SigV4 credential kwargs never reached sign_request and botocore fell back to the default credential chain ("Bedrock Mantle auth failed: no Bearer token and no usable AWS credentials"). Forward the whole AWS credential kwarg family, extracted from the OPTIONAL_KWARGS_KEYS set get_litellm_params already supports. * refactor(ui): convert entity usage and usage page charts to shadcn/recharts (#32729) * refactor(ui): convert entity usage and usage page charts to shadcn/recharts Swap the tremor BarChart/DonutChart render sites in EntityUsage, SpendByProvider, TopKeyView, TopModelView, KeyModelUsageView and UsagePageView to the shared shadcn/recharts wrappers. Convert the two sole-chart Daily Spend cards and the KeyModelUsageView card to the shadcn Card primitives. Close the donut parity gap with strictly additive optional DonutChart props: showLabel/label render a center total (tremor showed valueFormatter(sum) by default) and startAngle/endAngle forward to the Pie so both provider donuts keep tremor's clockwise-from-12 layout. Defaults preserve the previous wrapper behavior. DailyData and two site-local row types move from interface to type alias so they satisfy the wrappers' Record<string, unknown> constraint; interfaces lack implicit index signatures. Tests now assert on real recharts output: bar/sector counts, cyan fills, axis labels, donut center totals, and the TopKeyView bar-click drill-down into the key info modal. The dead tremor chart mocks in UsagePageView.test.tsx are removed and lint metrics/suppressions are regenerated for the dropped tremor imports. * fix(ui): compute donut center label only when shown and assert Model Usage renders as a card title * refactor(e2e): bucket rate limits, budgets, and spend tracking under quota_management * fix(e2e): name the route the spend_calculate registry cell actually exercises * refactor(e2e): move budgets and spend_tracking suites under quota_management * test(e2e): cover key rpm/tpm rate limiting, window reset, and pacing headers * test(e2e): assert the tpm block at its exact token crossing instead of a call-count heuristic * test(e2e): fail the rpm reset test when the limiter resets early * test(e2e): name the tpm window deadline's latency margin * refactor(e2e): model the tpm spend loop's two outcomes as values * test(e2e): source the ratelimit suite's model from E2E_CHEAP_ANTHROPIC_MODEL * feat(guardrails): add pre_mcp_call support to Content Filter (#32936) * feat(guardrails): add pre_mcp_call support to Content Filter * test(guardrails): cover canonical MCP key gate under pre_mcp_call mode * fix(guardrails): scan MCP arguments per value and gate mixed-mode scans by call type * fix(guardrails): cap MCP argument scan depth and register the walker with the recursion detector * test(guardrails): update LIT-4226 UI settings tests for content filter pre_mcp_call support * fix(guardrails): use builtin generics in MCP scan annotations to satisfy strict-rule budget Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(bedrock): allow bedrock-mantle:CreateInference in the web identity session policy * fix(ui): drop max_tokens from the auto-router connection probe max_tokens=1 makes reasoning models (o1/o3/...) return a 400 "max_tokens reached" because reasoning tokens count against the cap, so a reachable reasoning tier showed a false failure in Test Connection. Live-verified: o3 400s with the cap and succeeds without it. Extract the request shape into a pure buildModelGroupTestRequest and cover it with a test asserting the chat body carries no max_tokens (or max_completion_tokens), so this regression is caught in unit tests instead of only against a live reasoning model. * test(main): assert the responses bridge forwards static aws keys as well as web identity params * bump: litellm-proxy-extras 0.4.75 -> 0.4.76 (#32957) * feat(proxy): add expires filter to GET /key/list (#32953) * feat(proxy): add expires filter to GET /key/list Add an opt-in expires query param to GET /key/list so callers can fetch only expired or only active keys without paginating every page and filtering client-side. 'expired' matches keys whose expires is in the past (NULL expires excluded); 'active' matches keys that never expire or expire in the future. Omitting the param preserves existing behavior for every caller. An unrecognized value returns HTTP 400 rather than silently returning all keys. The filter is pushed to the database via the existing Prisma where builder so callers avoid pulling the full key table into application memory. Resolves LIT-3387 * refactor(proxy): declare VALID_EXPIRES_FILTER_VALUES before its first use * refactor(ui): colocate the usage view, keeping the shared usage components (#32952) Split for the usage (UsagePage) segment. Most of the folder is the usage page's own view, but four pieces are reused elsewhere and stay in @/components/UsagePage: TopKeyView (old-usage), KeyModelUsageView and value_formatters (activity_metrics), and the shared types (activity_metrics, chartUtils). The other 21 files move into usage/_components, preserving the folder structure. The external consumers import only the retained files, so they are untouched. The moved files' imports of the retained files become @/components/UsagePage paths, other escaping relative imports are absolutized, and lint suppressions are re-keyed for moved files only. No behavior change. * chore: update Next.js build artifacts (2026-07-11 23:35 UTC, node v20.20.2) (#32960) * test(e2e): cover model-aware mid-conversation system handling on Bedrock Invoke /v1/messages * docs(github): add QA runbook section to the PR template * docs(github): scope the QA runbook to tests/e2e edits and add example checklists * docs(github): shape QA runbook examples as node id plus behavior bullets * refactor(ui): convert projects page chart to shadcn/recharts (#32722) * fix(xecguard): use StandardLoggingGuardrailInformation in logging hook (#32911) XecGuard's async_logging_hook wrote a bare dict to standard_logging_object["guardrail_information"] while the typed contract is Optional[List[StandardLoggingGuardrailInformation]]. Readers that iterated the field walked dict keys, raised on info.get, or silently dropped the entry from guardrail usage tracking and spend-log writes Construct the typed entry and append it to the existing list or create a new one, matching the shared helper pattern. Record the configured guardrail name instead of a hardcoded "xecguard" and pass the GuardrailEventHooks enum for guardrail_mode * feat(ui): adopt openapi-react-query ($api) and convert useCustomers (#32949) * feat(ui): adopt openapi-react-query and convert useCustomers to $api Add openapi-react-query and expose $api = createQueryClient(fetchClient) alongside fetchClient. Rewrite useCustomers as $api.useQuery("get", "/customer/list", {}, { enabled, select }), which derives the query key from method + path (dropping the hand-written createQueryKeys entry and the manual key) and forwards the request signal for cancellation. The response type still flows from schema.d.ts as CustomerResponse[]. Tests assert the path, the admin/token enabled gate, and the empty-body select fallback. * test(ui): read the last render's options in useCustomers helper The lastCallOptions helper was named for the last call but read mock.calls[0]. Harmless while each test renders once, but it would silently assert against first-render options if a test ever re-renders. Read the final call instead. * refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface (#32968) * refactor(ui): colocate the usage view, keeping the shared usage components Split for the usage (UsagePage) segment. Most of the folder is the usage page's own view, but four pieces are reused elsewhere and stay in @/components/UsagePage: TopKeyView (old-usage), KeyModelUsageView and value_formatters (activity_metrics), and the shared types (activity_metrics, chartUtils). The other 21 files move into usage/_components, preserving the folder structure. The external consumers import only the retained files, so they are untouched. The moved files' imports of the retained files become @/components/UsagePage paths, other escaping relative imports are absolutized, and lint suppressions are re-keyed for moved files only. No behavior change. * refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface * docs(github): add Final Attestation and per-test sanity-check step to QA runbook * refactor(ui): convert endpoint usage charts to shadcn/recharts (#32723) * refactor(ui): convert endpoint usage charts to shadcn/recharts Adds a LineChart wrapper to the shared charts kit, mirroring the BarChart/AreaChart composition with connectNulls and curveType props, and converts EndpointUsageBarChart and EndpointUsageLineChart from tremor to the shared wrappers. Both endpoint chart tests now assert on real recharts SVG output instead of tremor mocks. * refactor(ui): drop unused endpointData prop from EndpointUsageLineChart * fix(ui): point endpoint chart test type imports at the UsagePage types alias after colocation move * fix(auto_router): filter embedding models out of tier selects, require all tiers, add inline validation The Add Auto Router complexity tab let chat models fill the embedding-model slot (and vice versa) since neither dropdown filtered on ModelGroup.mode, and submit only required at least one of the four tiers instead of all four. Adds getMissingTiersError alongside the existing getSemanticConfigError, and highlights unfilled tier/embedding selects inline once a submit attempt fails. * fix(model-cost-map): anchor the bedrock-claude-ids routing rule to the start of the id * fix(proxy-auth): deny provider-wildcard access inferred through an unrecognized model namespace * fix(auto_router): reset inline validation errors when switching router type * fix(auto_router): flag name field and tier fields together on empty submit Clicking Add Auto Router with the name empty returned early with only a toast, so blank tier selects never got their inline error state. The empty-name branch now sets showValidationErrors and triggers antd validation on the name field, so every unfilled mandatory field is flagged at once. Adds a regression test for the tab component. * feat(mcp): mint gateway-bound envelope at the token endpoint for dcr_bridge oauth_delegate * feat(mcp): seal the authorizing key hash in the dcr_bridge envelope The mint bound only user_id/server_id into the envelope, which gave admission no way to reload the caller's key and enforce its current restrictions. Seal the hashed authorizing key instead (a one-way digest, not a usable credential), so admission reloads the live UserAPIKeyAuth by it and the key's team/org/tool permissions and revocation apply per request. Extract the token endpoint's key resolution into a shared _resolve_active_litellm_key so the per-user token store (user_id) and the bridge mint (key hash) derive from one active-key-gated path, and fail the mint closed with invalid_request when no active key accompanies the request. * fix(mcp): return 502 not KeyError when a bridge upstream response lacks access_token The eager access_token = token_response["access_token"] extraction ran before the dcr_bridge branch, so a missing upstream access_token raised an unhandled KeyError and _bridge_grant_from_token_response's nil guard (which maps to a clean 502) was dead code. Move the extraction onto the non-bridge result path so the bridge branch reaches its 502 guard. * fix(mcp): let a keyless-user active key mint a bridge envelope _resolve_active_litellm_key gated on _active_key_user_id, which returns None both for blocked/expired keys AND for valid keys with no user_id, so a team-scoped or service-account key was wrongly rejected with invalid_request at bridge token exchange. Split the active-state gate (_key_is_active: blocked/expiry only) from the user_id extraction; the mint seals the key hash, not the user, and admission already handles a keyless-user key. The per-user token store still gets no user for such a key, as there is none to key a stored credential by. * style(mcp): use X | None annotations on the touched key-resolution helpers The keyless-user fix moved these signatures, so their pre-existing Optional[...] annotations counted against the diff and tripped the UP045 strict-budget gate. Modernize the four touched return annotations to the X | None form the gate wants; runtime behavior is unchanged. * fix(mcp): coerce numeric expires_in and make the active-key check total Two correctness gaps in the bridge mint. _bridge_grant_from_token_response only accepted an int expires_in, dropping a float (3600.0) or numeric-string ('3600') lifetime to None so the envelope fell back to its 1h cap and could outlive a shorter-lived upstream token; coerce it to a positive int (bool excluded). And _key_is_active called datetime.fromisoformat on the str|datetime expires outside the resolver's try, so a malformed stored expiry raised an unhandled 500 instead of the fail-closed invalid_request; it now fails closed (inactive) on an unparseable expiry. Regression tests cover int/float/string/bool coercion, the short-float TTL, and the malformed-expiry fail-closed path. * fix(mcp): harden the bridge token mint (multi-lens review pass) Findings from a full adversarial review of the mint path across security, correctness, error-handling, concurrency, and OAuth-protocol dimensions. - expires_in coercion is now total: int(float(...)) can raise OverflowError on Infinity / a giant numeric string, which escaped the ValueError/TypeError catch and 500'd the token endpoint. Unified to catch OverflowError too. - Resolve the litellm identity BEFORE exchanging the single-use upstream code, so a missing or transiently-unresolvable identity fails closed with invalid_request without burning the code (the mint re-resolves via a cache hit). - The no-identity failure is now an RFC 6749 5.2-shaped invalid_request (JSONResponse, top-level error, no-store) instead of a detail-wrapped HTTPException, matching the BYOK OAuth endpoint. - EnvelopeTooLarge (upstream token too big to seal) surfaces a 502, not a 500. - The upstream refresh_token is no longer sealed into the envelope: the edge never consumes it, so it was dead weight embedding a long-lived upstream credential in the client bearer and enlarging the envelope; refresh is a follow-up (a dedicated refresh-envelope). Security review found no exploitable defect (forgery, cross-server/user replay, leakage, confused-deputy all closed). Regression tests cover the OverflowError, the code-not-burned path, the RFC-shaped error, the 502, and the dropped refresh. * fix(mcp): close the burn-before-check gate for both grants and validate master_key first Follow-up to the pre-exchange identity gate, which I had only added to the authorization_code branch and which left the master_key check inside the mint (after the upstream exchange) - so the very burn-then-fail pattern it was meant to prevent still applied to refresh_token grants and to a misconfigured gateway. - Hoist a single pre-exchange gate above the upstream call that covers BOTH grant types: it fails closed (invalid_request) on an unresolvable litellm identity and 500s on an unset master_key BEFORE the single-use code or refresh token is exchanged/rotated, so a bad key or a misconfigured gateway never burns the upstream credential. - Report expires_in from the envelope JWT's own second-truncated exp (rounding the elapsed portion up) instead of the raw expires_at - now delta, so the client is never told the bearer is valid past the ~1s point admission already expires it. Regression tests assert the upstream exchange is never called on the no-identity refresh grant and the master_key-unset path, and that the reported expires_in does not overstate the JWT exp. * refactor(mcp): make the bridge delegate mint a phased failures-as-values pipeline The dcr_bridge oauth_delegate token mint validated its preconditions in two places: a pre-exchange guard inside exchange_token_with_server (master_key set, resolvable litellm identity) and an authoritative re-check inside the post-exchange _mint_bridge_delegate_token_response. Keeping the two in step by hand is what kept producing the same class of finding: a precondition guarded on one grant branch but not the other, master_key checked after the exchange on one path, identity resolved twice, and each failure raising an ad-hoc HTTPException with its own status and body shape. Model the mint as three phases whose failures are values. _prepare_bridge_mint runs before the exchange, checks every precondition once (master_key, then identity), and returns either a frozen _BridgeMintReady carrying the resolved key hash and the master-key-derived envelope keys, or a _BridgeMintError literal. Because every precondition lives in prepare, and prepare runs before the upstream POST, no failure can burn the single-use code or rotate a refresh token, for either grant type, by construction rather than by a guard we have to remember to keep in sync. _finish_bridge_mint runs after the exchange and has no preconditions left that can fail; its only failure values are properties of the upstream response itself (no usable access_token, or a token too large to seal). One mapper, _bridge_mint_error_response, turns each _BridgeMintError into an RFC 6749 section 5.2-shaped body with a status truthful about where the failure is (400 for the caller, 500 for gateway config, 502 for the upstream), with an exhaustive match plus assert_never so a new failure mode cannot be added without a matching status. Behavior is unchanged for the client. Every failure that previously raised now returns the same status as an OAuth error body, which is the correct token-endpoint contract; the three tests that asserted a raised HTTPException now assert the returned response. _exchange_for_bridge_server additionally asserts the identity resolver is awaited exactly once for a bridge server and never for a non-bridge one. * fix(mcp): let the bridge envelope report expires_in 0 at the jwt exp boundary _finish_bridge_mint floored the reported expires_in at 1. Admission expires the envelope against the JWT's second-truncated exp, so when the mint lands in the same second that exp falls on (a sub-second upstream lifetime, for instance), the true remaining life is 0 and reporting 1 tells the client the bearer lives one second past the point admission already rejects it. Floor at 0 instead so the reported lifetime never overstates the exp; the value still cannot go negative. The regression pins the boundary directly: minting at now=100.25 with a 1s upstream token seals exp=101, and the reported expires_in is max(0, 101 - ceil(100.25)) = 0. Under the old floor of 1 it reads 1, so the test fails on that mutation. Also drops the unused mcp_server parameter from _prepare_bridge_mint; identity and key derivation there never referenced the server. * refactor(mcp): make bridge-mint resolvers return tagged unions so status is truthful by construction Three findings landed together, all one defect: a resolution step crushed several distinct outcomes into a single None or a silent default, so the mint's error mapper could not tell them apart and assigned the wrong status. Identity resolution mapped a database outage to the same None as a missing credential, which the mint reported as 400 invalid_request, blaming the caller for a gateway outage while admission statuses the same outage 503/500. Lifetime coercion mapped an explicit non-positive expires_in to the same None as an absent one, so an upstream token the IdP reports as already dead was sealed into an hour-long envelope. And the refresh_token grant was run through the upstream exchange (which can rotate the client's upstream refresh credential) and its result then discarded, even though a bridge server seals no refresh_token and the client never holds one to present. Rather than add a mapping branch per finding, the fix changes the return types so a wrong status is not representable. Each resolution step now returns a precise tagged value instead of None: identity resolution returns a _ResolvedKey or one of no_active_key / unavailable / unresolvable, classified the same way admission's _reload_admitted_key classifies the same conditions; upstream-lifetime classification returns a positive number of seconds, "unspecified" (absent or unparseable, which the envelope caps), or "expired" (a parseable non-positive value, an already-dead token); and upstream-grant validation returns a typed grant or one of no_access_token / expired_lifetime. Thin exhaustive mappers (match plus assert_never) lift each vocabulary into one bridge-mint taxonomy of eight named failures, and a single _bridge_mint_error_response gives each its truthful RFC 6749 §5.2 status: 400 for the caller's missing credential or an unsupported grant, 503 for a transient auth-DB outage, 500 for a gateway that cannot resolve identity or is not configured, and 502 for an upstream response with no usable token, an already-expired lifetime, or a token too large to seal. Adding a failure mode now requires a new literal and a match arm the type checker forces, so the class of wrong-status bug cannot recur silently. The refresh_token grant is rejected in _prepare_bridge_mint before the exchange with unsupported_grant_type, so it can never rotate or consume the client's upstream refresh credential; renewal is re-running authorization_code, as the sealed refresh_token=None already intends. An absent or unparseable expires_in still mints a capped envelope (the by-design behaviour for an upstream that omits the field); only an explicitly-dead lifetime is rejected. Tests cover the resolver's three failure classes (including a real connection-error outage and a missing prisma_client), the mint statuses for each (503 before the upstream exchange, 500, 502 on an expired upstream lifetime, and a capped mint on an unknown one), and the refresh-grant rejection before any exchange. The three findings are mutation-checked: reverting each fix turns its regression test red. * fix(mcp): treat a positive sub-second upstream lifetime as alive, not expired _classify_upstream_lifetime decided "expired" from int(float(expires_in)), which truncates toward zero, so a positive fractional lifetime in (0, 1) became 0 and was misread as already elapsed. That rejected the mint with 502 in _finish_bridge_mint after the single-use upstream code had already been consumed, even though the upstream reported a positive remaining lifetime. Decide expired on the parsed numeric value rather than its truncated int, so only a genuinely non-positive value is expired. The envelope works in whole seconds and cannot represent a sub-second lifetime, so a positive value that truncates to 0 clamps up to the 1s floor instead of being rejected. Values >= 1 still truncate toward zero so the envelope never claims more life than the upstream stated, and NaN / Infinity / oversized input still read as unparseable ("unspecified"). Regression covers the classifier (0.5 and 0.001 clamp to 1, 1.9 truncates to 1, -0.5 stays expired) and the mint (a 0.5s upstream lifetime mints a 200 envelope rather than a 502); reverting to the truncate-then-check reddens both. * fix(auto_router): inline error for missing LLM classifier model Selecting the LLM classifier without picking a model only surfaced a toast on submit; the classifier model select now gets the same red outline and helper text as the tier and embedding selects once a submit attempt has failed. * build(dev-env): add make bootstrap and unprovisioned-checkout preflight to pre-commit * feat(router): random-pick multi-model complexity tiers (#32967) * feat(router): random-pick multi-model complexity tiers Tier pools already make sense without adaptive; stop pinning lists to index 0 and shuffle within the classified tier instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): format complexity router config Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): use PEP 585 types for tier pools Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(xecguard): sanitize scan result before recording it for logging (#32935) * chore: keep it brief * chore: keep it brief * docs(readme): point developer-mode setup at make bootstrap * chore: keep it concise * feat(router): add Router(plugins=[...]) routing-plugin pipeline (#32972) * feat(router): add Router(plugins=[...]) routing-plugin pipeline Runs a sequence of user-supplied plugins before the routing decision is made. Each plugin reads/mutates a RoutingContext (messages, candidate models, metadata, signals); the narrowed candidate list is enforced when picking a deployment, raising rather than silently falling back if a plugin narrows to zero candidates. Prototype for the routing-plugin pipeline discussed in #32168. * fix(router): use ruff-modern typing, add raw/structured messages to RoutingContext - Use dict/list/X|None instead of Dict/List/Optional in new code, staying within the ruff strict-rule budget ratchet - Extract the guardrail-translation message normalization ComplexityRouter already had into a shared resolve_structured_messages() helper (litellm_core_utils/prompt_templates/factory.py), reused by ComplexityRouter and the new routing-plugin pipeline instead of duplicating it - RoutingContext now exposes both raw_messages (as received) and structured_messages (normalized across chat completions / Anthropic messages / Responses API), mirroring CustomGuardrail.apply_guardrail's pattern, per review feedback on #32972 - Add direct unit tests for _run_routing_plugins and _filter_by_routing_plugin_candidates (router_code_coverage gate requires every router.py function be called by name somewhere in tests/) * fix(test): rename to test_router_routing_plugins.py router_code_coverage.py's AST scanner only inspects test files whose filename contains the substring "router" -- test_routing_plugins.py doesn't match (routing != router), so it silently skipped this file and flagged _run_routing_plugins/_filter_by_routing_plugin_candidates as untested despite the direct unit tests added for them. * fix(router): fail closed when plugins are configured but the resolved routing path can't run them Router.completion() (and other sync entry points) resolves deployments via the synchronous get_available_deployment(), which never runs async_pre_routing_hook and therefore never runs the routing-plugin pipeline. async_get_available_deployment() itself falls back to that same synchronous method for routing strategies without an async-native selector (e.g. legacy "usage-based-routing" v1). Both paths would let a policy plugin (e.g. a deny-all rule) be silently bypassed. Raise instead of silently proceeding when self.routing_plugins is configured and the sync path is reached, since applying the pipeline to every selector path is a larger change out of scope for this PR. Per review: https://github.com/BerriAI/litellm/pull/32972/changes/BASE..bdfb583c2c6f8df10004fb249e11629d41ce71fa#r3565373303 * feat(router): soft-floor adaptive mode for complexity router (#32947) * feat(router): soft-floor adaptive mode for complexity router Let complexity_router_config.adaptive=true Thompson-sample across the union of tier pools with a tier-distance penalty, and wire the existing adaptive post-call bandit so mis-tiered requests can still recover. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): reattach adaptive hooks for hybrid complexity Finalize was wiping every AdaptiveRouterPostCallHook and only re-registering standalone auto_router/adaptive_router deployments, so complexity adaptive=true never received bandit updates. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): drop unnecessary hybrid docstrings Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attribute adaptive feedback Credit user reactions to the model that produced the previous response while keeping current-response signals on the serving model Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): tune hybrid cold defaults Use the cost-weighted policy that beat equal-pool complexity in the full bakeoff, and make the committed harness compare identical tier pools Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): preserve hybrid cold quality floor Sample only unobserved models in the classified tier until feedback exists, then apply adaptive scoring without mis-penalizing models shared across tiers Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): bound feedback context cache Cap retained session feedback so unique session IDs cannot exhaust router memory Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): preserve exhaustion signals Include tool-result exhaustion in adaptive feedback and clear strict lint regressions blocking CI Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): remove stale owner cache Remove obsolete attribution state, tighten the embedded router type, and keep the test diff focused on adaptive behavior Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): centralize hook cleanup Use the callback manager to discover and remove adaptive hooks across every registered callback list Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(responses): continue MCP gateway tool turns from the final response and surface failures When a /responses request uses a hosted MCP tool (server_url: litellm_proxy/<label>) with store=true and the model calls a tool, the gateway auto-executes the tool and streams one logical response stitched from several upstream responses: an interim response whose only output is the function_call, then the post-tool answer B1 (correctness): every streamed event was pinned to the first round's response id, i.e. the interim response that carries the function_call but no tool output. The client then continued the next turn from that dangling response and the provider rejected it with "No tool output found for function call <id>", which on the streaming path surfaced as a silent empty completion. The fix adopts each auto-execute round's own response id (the cached id is reset when a follow-up round starts) so the client continues from the final round, whose stored input chain includes the function_call_output B2 (robustness): initial and follow-up call failures were swallowed; the stream emitted the mcp_list_tools discovery events and then closed with HTTP 200 and no output and no error. The fix stashes the failure, makes the initial call eagerly in aresponses_api_with_mcp so a pre-stream failure re-raises as a real 4xx before any SSE bytes are written, and emits a terminal error event when a follow-up call fails mid-stream Adds regression tests covering continuation exposing the final round's response id rather than the interim tool-call id, a follow-up failure emitting a terminal error event, and an initial-call failure being stashed for eager re-raise * ci(ui): report only error-level knip findings in CI (#32971) * feat(batches): track cost for unmanaged Bedrock batches, generalize the flag (#32315) * feat(batches): track cost for unmanaged Bedrock batches, generalize the flag CheckBatchCost skipped Bedrock batches whose unified_object_id is a raw model-invocation-job ARN, the same root cause previously fixed for unmanaged Vertex batches. Bedrock batches embed the model name in their s3:// input file name instead (litellm-bedrock-files-{model}-{uuid}.jsonl), so the same routing mechanism now derives the model from that layout and matches it to a configured bedrock deployment. track_unmanaged_vertex_batch_cost is renamed to track_unmanaged_batch_cost since two providers now share this mechanism. * fix(batches): parse Bedrock batch output and price with deployment model name Bedrock model-invocation-job results use modelOutput/error rows and short internal model ids that are not in the cost map, so unmanaged batch cost tracking logged tokens but $0 spend. Use deployment model name for pricing and add regression tests. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): walk custom_tool_call_output items in _content_utils (#32969) * fix(guardrails): walk custom_tool_call_output items in _content_utils * Change _OUTPUT_ITEM_TYPES to Frozenset type * fix(guardrails): use builtin frozenset generic for _OUTPUT_ITEM_TYPES annotation Frozenset is not a defined name (typing exports FrozenSet, the builtin is frozenset), so module import raised NameError and broke every proxy test suite. The builtin generic is valid on the supported python floor (3.10) and keeps the UP006 ruff-strict budget at its ceiling, which the typing alias would exceed * fix: show and allow editing team model aliases after team creation (#33047) * refactor(ui): rename OldTeams component file to Teams * fix: show and allow editing team model aliases after team creation * fix(ui): mark team model_aliases as nullable to match the prisma schema * fix(ci): bump pillow to 12.3.0 to resolve osv-scan CVEs (#33093) * fix(proxy): track unauthenticated pass-through requests in spend logs (#32410) Pass-through endpoints configured with auth=false reach the cost-tracking callback with no key/user/team/end-user, so _should_track_cost_callback returned False and the spend-log write was skipped, leaving the request out of request/usage logs. Track pass-through call types even when unauthenticated so the SpendLog row is still written. Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(lasso): send source.type=litellm for Used By attribution (#33090) Co-authored-by: Or Gershoni <org@lasso.security> * feat(mcp): generalize the bridge envelope identity to a key_hash or user_id subject The scripted two-header client mints under a virtual key it presents at the token endpoint (key_hash), but the interactive DCR client authenticates via SSO at the bridged authorize, which yields a user, not a key. Make EnvelopeIdentity a discriminated subject (subject_type key_hash | user_id) with key_hash_identity / user_identity constructors, and dispatch admission on it: a key_hash reloads the key, a user_id reloads the user and admits them as themselves (user-level budget and SCIM enforced via the same centralized gate; no team bound, since a user belongs to many teams or none). The interactive producer that mints a user_id envelope lands in the follow-up commit. * feat(mcp): interactive SSO sign-in for dcr_bridge oauth_delegate DCR clients Completes the oauth_delegate bridge for real DCR clients (Claude Code, Claude Desktop), which send no litellm key and cannot use the scripted two-header path. On the short-circuit bridge arm the gateway now captures the SSO-authenticated litellm user from the browser session at /authorize and seals it into the OAuth state; at /callback it seals that user plus the upstream code into a gateway authorization code the client echoes back; at /token it recovers the user, exchanges the real upstream code, and mints a user-subject envelope. The user identity captured in the browser thus rides to the back-channel token call with nothing stored server-side, and admission opens the envelope under that user. The scripted key_hash path is unchanged (raw upstream code, key from the request); without a session the browser is sent through login first. * fix(mcp): classify the user-subject reload's errors like the key path (503 outage, 401 missing) _reload_admitted_user mirrored only part of _reload_admitted_key's error contract: it caught ProxyException and HTTPException but had no arm for anything else, so a transient DB outage surfaced as an opaque 500 instead of the retryable 503 the key path guarantees, and a missing user surfaced as a 500 too. The missing-user case is the subtle one: get_user_object raises a bare Exception for a deleted user (not a ProxyException like get_key_object does for a missing key), so the ProxyException/HTTPException clause never caught it and the user_object-is-None branch it was supposed to hit is unreachable on the production path. Add the same except-Exception arm the key path uses, with the one deliberate difference the differing get_user_object contract requires: a database-service-unavailable error still raises the retryable 503, while a missing user or any other non-outage resolution failure fails closed as a 401 rather than propagating as a 500. The regression tests now drive the real behavior (get_user_object raising) rather than a None return that never happens in production, and cover both the 503 outage and the 401 missing-user paths. * fix(mcp): admit a user-subject envelope with the user's own MCP object permission _reload_admitted_user returned a bare UserAPIKeyAuth(user_id=...), so the shared get_allowed_mcp_servers found no key/team/object-permission grants and an interactive SSO client could admit successfully yet see zero tools on a normal (allow_all_keys=False) server. The key path returns the full key record whose object permission drives that computation; the user path dropped it. Resolve the user's own MCP object permission and put it on the returned auth, so the same get_allowed_mcp_servers the key path uses grants the user their litellm-granted servers and access groups. This reuses get_object_permission (the id-to-grants resolver keys and teams already use) and does not duplicate any permission logic; get_user_object does not load object_permission, so it is resolved from the user's object_permission_id the same way the key and team paths do. Only the user's own object permission is bound. A UserAPIKeyAuth carries a single team_id while a user may belong to many teams, so team-inherited MCP grants for a user are a follow-up: they need a many-teams union get_allowed_mcp_servers does not do off one auth object, and faking one here would be the kind of half-measure that spawns more bugs. Tests cover the user's object permission riding onto the admitted auth, and the existing admit/SCIM/missing-user/503 cases still hold. * fix(ui): respect litellm_key_header_name in BYOK credential save and workflow runs fetches (#33103) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant (#33040) * feat(ui): rebuild the Virtual Keys table on the shared DataTable (#32991) * feat(ui): rebuild the Virtual Keys table on the shared DataTable Replaces the hand-rolled Tremor table and bespoke toolbar/pagination on the admin Virtual Keys page with the shared DataTable: server-side sort, paginate, and filter, a sticky scrolling body, a search plus column-visibility plus filters toolbar, a right-side filter drawer, and a rows-per-page footer. A page header with the existing key icon carries the Create New Key action. Adds reusable, shadcn-default building blocks for the tables migrating onto the DataTable next: shared IdentityCell, ModelsCell, and SpendBudgetCell in shared/table_cells, plus a shared PageHeader. The models cell reveals overflow in a hover tooltip and the spend/budget cell uses the Meter primitive. All data and domain logic is preserved, including the useKeys query, team and org alias resolution, the user popover, and the KeyInfoView detail swap. The rich async Team/Org/Alias filters move into the drawer, and the toolbar search maps to the key-alias substring search. Status now also reflects key expiry alongside blocked and SCIM-blocked. The VirtualKeysTable tests are updated to the new markup and extended with focused coverage for each new shared cell * fix(ui): address Virtual Keys redesign review feedback Fold the status badge into the clickable Key cell and drop the separate Status column so a key's alias, secret, and status read as one unit. The Key cell is now the single click target that opens the key detail; the whole-row click is removed Migrate the filter drawer off AntD to shadcn. A new Combobox composed from Popover and Input backs the Team, Organization, and Key Alias filters, keeping search and the alias infinite-scroll Show $0.00 for zero spend instead of a hyphen, and extend the shared DataTable with badge, chips, and meter skeleton shapes so the loading state matches the loaded cells (status pill, model chips, spend meter) rather than uniform bars Fix key sorting: the Key column sent its column id "key" as sort_by, which /key/list rejects with 400. It now sorts by the backend field key_alias * fix(ui): use the shadcn base combobox and refine the keys filters and skeletons Replace the hand-rolled filter combobox with the supported shadcn Base UI combobox (ui/combobox, added via the CLI and reused through a small SearchSelect wrapper). Its vended input-group and textarea deps are written for React 19 (plain functions with ref-as-prop); this app is on React 18, where those subcomponents drop the refs Base UI passes for focus and anchoring, so InputGroupInput, InputGroupButton, and ComboboxTrigger are adapted to forwardRef. Those ui/ files now diverge from the registry, and a future shadcn add would overwrite the adaptation until the app moves to React 19. Adds class-variance-authority, which input-group needs Give loading skeletons a per-column renderSkeleton escape hatch on the shared DataTable and mirror the Key cell exactly (alias line, secret, status pill), so skeleton rows match the real rows instead of being shorter and simpler Resolve the automated review: the toolbar search and the drawer Key Alias filter both mapped to the key-alias query, so the search silently overrode the drawer value while its chip stayed visible. Consolidate to a single alias search in the toolbar (placeholder now "Search by key alias…") and drop the redundant drawer field. Re-add coverage for the Created By column's alias-over-email display Refine the Team and Organization filters: they match on name and id, so the labels read "Team" and "Organization" rather than "... ID", each option shows the name with the id on a muted second line instead of "name (id)", and the active-filter chip shows the friendly name * chore(ui): drop duplicate class-variance-authority, use the repo cva package in input-group * fix(mcp): relay upstream OAuth token and DCR rejections instead of a generic 500 An upstream token endpoint rejection (e.g. Google requiring client_secret even for PKCE web clients) escaped exchange_token_with_server as a raw httpx.HTTPStatusError, which the global exception handler turned into an opaque 500 Internal server error in the create-flow UI. The RFC 6749 section 5.2 error body the IdP sent (error, error_description, error_uri) is now relayed with the upstream's own 400/401 status; rejections outside the section 5.2 contract map to 502 so a broken upstream is not misattributed to the caller. The same relay covers the non-bridge DCR registration arm, and a 200 token response without a usable access_token now answers 502 instead of a KeyError 500. The catch wraps the post call itself because litellm's AsyncHTTPHandler raises MaskedHTTPStatusError at call time, which also made the pre-existing bridge-relay status check unreachable in production. The dashboard's token exchange error message now composes error and error_description so the form shows the IdP's reason * refactor(mcp): drop bridge relay status check made unreachable by the unified relay The try/except around the registration post now relays every upstream 4xx/5xx for both arms, so the bridge_relay status_code check could never fire; removing it addresses the Greptile P2 dead-code finding * fix(mcp): classify get_user_object's wrapped DB outage across the exception chain get_user_object catches every DB failure in a broad except and re-raises a bare ValueError (litellm/proxy/auth/auth_checks.py), so a real outage and a missing user look identical and the original error survives only as __context__. The dcr_bridge admission path keyed its 503-vs-401 decision on the exception type, so a transient outage during a user-subject reload surfaced as a 401 rather than a retryable 503, and the regression test injected a raw ConnectionError, a shape get_user_object never produces, so it passed on a fiction Add PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain, which walks __cause__/__context__ (bounded and cycle-safe) the PEP 3134 way, and route _raise_503_if_db_unavailable through it. Move the user's object_permission resolution inside the single classified try so an outage there is a 503 too, never an opaque 500. Pin get_user_object's wrapping with a contract test that drives the real function, and drive the reload tests with that same faithful shape so a chain-blind regression fails them * fix: redact async complete streaming response for custom callbacks (#33106) * fix response not being redacted for custom callbacks with streaming enabled * reduce code duplication * add unit test * fix: resolve lint violations in adopted redaction fix * fix: scope streaming response redaction to the opted-out custom logger --------- Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de> * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 (#33041) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 * fix(ui): address Virtual Keys redesign review nits (#33112) * fix(ui): address Virtual Keys redesign review nits Restore sorting by budget on the merged Spend / Budget column. The column now uses a new DataTableMultiSortHeader whose chevron opens a menu offering Spend and Budget in both directions plus Reset, so the progress-bar cell stays merged while the sort field becomes an explicit choice. Sorting is server-side, so the chosen field id (spend or max_budget, both accepted by /key/list) flows straight through as sort_by Fill the DataTable to its container width when column resizing is on. The table width was pinned to the sum of column widths, so hiding columns left an empty gutter on the right. It now keeps that width as a minimum and stretches to 100% on underflow while still scrolling on overflow, which also covers the same gap in TeamVirtualKeysTable since both share the component Drop the dark background box behind the page-header icon so the Virtual Keys header reads like the Teams header, and pull the 4-line inline filter lambda in SearchSelect out into a named matchesQuery helper Extends the DataTable and VirtualKeysTable tests to cover the new multi-field sort menu (field id maps to sort_by, active indicator, reset) and the fill-to-container width * fix(ui): emphasize the active field in the Spend / Budget sort header The merged Spend / Budget header always read "Spend / Budget" regardless of which field drove the sort, so after picking Budget descending there was no way to tell what was sorted without reopening the menu. The header now builds its label from the sort fields and emphasizes whichever one is active (bold, full-strength text) while muting the other, so the sorted column reads at a glance alongside the direction chevron. Drops the now-redundant title prop since the label is derived from the fields * fix(ui): remove w-full so the keys page content stops overflowing by 32px The virtual keys content wrapper used "w-full mx-4", which sets the width to 100% of the parent and then adds 16px of horizontal margin on each side, so its margin-box came to 100% + 32px and overflowed the scrollable main region by exactly 32px. That surfaced as a horizontal scrollbar along the bottom of the whole content area, under the pagination. A block div is already full-width, so dropping w-full lets mx-4 inset it correctly with no overflow * fix(ui): darken the clickable Key cell on hover so it reads as clickable The Key cell was the click target that opens the key detail, but hovering only faded the chevron in with no change to the cell itself, so there was no cue that the area was clickable. Give the cell a subtle muted background and a pointer cursor on hover. The button spans the full cell (a negative inline margin plus a matching width offset so the hover fill reaches both cell edges while the title stays aligned with the other columns) * fix(openai/responses): clamp max_output_tokens below API minimum (#33098) * fix(openai/responses): clamp max_output_tokens below API minimum Claude Code sends a max_tokens=1 warmup probe when running /model, which the Anthropic Messages -> Responses adapter forwards as max_output_tokens=1. OpenAI's Responses API rejects values below 16, so the probe failed with a 400. Clamp anything below the minimum up to 16 in map_openai_params so all Responses API entrypoints (direct, chat->responses, anthropic->responses) are covered. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(openai/responses): extract _enforce_min_max_output_tokens helper Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(prometheus): read v3 rate limiter remaining values for per-key model gauges (#33119) * fix(ui): drop w-full from page-content wrappers to remove 32px horizontal overflow (#33118) Several dashboard pages wrap their content in a div styled w-full mx-4, so the element's width is 100% of the scrollable main while mx-4 adds 16px of margin on each side. That makes the margin-box 100% + 32px wide, which overflows main by exactly 32px. Because main uses overflow-y-auto its overflow-x computes to auto, so the overflow surfaces as a horizontal scrollbar along the bottom of the whole content area under the pagination The wrapped block is already full width without w-full, so removing that one token keeps the layout and drops the overflow to 0. This is the same fix already applied to the Virtual Keys page in #33112, extended to the remaining pages that share the wrapper: Models + Endpoints, Tag Management, Organizations, Vector Stores, AI Hub, and Logging & Alerts * refactor(ui): migrate straightforward value debounces to react-pacer (#33042) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 * refactor(ui): migrate straightforward value debounces to react-pacer * feat(mcp): client-held refresh envelope for the dcr_bridge oauth_delegate flow A dcr_bridge oauth_delegate access envelope is capped at one hour, and until now the mode had no refresh at all: when the envelope expired the client had to re-run the interactive authorization_code flow. This adds a second client-held credential, the refresh envelope, so the client renews on a back channel and only re-authenticates when the refresh envelope expires or the upstream refresh token dies. The refresh envelope is a distinct llm_refresh_ credential that seals only the upstream refresh token (never the access token) bound to the same litellm identity and MCP server as the access envelope, under the same master-key-derived keys, with nothing stored server-side. Both envelopes now carry a signed kind claim ("access" or "refresh") that open() requires to match, so a refresh envelope can never open as an access credential even if its wire prefix is swapped (the prefix is not signed; the claim is). A refresh envelope presented at the MCP tool-call edge is not an access envelope, so admission fails it closed the same way it already fails any non-access bearer. At the token endpoint the authorization_code mint now returns a refresh envelope alongside the access envelope whenever the upstream returned a refresh token, and the refresh_token grant is supported for bridge servers: the client presents its refresh envelope, the endpoint opens it, re-validates the sealed litellm key so a revoked key cannot keep refreshing, unwraps the real upstream refresh token, exchanges it with the upstream IdP, and returns a fresh access envelope. Because the endpoint re-seals a refresh envelope only when the upstream returns a new refresh token, the design mirrors the upstream's own rotation policy rather than reinventing it: with a rotating upstream the client rotates and reuse is detected upstream; with a non-rotating upstream the original refresh envelope stands until its bounded 14-day TTL. Both preconditions and the unwrap run before the exchange, so a rejected refresh never consumes or rotates an upstream token. The pure envelope and credential layers stay side-effect free: mint/open share one signing, size, and kind gate across both envelope kinds, and every failure is a value. Tests cover the refresh round-trip, the kind-claim and server-id bindings, the revoked-key gate, upstream rotation carried through, the unwrap sending the real upstream token upstream, and edge rejection of a refresh envelope; the three security bindings are mutation-checked. Limitation documented in the PR: gateway-enforced refresh rotation with reuse detection would require server-side state, which this zero-custody mode omits by design, so the refresh envelope inherits the upstream's rotation posture plus gateway identity binding and a bounded TTL. * fix(mcp): reject a refresh envelope explicitly at the tool-call edge The live proof showed a refresh envelope presented at the MCP tool-call edge was rejected, but through the generic oauth2 arm ("expected a virtual key starting with sk-") rather than the bridge arm, because the admission routing gate is_bridge_envelope_shaped matched only the access prefix. The rejection was already fail-closed and never forwarded anything upstream, but the path was imprecise and the unit test modelled a route the real router did not take. Match either envelope kind in is_bridge_envelope_shaped so the bridge arm engages for a refresh envelope too, and have resolve_bridge_envelope return BridgeEnvelopeInvalid for it: a refresh envelope is a valid gateway credential but only ever presented back to the token endpoint, never usable to authenticate a tool call. Admission now fails it closed with the bridge arm's own 401 ("Invalid or expired credential"), live-verified, with the upstream never touched. is_bridge_envelope_shaped has a single caller (the admission routing gate), so the change is contained. * fix(mcp): SecretStr the unwrapped refresh token, drop the dead request arg, fail closed on a missing user Three review findings on the refresh path, addressed at the root: _BridgeRefreshReady.upstream_refresh_token was a plain str, the one credential in the envelope/bridge layer that escaped the SecretStr discipline every other one follows (RefreshCredential.refresh_token, UpstreamTokenGrant.access_token, EnvelopeKeys.signing_key). A repr or a traceback capturing a local _BridgeRefreshReady would have logged the raw upstream refresh token. It is now a SecretStr, carried as the SecretStr open_bridge_refresh_envelope already returns and unwrapped only at the point the exchange builds the upstream request body. _prepare_bridge_refresh took a request it never read; on the refresh path identity comes entirely from the sealed envelope, not the HTTP request, so the parameter was dead and misleadingly implied it read from the request the way the authorization_code prepare does. Removed, and the caller updated. _reload_active_user_by_id misclassified a missing user as unresolvable (500). This is the same root cause as the admission user-reload fix: get_user_object raises a bare Exception for a deleted user rather than a ProxyException, so its except-Exception arm must fail closed to no_active_key (which the refresh path maps to invalid_grant) for anything that is not a database-service-unavailable outage, rather than treating a missing user as an opaque gateway fault. Regression tests cover the missing-user and DB-outage classifications directly. * fix(mcp): make the dcr_bridge refresh path fail correctly on outages, dead tokens, and revoked owners Four fixes to the refresh_token grant for dcr_bridge oauth_delegate, surfaced by an adversarial pass over the exchange path Route the user-subject re-validation's outage check through the chain-aware classifier, so a transient DB outage (which get_user_object wraps in a bare ValueError) reports as unavailable (a retryable 503) rather than collapsing to no_active_key and an invalid_grant, matching how admission now handles the same wrapper When the upstream reports its own refresh token as already elapsed (refresh_expires_in non-positive), do not seal it into a full-TTL refresh envelope; return no refresh so the exchange degrades to an access-only response, mirroring how the access grant refuses an already-elapsed access token instead of capping it When the upstream rejects the sealed refresh token with 400 invalid_grant (revoked or expired at the IdP), return an RFC 6749 invalid_grant response so the OAuth client re-runs authorization_code, rather than surfacing the opaque upstream error it cannot act on Gate key-subject renewal on the owner's SCIM state, mirroring admission's _reject_if_admitted_owner_scim_deactivated, so an offboarded user cannot keep refreshing a still-active key; the check fails open on a missing owner or a DB blip so a key that outlives its owner record does not get wrongly revoked Each fix has a mutation-checked regression test * test(proxy): add regression tests for management_endpoints edge cases (#32976) Mutation testing surfaced branches in cost_tracking_settings and common_utils that the suite executed but never asserted on. Pin those behaviors with targeted tests: the returned (model, provider) from _resolve_model_for_cost_lookup for deployments carrying a custom_llm_provider and for deployments missing the litellm_params / model_info keys, plus the exact error-response bodies, the caller-identity lookup arguments, and the member and guard branches in common_utils. * fix(auto-router): correct Responses API tool_choice shape and propagate alias litellm_params (#32974) * fix(anthropic-messages): send bare-string tool_choice to Responses API, propagate router-alias litellm_params The Anthropic /v1/messages -> Responses API adapter always wrapped tool_choice in an object ({"type": "auto"}, {"type": "required"}), but the Responses API's tool_choice schema for these cases is a bare string ("auto"/"required"/"none"). Sending the object shape to an OpenAI-compatible backend (e.g. vLLM) fails Pydantic validation with a 400. The "none" case also fell through to "auto" instead of mapping to "none". Separately, litellm_params configured directly on a router-alias deployment (auto_router/complexity_router, adaptive_router, quality_router, or semantic auto_router) - e.g. cache_control_injection_points, drop_params - were silently dropped for every request through that alias. async_pre_routing_hook swaps `model` from the alias name to the selected tier/route's model before the deployment lookup runs, so the outbound call only ever merged in the tier deployment's own litellm_params, never the alias's. Register non-routing-config litellm_params from the alias deployment and apply them to the request whenever a pre-routing hook substitutes the model. * fix: satisfy ruff-strict-budget UP006 and router coverage checker Use builtin dict[...] generics instead of typing.Dict for the two new annotations introduced in the previous commit, since they pushed UP006 over the codebase ceiling in ruff-strict-budget.json. Add a direct unit test for _register_pre_routing_alias_overrides so the text-based router_code_coverage.py checker sees it exercised by name. * fix(router): replace alias-param denylist with a tight allowlist _PRE_ROUTING_ALIAS_RESERVED_PARAMS excluded router-init-only keys from the alias's litellm_params before forwarding the rest as request kwargs, but GenericLiteLLMParams also holds deployment-management fields (tpm, rpm, weight, tags, max_budget, budget_duration, use_in_pass_through, litellm_credential_name, ...) on the same object. Any of those left off the denylist would get silently forwarded as if they were request kwargs. Replace the denylist with a tight allowlist of exactly the two request-shaping params this feature exists for - drop_params and cache_control_injection_points - so unrelated management fields never reach the outbound call regardless of what else GenericLiteLLMParams grows to hold. * fix(router): re-register adaptive-alias overrides on set_model_list reload set_model_list() unconditionally clears pre_routing_alias_overrides on every call (e.g. /config/reload), but _finalize_adaptive_router_if_configured() skips rebuilding an AdaptiveRouter whose model_name already exists in self.adaptive_routers - so _register_pre_routing_alias_overrides() never ran again for an auto_router/adaptive_router alias after a reload, silently dropping its drop_params/cache_control_injection_points. Build the Deployment unconditionally and re-register its overrides even on the skip-existing-router path; only the (expensive) AdaptiveRouter construction itself stays skipped. * style: ruff format after merging litellm_internal_staging * fix(router): drop the alias-param allowlist, exclude only model Per review discussion: instead of a router.py-local allowlist of exactly which litellm_params an alias (auto_router/complexity_router, adaptive_router, quality_router, semantic auto_router) can forward to the request it routes, _register_pre_routing_alias_overrides now forwards everything except `model` (the alias marker itself, e.g. auto_router/complexity_router, never a real provider model). Router-init-only fields (complexity_router_config, complexity_router_default_model, auto_router_config, auto_router_config_path, auto_router_default_model, auto_router_embedding_model, adaptive_router_config, adaptive_router_default_model, quality_router_config, quality_router_default_model) now flow into request_kwargs unfiltered too. That's safe because litellm.completion()/acompletion() already strips anything in litellm.types.utils.all_litellm_params before building the provider request - added these 10 keys there, alongside the deployment-management fields (tpm, rpm, weight, ...) already listed. Verified live: without that addition, complexity_router_config lands in extra_body and ships raw to the provider; with it, it's stripped. This moves the "which fields aren't real LLM params" list from a router.py-local allowlist to the single existing global list every completion() call already depends on, instead of maintaining two. * refactor(router): look up alias litellm_params on demand instead of caching them _register_pre_routing_alias_overrides cached each alias's litellm_params into self.pre_routing_alias_overrides at deployment-init time, which required keeping that cache in sync with set_model_list() reloads - the exact bug the previous adaptive-router-reload fix was patching around (AdaptiveRouter survives a reload, but the cache didn't always get refreshed to match). Delete the cache and the registration method entirely. async_pre_routing_hook now looks up the alias's own litellm_params directly from self.model_list via self.model_name_to_deployment_indices at request time, the same model_list that's already correctly rebuilt on every set_model_list() call. No second piece of state to invalidate, so the reload staleness bug class isn't possible anymore, and it's less code than before. * fix(mcp): keep out-of-contract upstream error bodies out of client responses The token and DCR relays serve unauthenticated OAuth clients, so only the RFC 6749/7591 error fields may cross the trust boundary. A rejection body outside those contracts (HTML error page, proxy banner, stack trace) is now logged server-side, bounded, and the client response names only the upstream status. Addresses the Veria information-exposure finding * fix(mcp): re-request the sealed scope on a bridge refresh when the client omits it The refresh envelope seals the upstream scope as the scope to re-request (RefreshCredential), but _prepare_bridge_refresh dropped it, unwrapping only the refresh token, and the exchange added scope to the upstream request only from the client's HTTP form. A DCR/MCP client typically omits scope on refresh, so the sealed scope was never sent and a stricter upstream could narrow or drop the renewed token's scope Thread the sealed scope through _BridgeRefreshReady.upstream_scope and fall back to it when the client sends none; a client-supplied scope still wins, which RFC 6749 section 6 bounds to the original grant. The regression test drives a refresh where the client omits scope and asserts the upstream POST carries the sealed scope, mutation-checked against both the drop and the fallback * fix(ui): render the sidebar scrollbar with shadcn ScrollArea (#33124) * fix(ui): render the sidebar scrollbar with shadcn ScrollArea The sidebar navigation scrolled through a native overflow-y-auto container, so the browser drew its default scrollbar. It now scrolls through the shadcn ScrollArea primitive so the thumb matches the rest of the dashboard Switching to ScrollArea surfaced a latent styling gap. The Base UI scroll-area, tabs, and separator primitives rely on data-horizontal and data-vertical Tailwind variants that resolve to [data-orientation="horizontal"] and [data-orientation="vertical"], and those variants ship in shadcn's shared stylesheet. The project never imported it, so the classes matched nothing and the scrollbar collapsed to zero width. This adds shadcn as a devDependency and imports shadcn/tailwind.css, which also repairs the vertical tabs and separator styling. See shadcn-ui/ui#9196 for the upstream tracking issue * refactor(ui): inline the Base UI data-* variants, drop the shadcn dep The earlier fix imported shadcn/tailwind.css through the shadcn devDependency, which pulled 219 packages and tied the CSS build to shadcn's package exports (an open Turbopack-breaking bug, shadcn-ui/ui#10931). shadcn's model is that we own the components, so the custom variants those components depend on belong in our own stylesheet rather than a runtime dependency. This inlines the nine data-* custom variants and the no-scrollbar utility that the Base UI primitives reference into globals.css, and removes the shadcn package. * refactor(ui): migrate callback debounce sites to react-pacer with regression tests (#33043) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 * refactor(ui): migrate straightforward value debounces to react-pacer * refactor(ui): migrate callback debounce sites to react-pacer with regression tests * chore(ui): restore trailing newline in eslint-suppressions.json * test(ui): mock all pacer debounce hooks in VirtualKeysTable test * fix(ui): update merged debounce tests for OldTeams to Teams rename * fix(mcp): carry the requested scope forward when the upstream omits it on a bridge refresh The prior fix sent the sealed scope on a refresh, but the re-minted refresh envelope re-seals scope from the upstream response, and RFC 6749 section 5.1 lets an upstream omit scope when it is unchanged. So after one refresh whose response omitted scope, the new envelope sealed scope=None and every subsequent refresh dropped it, letting a stricter upstream narrow the renewed token When the upstream omits scope on a bridge refresh, seal the scope we requested (which RFC 6749 section 5.1 defines as the granted scope when omitted) into the renewed access and refresh envelopes, so the scope survives the whole refresh chain. The regression test refreshes against an upstream that omits scope, asserts the new refresh envelope still carries it, and refreshes again off that envelope to prove the chain does not lose it, mutation-checked * refactor(mcp): classify upstream OAuth faults once and derive status, code, and prose from the value Replaces the accreted relay helpers with a faults package (types, classify, render_oauth): every upstream token/DCR rejection is classified into exactly one fault value and the response status, wire error code, and prose are all derived from that value, so a caller-fault code can never ship on a server-fault status (the bugbot finding on invalid_grant over a 500). Classification takes the credential source into account: invalid_client and friends against the server's stored credentials are the operator's fault and render as 502 server_error with gateway-authored prose while the IdP's prose stays in server logs; the same codes against caller-supplied credentials relay on the status the code implies. Classifiers are total, so an unreadable rejection body (lying content-encoding, unconsumed stream) yields the same 502 fault instead of resurrecting the opaque 500 (the second bugbot finding); DCR rejections normalize to 400 per RFC 7591 regardless of the upstream's status * style(mcp): unquote annotations and use PEP 604 unions in the faults package * fix(mcp): detect upstream invalid_grant by the RFC 6749 error field, not a body substring The bridge refresh path decided whether an upstream token-endpoint rejection was invalid_grant by substring-matching the raw response body, so a rejection whose actual error is something else but whose error_description merely contains the string invalid_grant would false-match, map to invalid_grant, and trigger a needless authorization_code re-run Parse the RFC 6749 section 5.2 error object and compare the error field. A non-JSON body, or an error that is not invalid_grant, now propagates as the upstream error rather than being reinterpreted. The regression test drives an invalid_client rejection whose description contains the string invalid_grant and asserts it is not mapped, mutation-checked against the substring match * fix(mcp): keep upstream self-blame codes and gateway capability gaps off the caller Extends the fault matrix per review: server_error and temporarily_unavailable are codes by which the upstream blames itself, so they classify as a new UpstreamReportedFault arm rendering 502/503 with a matching wire code instead of a 400 that blames the caller; invalid_target is a gateway capability gap (RFC 8707 resource indicators, LIT-4339) and is gateway-blamed regardless of whose credentials were presented; the DCR classifier shares the same blame assignment. The gateway-fault arm is renamed GatewayRejected since it now covers capability gaps as well as stored-credential rejections * chore: add CODEOWNERS for ui and proxy UI build artifacts (#33131) * feat(ui): rebuild the Teams table on the shared DataTable (#33128) * feat(ui): rebuild the Teams table on the shared DataTable The Your Teams tab moves off the Ant Design table onto the shared DataTable that the Virtual Keys page uses, following the new dashboard design. It gains server-side sort, pagination and filtering, a toolbar with a filter drawer and a columns menu, and a per-row actions menu Sorting is wired only to the columns /v2/team/list can actually order by (team_alias, created_at); Spend / Budget and Updated stay unsorted because the endpoint silently ignores those fields. The design's "Created by" column is dropped since the team object has no such field, and the drawer's "Has keys" filter is dropped for the same reason. The Resources cell shows members, models and keys as colored pills, and the actions menu keeps the existing Edit, Copy team ID and Delete behaviors, with Edit and Delete gated to Admin The teams grid gets its own unit tests in TeamsPage/TeamsTable.test.tsx. Teams.tsx keeps the create-team modal, delete modal, detail view and tabs, now refreshing the list through React Query invalidation instead of a manual refetch * fix(ui): match Teams loading skeletons to the rendered row height The default twoLine and chips skeleton shapes rendered the Team and Resources cells shorter than the loaded row (a real row measures 55px, the old skeleton ~49px), so the loading state looked visibly squat. Give the Team column a custom renderSkeleton that mirrors the two-line IdentityCell (measured 54px) and the Resources column one that mirrors the pills, and mark the hidden Rate Limits column as two-line so it matches when shown * fix(ui): keep team admins' Members tab by deriving is_team_admin from the selected team The redesign computed is_team_admin from useTeam(selectedTeamId), but that hook returns teamInfoCall's nested { team_info: { members_with_roles } } shape, so the top-level members_with_roles read was always undefined and is_team_admin was always false. For a non-proxy-admin team admin that hid the Members, Member Permissions and Settings tabs in the team detail view, which broke the team-admin add/remove member e2e tests. Pass the Team object up from the table instead (/v2/team/list returns it with a top-level members_with_roles), matching the pre-redesign behavior; proxy admins were unaffected because is_proxy_admin already granted access Also point the Delete-a-team e2e at the new kebab: open the row actions menu, then click Delete team, rather than clicking the old inline delete icon * fix(keys): persist key_type so the UI shows correct key scope instead of "All Proxy Models" (#33115) * fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models' key_type is not persisted on a key (the proxy maps it to allowed_routes and drops it), so the keys tables only inspected the models list and rendered 'All Proxy Models' for any key with an empty models array, including SCIM, Management and Read-only keys that cannot call a single model. Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a scope tooltip for those recognized scopes; unrestricted, AI-API and custom keys keep the existing model-list rendering. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): move key_scope helper to components root Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(keys): persist key_type on virtual keys so the UI reads scope directly Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy, and proxy-extras schemas plus an additive migration) and stop dropping the value in handle_key_type, so management/read_only/llm_api/default keys store their type alongside the derived allowed_routes. Surface it on the key read and create response models. The dashboard now prefers the persisted key_type for the no-inference buckets and keeps the allowed_routes derivation as the fallback for keys created before the column existed (key_type null), so no backfill is required. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(keys): add key_type column to LiteLLM_DeletedVerificationToken The deleted-token archive model inherits key_type from the verification token, so regenerate/delete flows write key_type into LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration) so the archive insert does not fail with FieldNotFoundError. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(router): opt-in session affinity for complexity router (#33126) * feat(router): opt-in session affinity for complexity router Complexity router reclassified every turn, which could flip the routed model group mid-session and break provider-side prompt caching. Add a session_affinity config flag: when a session_id is resolvable, pin the model chosen on the first turn and reuse it for the rest of the session, skipping reclassification. Pinned turns still stamp the adaptive bandit's chosen-model metadata so reward feedback keeps working when adaptive=True. * fix(router): refresh session-affinity TTL on hit, scope pin by API key Two issues from review: the TTL was only set on the first classification, so an active session outliving session_affinity_ttl_seconds silently lost its pin instead of refreshing as documented. And the cache key was scoped only by session_id, which is client-supplied and unauthenticated, so two different callers reusing the same session_id could poison each other's routing pin. Refresh the TTL on every cache hit, and namespace the cache key by the proxy-derived API key hash when available. * feat(prometheus): expose video duration and image count consumption metrics (#33138) * test(e2e): otel trace completeness on /chat/completions (#33132) * test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage: a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export through a preset-owned provider - the code path where trace splits happen), a typed Jaeger query read-back client, and the first test: one successful non-streaming /chat/completions call exports ONE complete trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). * test(e2e): harden the otel trace read-back per review Jaeger reads now query server-side by the litellm.call_id span tag instead of paging recent traces and filtering client-side; the compose stack's background jobs alone can push a request trace past the page. A failed query hard-fails instead of reading as an empty result, the settle predicate now also waits for the prefix-matched db span the assertion demands, parent-chain walking follows CHILD_OF references only, the zero-trace and split-trace failures get distinct messages, jaeger gets a healthcheck so the depends_on condition is accurate, and the chat docstring names the route the code actually asserts * test(e2e): author the chat trace test docstring * Update logging section in CLAUDE.md Removed mention of OTEL trace-tree completeness from logging integration section. * fix(sso): paginate through all pages when fetching service principal group assignments (#33149) get_group_ids_from_service_principal only read the first page of the Graph API appRoleAssignedTo response, so tenants with more than 100 groups assigned to the enterprise application silently lost group memberships during SSO login. Loop over @odata.nextLink with the same MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already uses, and warn when the cap is hit. Ported from #32792 by @saisurya237 so CI can run. Fixes #32790 Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com> * test(e2e): otel trace completeness on /v1/messages (#33133) * test(e2e): OTEL trace completeness on /v1/messages Extends the LIT-3787 trace-completeness suite to the Anthropic-native route: one successful non-streaming /v1/messages call must land at the destination as ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). Adds the raw /v1/messages sender to the logging suite client. * test(e2e): reuse the shared AnthropicMessagesBody per review Drops the duplicate /v1/messages request model in favor of the one models.py already provides (budget_client uses the same one), passes max_tokens at the call site to match the sibling chat test, notes in the docstring why the gen-AI span is named chat on this surface, and adopts the hardened read-back signature * test(e2e): author the messages trace test docstring * test(e2e): declare the messages surface on the covers marker * test(e2e): otel trace completeness on /v1/responses (#33134) * test(e2e): OTEL trace completeness on /v1/responses Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API route: one successful non-streaming /v1/responses call must land at the destination as ONE connected trace. Adds the raw /v1/responses sender, a CHEAP_OPENAI_MODEL config constant, and registers responses in the otel registry cell's exercised_on. * test(e2e): author the responses trace test docstring * test(e2e): declare the responses and chat surfaces on the covers markers * feat(ui): add adaptive routing settings to Auto-Router v2 (#33146) * refactor(mcp): extract the dcr_bridge token flow into bridge_token_flow.py discoverable_endpoints.py had grown to 2695 lines mixing FastAPI route handlers with the dcr_bridge token-flow logic, against the no-monster-files convention. This moves the bridge token flow (the litellm-key/user resolution, the SCIM revalidation gate, and the mint/refresh envelope logic with their types and error mappers) into a dedicated bridge_token_flow.py, leaving the route handlers and the shared exchange_token_with_server orchestrator in discoverable_endpoints.py importing from it Pure relocation, zero behavior change. The moved code is byte-verbatim except one type annotation quoted as a forward reference (_BridgeAuthorizationCode is used only for typing and imported under TYPE_CHECKING to avoid a cycle), and the new module imports nothing from discoverable_endpoints at runtime. 275 tests pass unchanged; the test patch targets for moved internals were repointed to the new module and verified to still apply * bump: litellm-enterprise 0.1.49 -> 0.1.50, litellm-proxy-extras 0.4.76 -> 0.4.77, litellm 1.93.0 -> 1.94.0 (#33229) * chore(deps): pin httplib2 and setuptools transitive floors (#33233) Raise the constraint floors for two transitive dependencies so resolution moves them to their latest maintenance releases: httplib2 0.31.2 -> 0.32.0 and setuptools 82.0.1 -> 83.0.0. Both are pulled in only by optional integrations (Google API client, grpc tooling, lunary observability, the nvidia-riva extra), all lower-bound only, so the floors stay inside every requirer's allowed range and a default install is unaffected * feat(ui): left-anchor the Create Key and Create Team CTAs (#33248) Move the Create New Key and Create Team buttons out of the page header's right-side action slot. On Teams the button now sits in the tab bar's left slot, separated from the three tabs by a vertical rule, so the CTA and tabs read as one left-anchored cluster. On Keys, which has no tabs, the button anchors left on its own row beneath the title. * fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models (#33244) * fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models * test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping * fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models Narrow the fix to the temperature reconciliation; the reasoning_effort budget cap is reverted because the live translation grid relies on budget_tokens >= max_tokens to reject unsupported effort tiers (xhigh/max) on budget-mode models, so capping turned those 400s into 200s. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook (#33136) * fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook * fix(guardrails): keep request-body dispatch predicate unchanged * fix(guardrails): fail closed when proxy extras are missing at deployment hook * fix(proxy)!: enforce user budget on team keys (read-time + reservation) with UI opt-out (#32005) * fix: enforce user budget on team keys User budget was skipped when the key belonged to a team, letting users exceed their personal budget by going through a team key. Remove the team_object guard in _user_max_budget_check so user budgets are always enforced. Add skip_user_budget_on_team_key general_settings flag to opt back into the old behavior. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: update test to expect user budget enforcement on team keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert budget_exceeded ProxyException in personal budget test Tighten the broad pytest.raises(Exception) so the test only passes when the auth flow rejects with a budget_exceeded ProxyException, and switch the new ConfigGeneralSettings field to Optional[bool] to match the surrounding annotation style * fix: revert to bool | None to stay under UP045 strict budget --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> * fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265) The rate-limited batch spend test snapshotted unattributed rows via the unpaginated /spend/logs whole-table read, which grows with the environment (58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an explicit date window instead, and SpendLogsParams now rejects a filterless read so the whole-table call cannot come back * refactor: make the code easier to read * feat(pricing): add gemini-omni-flash-preview with video output token pricing * fix(gemini): map video response modality instead of MODALITY_UNSPECIFIED * fix(anthropic): use native output capability (#33235) * fix(anthropic): route native structured output Use model capability metadata so new native structured-output models do not require transformation allowlist changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): pass provider to capability Co-authored-by: Cursor <cursoragent@cursor.com> * test(anthropic): cover dotted model IDs Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): handle remote capability lag Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): retry setup-uv installs to survive transient manifest fetch failures * docs(e2e): add cache_hit to the naming grammar assertion vocabulary * refactor(e2e): share anthropic cache-control shapes in endpoints_client * fix(proxy): never log raw virtual keys in key insertion debug output (#33268) * fix(proxy): never log raw virtual keys in key insertion debug output * fix(proxy): tolerate None token in insert_data debug log redaction * fix(auth): scope the JWT enterprise gate to actual JWTs (#33296) With enable_jwt_auth enabled but no enterprise license (premium_user False), the JWT premium check fired on every request before the token was inspected, so the master key, sk- virtual keys, and the encrypted CLI/UI SSO session token that `lite login` issues all 401'd with "JWT Auth is an enterprise only feature" and were never decoded. That broke `lite login`, `lite claude`, and the proxy master key on any deployment that turned JWT auth on without a license. Move the premium check inside the is_jwt branch so it gates only real JWTs. Non-JWT credentials fall through to their own auth paths regardless of license; actual JWTs still require premium, so the enterprise gate is unchanged for the feature it protects. * test(e2e): scope virtual keys to the deployment under test * fix(s3): sanitize slashes in response-id-derived object key file name (#33271) * refactor(ui): migrate guardrails table onto shared DataTable (#33303) * feat(ui): migrate guardrails table onto shared DataTable Move the guardrails list onto the shared DataTable + cell library as the proof-of-concept for the simple-tables design migration, following the Teams reference pattern. Split the table into a thin container (guardrail_table.tsx) and column defs (guardrailTableColumns.tsx): client-side sort defaulting to created_at desc, a search + refresh toolbar, IdCell / DateCell / StatusBadge cells, real provider logos, a rich empty state, and skeleton loading rows. Row actions move into a per-row overflow menu; deletion stays disabled for config-file guardrails, now surfaced as a disabled menu item instead of a greyed trash icon. Detail view and the delete modal remain owned by GuardrailsPanel. Restyle the "Add New Guardrail" control to the shared Button + dropdown menu. Update the regression tests for the menu-based actions and drop the now-stale eslint suppression entry that the rewrite eliminated. * fix(ui): match guardrails table to the design Address design-review feedback on the guardrails migration: - Drop the search + refresh toolbar. The original table had neither and the SimpleTable design has no toolbar; the container now just renders the sorted table and its empty state. - Give the Guardrail ID cell the design's hover affordance by rendering it with the shared IdentityCell (monospace, chevron on hover) instead of the blue IdCell pill. - Stop pinning the actions column. Pinning added a sticky divider that the design and the Teams table don't have; it is now a plain right-aligned menu column, matching Teams. * fix(ui): match loading skeleton row height to loaded rows The compact skeleton row did not carry the h-8 height that real compact rows get, so loading rows rendered shorter than loaded ones and the table height jumped when data arrived. Mirror the same size-based height on the skeleton row in the shared DataTable so every compact table loads at a stable height * test(ui): drop stale onGuardrailUpdated from guardrails table baseProps The prop was removed from GuardrailTableProps when the toolbar went away; the test baseProps still listed it. Harmless at the call site since it is spread rather than an object literal, but dead and worth removing * fix(ui): remove dead edit_guardrail_form after guardrails migration The guardrails table migration dropped the last import of EditGuardrailForm, which knip flags as an unused file. The form was already unreachable before the migration: the table wired a delete button only, and nothing ever called handleEditClick to open the modal, so the import was the sole thing keeping the file referenced. Delete it and prune its now-stale eslint suppression entry. Guardrail editing is unchanged and lives in the detail view (GuardrailInfoView) * feat(guardrails): streaming text transformation in generic_guardrail_api (#33110) * feat(guardrails): support streaming text transformation in generic_guardrail_api * chore(guardrails): address PR review feedback * fix(guardrails): fail closed on tool-call and prefix-rewrite leaks in streaming transform * fix(guardrails): address Bugbot review on streaming transform correctness * fix(guardrails): coerce holdback in handler for in-process guardrails * fix(guardrails): harden streaming transform (holdback coercion, tool-call passthrough, n>1 finish_reason) * test(guardrails): targeted _mode_matches coverage for all guardrail_mode shapes * fix(guardrails): inspect streamed tool calls and harden incremental_diff edge cases * test: move ComplianceChecker mode tests to the compliance PR * fix(guardrails): strip content from tool-call passthrough so streamed text can't bypass the transform * fix(guardrails): four correctness fixes for incremental_diff streaming path Four bug fixes on top of the OSS PR's incremental_diff streaming text transformation, all inside the incremental_diff code paths only. No existing block_only, non-streaming, or pre_call behavior is touched. Fix #1 — Mixed content+tool_call finish_reason ordering _tool_call_passthrough_chunk now takes an optional finish_reason_per_choice map. For a choice carrying both delta.content and delta.tool_calls, finish_reason is stripped from the passthrough and recorded on the map so the final synthetic text chunk delivers it. Without this, SSE-compliant clients stopping at finish_reason drop the guardrailed text — defeating the redaction the whole feature exists for. (Greptile P1 twice, Veria.) Fix #2 — Choice index sort in _process_streaming_transform indices/texts_to_check were derived from dict insertion order. For n>1 streams where choice 1 emits before choice 0, guardrail-returned texts aligned to the input order mapped back to the wrong choice indices on write-back — wrong text goes to wrong choice. Sort raw_by_index.keys() up front so realignment is deterministic. (Bugbot Medium.) Fix #3 — Cross-chunk pre-tool-call text flush With default streaming_sampling_rate=5, text chunks followed by a pure tool-call chunk carrying finish_reason='tool_calls' would emit the passthrough with finish_reason before any transformed text delta had fired. Same failure mode as fix #1 but cross-chunk. Now we flush any accumulated text via _round(is_final=False) BEFORE yielding the tool-call passthrough. (Greptile P1.) Fix #4 — Terminator chunk for deferred finish_reason on empty mutated_text _build_transform_chunk returned None early when mutated_text_per_choice was empty. If a mixed content+tool_call chunk had deferred its finish_reason (via fix #1) and the guardrail then suppressed the text (empty return), the deferred finish_reason was never delivered. Now on is_final=True with empty mutated_text_per_choice, we emit a terminator carrying finish_reason per choice from finish_reason_per_choice. (Bugbot High.) Also normalized Optional[X] → X | None across the OSS PR's added surface via ruff UP045 autofix to keep the strict-rule gate within budget. Pure mechanical typing style change, no semantic effect. Regression tests for all four fixes: - test_mixed_chunk_finish_reason_arrives_after_transformed_text (#1) - test_text_flush_precedes_tool_call_passthrough (#3) - test_final_finish_reason_flushed_when_guardrail_suppresses_text (#4) - test_transform_sends_texts_sorted_by_choice_index (#2) All fixes reachable only when streaming_transform_mode == 'incremental_diff' is configured (via _run_incremental_transform_stream) or when a StreamTransformSink is present (via _process_streaming_transform). Verified scope-clean: no changes to block_only, non-streaming, pre_call, moderation, or sibling guardrails. --------- Co-authored-by: Marton Schneider <marton@schneider.co.nl> * test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) * test(claude_code): move the Claude Code compatibility matrix under tests/e2e Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: restore the upload-coverage job dropped by mistake with the compat gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments * test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> Co-authored-by: tin-berri <tin@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Shivam Rawat <shivam@berri.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Thibault Serot <thibault@linktr.ee> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Or Gershoni <org@lasso.security> Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Marton Schneider <marton@schneider.co.nl> Co-authored-by: mateo <mateo@berri.ai>	2026-07-14 19:30:42 -07:00
Mateo Wang	5f864c83ce	chore(lint): zero out crash-class pyright rules and ban new type: ignore comments (#32152) ... * fix: zero out crash-class basedpyright rules across litellm/ * feat(lint): add LIT009 banning inert type: ignore comments * docs: require bracketed rule and reason on every suppression * chore(lint): ratchet budgets down and zero crash-class pyright limits * fix: narrow auto router routelayer through a local before calling * test: add regression tests for crash-class fixes * fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex	2026-07-04 16:56:12 -07:00
Mateo Wang	cf6fdac304	perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate (#32000) ... * perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate make pre-commit paid for a full second basedpyright pass over a merge-base worktree on every run even when no rule was over its ceiling, re-generated an unchanged Prisma client, and ran seven independent checks sequentially. The basedpyright and ruff strict gates now skip the base pass when head is within every limit (the same early-out type_discipline_gate already had), the basedpyright base counts are cached under the git common dir keyed by merge-base commit, pyrightconfig.json, and uv.lock, prisma generate only runs when the schema or prisma version changed, and make lint fans its checks out through a parallel sub-make after a single setup phase * fix(lint): keep the base-cache scratch file out of the prune glob The tmp+rename scratch in store_counts was named basedpyright-base-<hash>.json.tmp, which the stale-entry prune glob (basedpyright-base-*) also matches, so a concurrent lint run from another worktree sharing the same git common dir could unlink it between write_text and replace and crash the gate with FileNotFoundError. The scratch is now dot-prefixed so the glob can never see it, pid-suffixed so concurrent writers of the same entry never share a scratch, and the prune glob is restricted to committed *.json entries	2026-07-02 19:24:00 -07:00
yuneng-jiang	ae6dbb4a9b	fix(scripts): resolve worktree root before relative_to in type_check_gate (#31906) ... On macOS, tempfile.mkdtemp returns a path under /var/folders, a symlink to /private/var. The base pass in type_check_gate.py resolved each diagnostic path (yielding /private/var/...) but not the worktree root, so relative_to raised ValueError for every diagnostic, base counts came back empty, and the vacuous-run guard failed every local make lint-basedpyright run. type_discipline_gate.py already resolves root the same way; ruff_strict_gate.py counts rule codes without touching worktree paths, so it is unaffected. CI runs Linux where the temp dir is not a symlink, which is why this only bit local macOS runs	2026-07-01 14:09:07 -07:00
Mateo Wang	e141596204	refactor(lint): collapse type/lint budgets to a single per-rule limit (#31883) ... * chore(lint): raise basedpyright per-rule slack to 50% of baseline The per-rule ceilings in basedpyright-code-budget.json sat at roughly 10% slack over baseline, which several in-flight PRs are already bumping into. Raise the slack on every rule to at least 50% of its baseline so there is ample headroom for a long while, while never lowering any rule that already had more generous slack (e.g. reportReturnType stays at 100). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(lint): collapse type/lint budgets to a single per-rule limit The three non-frontend budget files (ruff-strict, type-discipline, basedpyright-code) tracked a per-rule baseline and slack whose sum was the ceiling. Nothing consumed the split beyond that sum, so this replaces both keys with a single limit equal to the old baseline + slack; the original baselines live in git history if anyone needs them. The gate scripts and the ratchet guard now read limit directly. lint-budget-update no longer re-captures raw counts; it ratchets each rule's limit down by the number of violations this branch cleared since its branch point (the merge-base), so the granted headroom shrinks by exactly what was fixed and a limit never rises. The ratchet guard reads either schema so it still compares correctly across the migration boundary. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(lint): surface staged-vs-working parity for pre-commit and budget-update make pre-commit selects which checks to run from the staged index but runs the linters over the working tree, so unstaged edits to tracked files and untracked files skew a green/red away from what a commit of only the staged changes would produce. There is no safe in-place way to lint the index, so the script now warns when unstaged or untracked changes are present, and CLAUDE.md documents that you must stage everything first for both make pre-commit and make lint-budget-update to predict CI correctly. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * docs(lint): list type-discipline budget in lint-budget-update instruction --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-07-01 18:12:35 +03:00
Mateo Wang	0965a4d1f4	chore: shift CI lint left with an opt-in make pre-commit and CLAUDE.md rule (#31544) ... * chore: shift CI lint left with a pre-commit hook and CLAUDE.md rule Add an opt-in pre-commit hook (.githooks/pre-commit, active after make install-hooks) that runs the CI-equivalent checks against staged files: make lint for Python, prettier plus eslint for the dashboard, and a gen:api drift check for the proxy OpenAPI types. Document the same expectation in CLAUDE.md so reds surface locally instead of in CI. * fix: make `make lint` isomorphic to the CI lint job `make lint` diverged from test-linting.yml in ways that produced both false reds and false greens: its format-check ran over the whole repo (CI scopes it to changed files vs the base), its ruff-strict budget ran in absolute mode (CI runs it as a delta vs base), and it omitted the type-discipline gate entirely. Recompose `lint` to replay CI's exact sequence: diff-scoped ruff format check, whole-tree ruff check, the strict / type-discipline / basedpyright budgets as a delta resolved the same way CI resolves it (merge-base with origin/litellm_internal_staging), then circular-import and import-safety. Factor the repeated base fetch into one shared prerequisite so the chain hits the network once. Align the pre-commit hook's eslint invocation with the CI frontend-lint job (`--pass-on-unpruned-suppressions`) and fix the CLAUDE.md guidance to point at the diff-scoped frontend commands instead of the whole-folder npm scripts, which are broader than CI. * fix(githooks): make pre-commit 1:1 with CI frontend-lint, lint, and type-gen The shift-left pre-commit hook diverged from the CI jobs it claims to mirror, so a clean commit did not actually mean a green CI lint. The dashboard block only ran prettier and eslint over js/jsx/ts/tsx/mjs/cjs, but CI's frontend-lint runs prettier over a wider set (also json, css, scss, md, mdx, yml, yaml, html) and additionally gates the whole-folder eslint lint budgets via scripts/check-lint-budgets.mjs. The hook now mirrors that split and runs the budget check, so a dashboard commit that passes locally passes the job. The API-types block ran npm run gen:api without LITELLM_PYTHON, so it shelled out to the system python3 which has no litellm installed and always failed with a false 'could not regenerate API types' red. It now passes LITELLM_PYTHON="uv run --no-sync python" the way check-ui-api-types.yml does. make lint format-checks the files in origin/base...HEAD, which at pre-commit time predates the staged change, so a brand-new commit's formatting went unchecked. The Python block now also runs ruff format --check over the staged litellm files directly to cover that case, and its trigger is scoped to staged litellm/ files (the only tree CI's lint job inspects) so a tests-only or scripts-only commit skips the slow make lint instead of wasting time on a run that could not catch anything. CLAUDE.md's shift-left rule was cut off mid-sentence and understated the frontend checks; it now describes all three gates accurately and points agents at make install-hooks to run them automatically before each commit. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(githooks): scope the API-types check to all of check-ui-api-types.yml's triggers spec_files was filtered from the staged Python files, so the gen:api drift check only fired for .py changes under litellm/proxy or litellm/types. CI's check-ui-api-types.yml triggers on any file under those directories (Prisma schema, configs) plus the generator script and the dashboard package files, so a non-Python proxy/types change could pass the hook and still fail CI. Match the workflow's full trigger set instead. * fix(pre-commit): run prisma generate before gen:api to mirror CI * refactor(githooks): run shift-left lint via on-demand make pre-commit, not an auto-firing hook The pre-commit hook ran make lint plus the dashboard eslint budgets, which are minutes of work (basedpyright over litellm/, a whole-folder eslint . pass at ~40s). Wiring that into core.hooksPath via make install-hooks meant every human commit, not just an agent's, paid that cost, which is real friction for interactive committers. Move the staged-file checks out of .githooks/ into scripts/pre_commit_lint.sh and expose them as make pre-commit, and keep .githooks/ to only the fast Conventional Commits / Branches hooks so make install-hooks no longer makes commits slow. Agents run make pre-commit right before each commit (CLAUDE.md instructs this), so the slow gates fire only for the commits an agent is making and never auto-fire for a human typing git commit. The script stays hook-compatible for anyone who still wants it to fire automatically via a symlink. Preferred this over sniffing an agent env var to auto-fire only for agents: that is fragile (misses agents when the var is unset, fires on humans when it leaks into their shell, and silently no-ops a hook a human deliberately installed), whereas an on-demand command achieves the same humans-never, agents-per-commit outcome deterministically. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(pre-commit): run make lint last so it can't prune the proxy deps gen:api needs make lint's install-dev prerequisite runs uv sync --frozen, which prunes the proxy extras (prisma, websockets, ...) from the venv. With the Python block running first, the subsequent API-types block then failed: gen:api imports litellm.proxy.proxy_server, which needs those deps, so every litellm/proxy change (the main trigger for the API-types check) hit a false 'could not regenerate API types' red. Run the dashboard and API-types blocks before the Python block so gen:api sees an intact env; CI is unaffected because there the lint and check-ui-api-types jobs run in separate environments. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix: make CLAUDE.md more concise * fix(makefile): give make lint the CI lint env and stop it pruning the venv make lint diverged from test-linting.yml's lint job in two ways: it never generated the Prisma client (so basedpyright resolved the DB wrappers as Unknown, drifting from CI's counts), and its bare uv sync --frozen pruned the proxy extras (prisma, websockets, ...) out of the venv on every run, which broke the gen:api step that imports litellm.proxy.proxy_server and left a dev unable to run the proxy until re-syncing. Add a lint-install target that mirrors the job's environment (the proxy-dev group plus prisma generate) and runs before the checks, and make both it and install-dev use uv sync --inexact so they top up the venv instead of tearing packages out. CI is unaffected since it installs its own env per job. Because make lint no longer prunes, the pre-commit reorder that ran it last (to dodge the prune) is no longer needed, so restore the original block order. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(makefile): drop lint-install so make lint matches CI's slimmer env test-linting.yml's lint job installs deps with a bare uv sync --frozen (default dev group only, no proxy-dev, no prisma generate), but the lint-install target chained into make lint pulled in --group proxy-dev and ran prisma generate. Because the basedpyright budget step compares head and base counts against fixed thresholds, the extra symbols and Prisma client locally resolved can shift error counts away from CI's, producing false greens or false reds on the type-check gate. Remove the lint-install target and its slot in lint. The remaining sub-targets already chain install-dev, which now uses uv sync --inexact --frozen, so the venv still isn't pruned but the installed set stays aligned with what CI sees. * ci(linting): install proxy-dev and generate prisma in lint job, matching make lint make lint now installs the proxy-dev group and generates the Prisma client so basedpyright resolves the DB wrappers; the lint job here still installed only the base env, so a local pre-commit could pass while the required CI lint failed (or vice versa). Bring this job in line, which is the same environment litellm_internal_staging's lint job already uses. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(makefile): keep make lint on the proxy-dev + prisma env to match CI A concurrent change dropped lint-install to match what looked like CI's slim env, but test-linting.yml's lint job (and the merge ref this PR's CI actually runs) installs --group proxy-dev and generates the Prisma client. With make lint slim and CI fat, basedpyright resolves fewer symbols locally than CI, so a prisma-typed error can stay Unknown locally (green) while CI catches it (red). Restore lint-install so make lint installs the same env CI does; the previous commit also brought this PR's test-linting.yml in line with that env, so the two now match. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-06-30 09:37:12 -07:00
Mateo Wang	92d0788da2	chore(lint): widen ANN slack to 10% of baseline and drop PLR0913 from the strict gate (#31335) ... * chore(lint): widen ruff budget slack to 10% of baseline for high-volume ANN rules and PLR0913 Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(lint): drop PLR0913 from strict gate to roll out rules gradually Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(lint): ratchet-guard rising baselines even when slack is cut to mask them Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-06-25 14:43:45 -07:00
Mateo Wang	f26dbb60be	ci: make the basedpyright budget gate delta-vs-base (#31106) ... * ci: re-run absolute basedpyright budget gate on push to long-lived branches The basedpyright budget gate counts codebase-wide errors per rule against a committed ceiling, but it only ran on pull_request against each PR's own head. Two PRs that each pass in isolation can together push a per-rule count over its ceiling once both merge, and nothing re-evaluated the budget on the merge commit, so the breach only surfaced on the next PR that happened to be checked out after the count crossed the line. Add a push trigger on the long-lived branches and a post-merge-budget job that re-runs the absolute gate on the merged tree, catching the accumulation on the merge commit itself. The existing pull_request jobs are guarded so their delta-vs-base gates don't misfire on push, where no PR base SHA exists. * ci: shallow-fetch the post-merge-budget checkout The post-merge-budget job only runs basedpyright over the working tree and the committed budget file; it never inspects git history, unlike the lint job whose delta-vs-base gates need full history. Drop its checkout from fetch-depth: 0 to fetch-depth: 1 to avoid cloning the whole repo history. * ci: scope post-merge-budget push trigger to long-lived branches On a push event the branches filter matches the branch being pushed to, not the PR target. The litellm_** glob, correct for the pull_request filter where it matches the target branch, therefore fired the post-merge-budget basedpyright job on every short-lived feature branch carrying the litellm_ prefix (litellm_dev_*, litellm_add_*, and so on), duplicating the PR lint job and burning ~10 minutes of CI per push. Restrict the push trigger to the long-lived branches PRs actually merge into (main, litellm_internal_staging, litellm_oss_branch), where budget accumulation happens. The pull_request filter keeps litellm_** so PRs targeting any long-lived branch are still linted. * ci: make the basedpyright budget gate delta-vs-base The basedpyright gate counted absolute codebase-wide errors per rule against a committed ceiling and ran only on each PR's own head. Two PRs that each pass in isolation could together push a rule past its ceiling once both merged, and because the gate had no comparison against the base, the next unrelated PR branched off the now-over-ceiling tree inherited a red it did nothing to cause. Give it the same shape as the ruff strict gate: a rule fails only when its total is both over the ceiling and higher than the count on the merge-base it merges into. Drift already in the base is never blamed on a bystander, while any change that actually grows a rule past the cap still fails. Head counts come from the existing stdin pipe; the base count is a second basedpyright pass over a detached worktree at the merge-base, reusing the head environment so import resolution matches and no second uv sync is needed. This obsoletes the push-triggered post-merge-budget job (and its event guards), which only detected accumulation after the fact; the delta check blocks it on the PR instead. Slack for reportReturnType and reportUnnecessaryComparison is raised to give real headroom under the cap. * refactor(ci): give the base ref its own name in type_check_gate cmd_check cmd_check took a parameter named base that held a git ref string, then rebound the same name to the dict of base-tree error counts returned by base_counts. Rename the parameter to base_ref so the ref and the counts each keep a single name and type, matching the no-reassignment style used elsewhere; behavior is unchanged. --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-06-23 11:10:05 -07:00
Yassin Kortam	84266bf924	feat(auth): resolve caller identity once into a Principal at the auth seam (#30887) ... Introduce a single, typed caller identity that is resolved once at the auth boundary and read by reference downstream, instead of being re-derived from a 50-field key object or rebuilt from request metadata. What this adds (litellm/proxy/auth/resolvers/), organized by responsibility: - Principal: a small, frozen, identity-only value type (user / organization / teams / project / end-user / roles / scopes / network), with its sub-models and the role mapping. No budget or policy state; those stay on the key object. - DbIdentityStore: the auth flow's resolver, owning both halves of resolving a caller. resolve_key does the one combined_view lookup (cache, then DB via the shared lower-level helpers, then write-back) and returns the key object, which still flows for budget / rate-limit / policy unchanged. principal_from_key projects the identity slice of that key object into a Principal, issuing no lookup. user_api_key_auth resolves every key through the store rather than calling get_key_object directly; auth_checks.get_key_object stays as the legacy entrypoint for its other callers until they migrate. - network: the X-Forwarded-For / trusted-proxy CIDR primitives live here in one place. trusted_proxy_utils now imports them rather than keeping a second copy. At the seam, user_api_key_auth projects one per-request Principal off the resolved key object and stamps the request network context onto it once (X-Forwarded-For is trusted only when trusted_proxy_ranges is configured). It is attached to request.state.principal for the downstream consumers later phases add. The projection is additive and defensive: a failure never rejects an already-authenticated request, and a missing principal must be treated as deny by any future reader. The Principal is always identifiable (credential_ref and a stable subject off the token), never anonymous. This is additive and changes no behavior today; it is the identity foundation the spend-attribution and authorization phases build on.	2026-06-20 18:49:41 -07:00
Mateo Wang	a7b0b0ba09	feat: add lint-gate target and truncation-proof summary to the strict ruff gate (#30877) ... * feat: add CI-parity mode and truncation-proof summary to strict ruff gate * refactor: tolerant worktree cleanup and concrete GateInputs types * fix: clean up temp dir when git worktree add fails * fix: align lint-gate with CI by dropping unused --ci-parity path The lint-gate Makefile target invoked ruff_strict_gate.py with --ci-parity, which counted violations on a throwaway merge of base into HEAD against base counts at the base tip. CI in test-linting.yml runs the same script without --ci-parity on a PR-head checkout, taking the gather_fast path that counts on the live tree against base counts at the merge-base. A local pass could therefore disagree with CI. Drop --ci-parity from the Makefile and remove the now-unused gather_ci_parity branch and flag so there is one code path that both local and CI exercise. The docstring claim that CI runs against the synthetic merge ref was also wrong; the workflow checks out github.event.pull_request.head.sha. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-20 11:46:01 -07:00
Mateo Wang	b8d79d1e0c	ci: drop mypy entirely, standardize type checking on basedpyright (#30648) ... * ci: drop redundant mypy type-check gate, standardize on basedpyright Type checking ran both mypy (via the pydantic.mypy plugin) and basedpyright. pydantic v2 emits dataclass_transform, so basedpyright understands models natively with no plugin, and its gated rules already cover what the mypy pass caught (no-untyped-def, no-any-return, valid-type, import-not-found all map to basedpyright equivalents). Running both meant two checkers, two budgets, and a plugin only mypy could load. This removes the mypy type-check gate: the lint-mypy/lint-mypy-budget-update Makefile targets, the CI MyPy step, mypy-code-budget.json, the budget-ratchet entry, and the vestigial [tool.mypy] pydantic plugin block (the gating pass used litellm/mypy.ini, which never loaded the plugin). type_check_gate.py is specialized to basedpyright since the mypy parsing path is now unused. mypy stays a dev dependency because the Any-discipline gate (scripts/check_any_discipline.py) imports it as a library to detect Any-typed values; it is no longer run as a type checker. * ci: remove the Any-discipline gate, rely on basedpyright's reportAny The Any-discipline gate (scripts/check_any_discipline.py) was the last consumer of mypy: it imported mypy as a library to detect values whose inferred type contains Any, gated per-file against any-discipline-budget.json. basedpyright already reports the same class of finding through reportAny/reportExplicitAny, which are gated tree-wide in basedpyright-code-budget.json, so the separate gate (and the mypy dependency behind it) is redundant. Removes the gate end to end: check_any_discipline.py and its test, the any-discipline CI job, the lint-any/lint-any-budget-update Makefile targets, any-discipline-budget.json, litellm/mypy.ini, the .mypy_cache_any references, and mypy from the dev dependencies. budget_ratchet_check.py drops the any-discipline entry and the now-unused zero-floor mechanism (rewritten as a comprehension). check_type_discipline.py drops the any-ok suppression token, since # any-ok suppressed only the deleted gate; the 134 now-orphaned # any-ok comments across 14 files are stripped (they never affected basedpyright, which uses # pyright: ignore). uv.lock is intentionally left untouched: uv still considers it consistent with the mypy-removed pyproject (uv lock --check and uv sync --frozen both pass), and a relock bumps 30+ unrelated packages because of the moving exclude-newer window. A future intentional relock will prune the now-unreferenced mypy entry. * build: relock to drop mypy from uv.lock CI's uv 0.10.9 honors the repo's exclude-newer window and correctly flags the lockfile as out of sync once mypy leaves pyproject; my earlier local uv 0.8.17 could not parse exclude-newer and silently passed --check. Relocking with the pinned CI version removes only mypy and its transitive librt, with no other version changes.	2026-06-17 09:42:00 -07:00
Mateo Wang	17b88719a2	ci(lint): grandfather any-discipline with a per-file ratchet budget (50% headroom) (#30582) ... * ci(lint): grandfather any-discipline with a per-file ratchet budget (50% headroom) The any-discipline gate previously failed on any Any-typed value touched on a changed line, which tripped on merely editing a legacy `X | Any` line. Switch it to a per-file budget: `any-discipline-budget.json` records each file's current Any count and a changed file fails only when its count exceeds `baseline + slack` (50% headroom, rounded up). New/unbudgeted files have baseline 0, so they stay airtight, while editing legacy files no longer forces cleaning pre-existing debt. Only changed files are re-type-checked (per-PR cost unchanged); the whole-tree scan to recapture the budget runs under `--update` (`make lint-any-budget-update`). The budget is a one-way ratchet guarded by `budget_ratchet_check.py`, matching the ruff/mypy/basedpyright budgets, and folds into `make lint-budget-update`. Also fixes a RecursionError in `contains_any` (recursive type aliases yield fresh objects per unfold, defeating the id() cycle guard) by walking iteratively with a depth cap, exposed by the whole-tree scan. * chore: make CLAUDE.md more concise * chore: rearrange Makefile * ci(lint): make any-budget --update git-failure-safe; clarify over-budget message all_litellm_py_files now returns None when git is unavailable (mirroring changed_line_map) instead of letting CalledProcessError/FileNotFoundError escape as a raw traceback, and update_budget reports a clean setup error (exit 2) for that case. The list-files dependency is injected so the path is unit-testable without monkeypatching. The over-budget diagnostic now reads "N value(s) total, over budget" so the count isn't misread as the excess over the ceiling. * ci(lint): exempt the file-keyed any-discipline budget from the ratchet's dropped-entry rule budget_ratchet_check treats a vanished budget entry as a loosening (an untracked rule whose ceiling is now unbounded). That holds for the rule-keyed budgets, but the any-discipline budget is keyed by file and its gate treats an absent file as ceiling 0 (the file must be Any-free). Cleaning a file to zero drops its entry on the next --update, so the generic rule flagged that as a regression: a false- positive red on exactly the cleanup the ratchet exists to encourage. Exempt the file-keyed budget from the dropped-entry rule while still catching a raised ceiling.	2026-06-16 19:23:20 -07:00
Sameer Kankute	1ccc1e5b23	chore: litellm oss staging160626 (#30527) ... * feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit 5272920fa0. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo (#30450) * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo Models that publish both a service_tier (e.g. priority) rate and an above-threshold tier (e.g. _above_200k_tokens) currently bill cached tokens at the standard above-threshold rate rather than the priority above-threshold rate. Affected entries in the live pricing JSON include gemini-3-pro-preview, gemini-3.1-pro-preview and their vertex_ai/ and gemini/ variants, plus azure/gpt-5.4 and azure_ai/gpt-5.4. For a 250K-token priority request with 200K cached tokens against gemini-3-pro-preview, the leak is about 44 percent of the prompt cost. Two stacked defects caused this. First, ModelInfoBase (and the ModelInfo pydantic class) and the get_model_info construction in litellm/utils.py omit the priority+above-threshold cost keys, so even if the calculator asked for them they would never reach it. Second, in _get_token_base_cost the cache_creation/cache_read tiered keys never get wrapped with _get_service_tier_cost_key, while the input/output tiered keys above and below do. The change here surfaces six new keys (input, output and cache_read at both 200k and 272k priority variants) and wraps the three cache tiered keys in _get_token_base_cost the same way input/output already are. _get_cost_per_unit's existing service_tier-to-base fallback covers models that ship the standard above-threshold rate without a priority variant. Adds one regression test in tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py that drives the actual generic_cost_per_token path for gemini-3-pro-preview at 200K cached + 50K text under priority and asserts the priority above_200k rates are picked. Verified the test fails on litellm_internal_staging without these changes and passes with them. * fix(cost): drop guard on cache tiered keys so service_tier fallback can reach standard above-threshold rate Addresses Greptile P1 on PR 30450. The previous commit wrapped cache_creation_tiered_key, cache_creation_1hr_tiered_key, and cache_read_tiered_key with _get_service_tier_cost_key (matching how the sibling input and output tiered keys are wrapped) but kept the surrounding 'if key in model_info' guards. For models that publish a standard above-threshold cache rate but no priority variant (gpt-5.4-pro, gpt-5.5-pro and their dated siblings, plus vertex_ai/claude-sonnet-4-5 for cache_creation), the guard short-circuits before _get_cost_per_unit's existing service_tier-to-base fallback can strip _priority and find the standard above-threshold key. The result on priority requests over the threshold was that those models silently dropped from the above-threshold rate back to the priority-base rate. Dropping the guard and calling _get_cost_per_unit unconditionally (mirroring how tiered_input_key and tiered_output_key are already handled) restores correct billing for that class of models while keeping the new priority+above-threshold behaviour for gemini-3-pro-preview and friends. Adds a second regression test that pins generic_cost_per_token for vertex_ai/claude-sonnet-4-5 priority + above_200k with cached and cache_creation tokens to the expected standard above-threshold rates, so the guard cannot be silently reintroduced for either the cache_read or cache_creation path. * fix(presidio): skip pre-call masking when guardrail is logging_only (#30461) The Presidio pre-call hook masked the live request unconditionally, ignoring the configured event hook. With mode: logging_only the masked request reached the model, so its response echoed anonymization tokens (e.g. <PERSON>) instead of the real output. Gate async_pre_call_hook on should_run_guardrail, matching every other guardrail; logging_only masking still happens via async_logging_hook. * fix(router): resolve list unhashable crash on model alias (#30464) * fix(router): resolve list unhashable crash on model alias Fixes the fallback parsing logic that mistakenly categorized standard array fallback definitions as override dictionaries when a deployment alias matches the literal string 'model'. Closes https://github.com/BerriAI/litellm/issues/30459 * fix(router): address greptile review for fallback parsing edge cases - Resolves ambiguity in standard vs override fallback dictionaries by iterating over all items and validating that no mapped litellm param resolves to a non-list type. - Adds regression tests in test_router_order_fallback.py to prevent unhashable type crash from silently re-entering the codebase. * chore(router): format code with black to pass CI * fix(hosted_vllm): remove thinking_blocks and convert list content to strings (#30475) * fix: hosted_vllm remove thinking_blocks and convert list content to strings vLLM endpoints reject assistant messages with thinking_blocks converted to content list blocks. This change removes thinking_blocks entirely and converts any list content back to strings. This fixes BadRequestError when using Claude Code with hosted_vllm models that pass thinking_blocks in messages. * fix(hosted_vllm): address Greptile review feedback - Join multiple text blocks with newline instead of empty string - Always set content to string (never None) to avoid vLLM validation errors * fix(hosted_vllm): update chat transformation to clean assistant messages * fix: re-raise exception instead of silently dropping MCP team permissions (#30477) * fix: re-raise exception instead of silently dropping MCP team permissions When MCPRequestHandler.get_allowed_mcp_servers raises, the broad except was swallowing the error and returning only allow_all_server_ids, silently discarding all team-level object_permission grants. Fixes #30476 * fix: log full traceback when MCP permission lookup fails Uses verbose_logger.exception() instead of warning() so operators can see the full traceback when team-level object_permission grants are dropped due to an internal error in get_allowed_mcp_servers. Fixes #30476 * fix: remove timezone date expansion in daily-activity aggregation (#29569) * fix: remove timezone date expansion in daily-activity aggregation Single-day spend queries from non-UTC timezones over-counted by ~2x because the previous implementation widened the SQL date range by a full UTC day on whichever side the offset pointed. Spend is bucketed in whole-UTC-day rows in LiteLLM_DailyUserSpend, so the expansion pulled an extra 24h of unrelated bucket data per boundary. Concretely on IST (UTC+5:30, offset -330): a single-day query for 2026-05-29 was rewritten to date >= 2026-05-28 AND date <= 2026-05-29 and returned spend across both UTC days. Sums of single-day queries across a 5-day window then exceeded the equivalent multi-day aggregate by ~50%, which is mathematically impossible. Treat the local date range as the UTC date range. The aggregation table has no hour-level granularity, so any conversion using only date arithmetic must round to whole UTC days; the previous fix turned that boundary slop into systematic over-counting. Pass-through trades a small one-time slop at each end of the range for correct, monotonic, additive results across single-day and multi-day queries. Repro from production: bedrock/global.anthropic.claude-opus-4-8 over 2026-05-29 to 2026-06-02, IST timezone: - 5-day aggregate: $701.39 / 1,831 reqs - Sum of 5 single-day queries: $1,070.94 / 2,755 reqs - Excess (was 1.527x): now matches within boundary slop Adds regression tests in TestAdjustDatesForTimezone and TestBuildAggregatedSqlQuery that pin the pass-through behavior and the additivity invariant for any future implementation. * ci: rerun checks on litellm_oss_branch base --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: buffer native gemini sse frames (#30225) * fix: buffer native gemini sse frames * fix: scope native gemini sse buffering * fix: check raw sse residual buffer size * feat: updated openrouter provider to map max level to xhigh (#28881) * feat(proxy): allow use_redis_transaction_buffer without redis cache (#28764) * feat(proxy): allow use_redis_transaction_buffer without redis cache * fix(proxy): require host or url for standalone buffer redis * fix(mcp): fail closed when scope filter resolves to no servers (#30353) `_get_allowed_mcp_servers_from_mcp_server_names` returned the caller's full allowed-server set when the requested `mcp_servers` list (path- or header-derived) resolved to nothing. URL/header namespacing therefore appeared to work even when the requested name was unknown or the caller had no grant — `/mcp/<typo>/` silently exposed every server the key could reach. Fail closed instead: when `mcp_servers` is explicitly provided but nothing resolves, return an empty list. The `mcp_servers=None` path (no scope requested) keeps its existing behavior. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs (#30302) * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs `token_counter` did not know about Anthropic tool-search `tool_reference` content blocks, a lightweight pointer to a deferred tool that shows up as `{"type": "tool_reference", "tool_name": ...}`. When such a block appeared in message content, `_count_content_list` fell through to its catch-all branch and raised `Invalid content item type: tool_reference`. On the streaming `anthropic_messages` proxy path that exception nulls `response_cost`, which makes the proxy drop the entire SpendLogs row. The result is a silent cost undercount on any tool-search traffic; the request succeeds for the caller but the spend is never recorded. This adds a `tool_reference` branch that counts the referenced `tool_name` (the full tool definition is already counted via the `tools` param, so only the name is added here) and handles an empty/missing name gracefully. The catch-all error message is updated to list `tool_reference` among the expected types. A regression test asserts that a message containing a `tool_reference` block no longer raises and returns a positive token count, and that an empty `tool_name` is handled without error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(token-counter): collapse explicit None tool_name to empty string In _count_content_list, c.get("tool_name", "") returns None when the key is present with an explicit None value, and str(None) == "None" which is truthy, causing a spurious token to be counted. Use c.get("tool_name") or "" so both a missing key and an explicit None collapse to an empty string and are skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(token-counter): cover catch-all for unknown content block type Adds a regression test that calls `_count_content_list` with an unrecognized content block type and asserts it raises `ValueError` whose message names the offending type and lists `tool_reference` among the supported types. This exercises the previously uncovered catch-all branch (codecov patch gap) and pins the error contract. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(token-counter): cover tool_reference on the spend/cost and streaming paths Adds end-to-end regression tests that exercise the real public entry points (`completion_cost` and `stream_chunk_builder`), not just the private `_count_content_list` helper, for Anthropic tool-search `tool_reference` content blocks. These pin the actual bug the fix addresses: before the fix the `tool_reference` block raised out of `completion_cost` -> the proxy logging layer nulled `response_cost` and the spend callback dropped the SpendLogs row (silent cost undercount on all tool-search traffic); and `stream_chunk_builder` swallowed the same raise and collapsed prompt_tokens to 0. With the fix, cost is positive and prompt_tokens are counted. Verified: 3 fail without the fix, 3 pass with it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro (#27056) * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix(cost): update backup registry for deepseek-v4 * style: remove print statement from deepseek-v4 test * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix: update deepseek-v4 prices to active discounted rates * test: update deepseek-v4 prices in tests to match active discounted rates * fix(deepseek): remove duplicate entries and update backup registry to active discounted rates * fix: update max_output_tokens to 384K for deepseek-v4 * fix: correctly restore upstream models accidentally dropped during merge * fix(tests): resolve failing claude-fable-5 and reasoning tests by safely updating cost map - Pulled the latest cost map from upstream staging - Safely appended deepseek-v4 mapping without deleting duplicate keys or formatting via json.dump * fix(tests): correct deepseek model cache prices and update JSON schema - Appended both prefixed and bare deepseek-v4 models to satisfy test assertions - Corrected deepseek-v4-pro expected cache hit and token prices based on latest review updates - Added missing realtime endpoint to test_utils.py INTENDED_SCHEMA * fix: remove accidental azure/gpt-realtime-whisper addition --------- Co-authored-by: Dushyant Acharya <dushyantacharya@Dushyants-MacBook-Pro.local> * feat(key/info): expose per-model budget usage in /key/info response (#30394) * feat(key/info): expose per-model budget usage in /key/info response Add model_max_budget_usage to /key/info and /v2/key/info responses. For each model in model_max_budget, reads current-period spend from the same DualCache used by the budget enforcer and returns it alongside the limit and time period so callers can see how much of each model budget has been consumed in the active window. * test(key/info): add coverage for model_max_budget_usage in v1 and v2 endpoints Add tests for the model_max_budget_usage enrichment in both info_key_fn and info_key_fn_v2, covering the budget-present path, the empty-budget path, and the v2 batch endpoint. * fix(key/info): source model_max_budget current_spend from SpendLogs instead of DualCache The DualCache used for enforcement is ephemeral and only populated when budget metadata is present at request time. Fall back to a direct LiteLLM_SpendLogs DB aggregation using the budget period window (budget_reset_at - budget_duration) for accurate reporting. Also fall back to litellm_budget_table.model_max_budget when the key's top-level field is empty, and round current_spend to 4 decimal places. * test(key/info): cover remaining branches in model_max_budget_usage helpers Add unit tests for: prisma_client=None early return, DB query exception swallowing, invalid budget_duration handled by _compute_budget_period_start, budget_reset_at received as a datetime object (Prisma native type), max_seconds=0 early return, and skipping models that lack a budget_duration. Also remove an unreachable except branch where fromisoformat would fail after _compute_budget_period_start already validated the same value. * test(key/info): cover except path for unparseable per-model budget_duration * fix(key/info): compute per-model rolling windows in model_max_budget_usage Each model in model_max_budget now gets its own time window derived from its own budget_duration, rather than sharing a single window computed as the max (or the budget table's reset_at). This matches what the DualCache enforcer actually tracks and prevents current_spend from being inflated for models with shorter windows. _query_model_spend_for_period is refactored to accept a model filter (handling provider-prefix variants in SQL) and return a float directly. _compute_budget_period_start and the budget_table window path are removed as they are no longer needed. * refactor(model_max_budget_limiter): remove dead get_current_period_spend method * refactor(key/info): strip synthetic formatter noise from PR diff Restore key_management_endpoints.py and test_key_management_endpoints.py to origin/litellm_internal_staging, then re-apply only the intentional additions: _query_model_spend_for_period, _build_model_max_budget_usage, the two endpoint patches (info_key_fn / info_key_fn_v2), and the new test suite. The previous commits had reformatted ~300 pre-existing lines across both files, making the functional diff unreadable. * test(key/info): cover empty-rows path in _query_model_spend_for_period * fix(model_max_budget_limiter): guard BudgetConfig construction inside try/except A malformed model entry in the DB (e.g. non-numeric max_budget from a manually edited or migrated row) caused BudgetConfig(**budget_info) to raise a Pydantic ValidationError outside any exception guard, surfacing as a 500 for the entire /key/info or /v2/key/info call. Merging both try/except blocks into one ensures bad entries are silently skipped, consistent with the existing duration_in_seconds guard. * fix: don't stack provider prefix on wildcard models with a custom prefix (#30360) * fix: don't stack provider prefix on wildcard models with a custom prefix get_known_models_from_wildcard expanded provider-prefixed model ids (e.g. "ollama/gemma3:1b" from get_provider_models) by prepending the wildcard's prefix whenever the id did not already start with it. With a custom wildcard prefix such as "ollama_server1/*" (used to distinguish multiple Ollama instances), this produced "ollama_server1/ollama/gemma3:1b", which is uncallable and breaks /v1/models. When the expanded id already carries a provider prefix, replace it with the wildcard's prefix instead of stacking both. Matching-prefix and bare-model cases are unchanged. Fixes #30358 * fix: only strip a known provider prefix when expanding custom wildcard prefixes The wildcard expansion replaced the leading slash segment of every expanded id with the wildcard prefix whenever the id did not already start with it. For ids whose first segment is an org rather than a litellm provider (for example a provider returning "meta-llama/Llama-3-8B" with no outer provider prefix), that dropped the org and produced an uncallable id Only strip the leading segment when it is a recognized provider (membership in LlmProviders); otherwise keep it and just prepend the wildcard prefix. Provider-prefixed ids like "ollama/gemma3:1b" still have their prefix replaced, so the original fix is unchanged for known providers * address greptile review feedback: log dropped non-text vLLM assistant content blocks (greploop iteration 1) * fix(ci): format credential_form_helpers test + regenerate dashboard schema.d.ts * fix(proxy): raise litellm.BadRequestError for missing model param When no model is passed, route_request now raises a litellm.BadRequestError ('Missing model parameter') instead of falling through to ProxyModelNotFoundError. This keeps the missing-param error clear and independent of router wildcard state. Unknown (non-empty) model names still raise ProxyModelNotFoundError. * Revert "fix(proxy): raise litellm.BadRequestError for missing model param" This reverts commit 9240da403c. * Revert "fix(router): clean pattern_router state on upsert/delete (#29601)" This reverts commit ad4e6e2395. * fix: correct streaming and key budget usage reporting * fix(hosted_vllm): type assistant tool_calls to satisfy mypy * feat: aws secret manager cross region replication (#30368) * feat(aws-secret-manager): add replica_regions cross-region replication after CreateSecret When store_virtual_keys is enabled, async_write_secret() only wrote secrets to the primary AWS region. Multi-region proxy deployments had no built-in way to synchronize virtual key secrets across regions through LiteLLM, requiring external replication mechanisms. Add replica_regions support to AWSSecretsManagerV2: - New replica_regions field in KeyManagementSettings (types/secret_managers/main.py) - New async_replicate_secret() method that calls ReplicateSecretToRegions API - async_write_secret() calls replication after successful CreateSecret - Replication failure is logged as a warning but does NOT fail key creation - load_aws_secret_manager() forwards replica_regions from key_management_settings Configuration example: key_management_settings: store_virtual_keys: true replica_regions: - us-west-2 - eu-west-1 When replica_regions is omitted or empty, behavior is unchanged. * test(aws-secret-manager): restore litellm.secret_manager_client after test to prevent state pollution * test(aws-secret-manager): add coverage for HTTP error and replication exception paths * fix: restore litellm.secret_manager_client global state in test; add replication log proof - Global state in test_load_aws_secret_manager_passes_replica_regions was already guarded with try/finally (committed in previous pass); no further change needed for Fix 1. - Fix 2: add verbose_logger.info("ReplicateSecretToRegions called …") inside async_replicate_secret so callers get an observable INFO log line whenever replication fires. - Add test_replication_fires_on_create: calls async_replicate_secret directly with caplog.at_level(INFO, logger="LiteLLM") and asserts "ReplicateSecretToRegions" appears in the captured log output, proving the code path executes. * fix: pass request to streaming generators * fix(hosted-vllm): preserve assistant structured content * fix(hosted_vllm): satisfy mypy on preserved structured content assignment * chore: resolve litellm_internal_staging merge conflicts for #30527 (#30554) * chore(codecov): add Batches, Videos, and Realtime components (#30517) * chore(codecov): add Batches, Videos, and Realtime components Define per-feature Codecov components so PR comments track coverage for batch API, video generation, and realtime streaming paths. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(codecov): use wildcard path for Batches proxy component Align batches_endpoints glob with Videos, Realtime, and Proxy_Authentication. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) Four batch-related tests lived under tests/litellm/ and were never picked up by GitHub Actions. Relocate them and fix gemini multimodal e2e to use the batchEmbedContents path expected for gemini/ provider. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): run pre_call hook once for model-level guardrails (#30543) * fix(guardrails): run pre_call hook once for model-level guardrails A CustomGuardrail attached to a deployment via litellm_params.guardrails gets its async_pre_call_hook invoked twice per request: once by the proxy pre-call loop and again by async_pre_call_deployment_hook after the router spreads the model-level guardrails into the top-level request kwargs. Record in request metadata that the proxy pre-call loop already ran a given guardrail, and have the deployment hook skip it when the marker is present. Direct-SDK usage never runs the proxy loop, so the deployment hook stays the sole invocation there and still fires exactly once. The marker key is stripped from untrusted caller metadata so a request body cannot suppress a model-only guardrail by pre-seeding it. * fix(guardrails): mark pre_call dedup on the post-hook request data Record the exactly-once marker after async_pre_call_hook runs, on the data object that flows downstream, rather than before it. A guardrail whose hook returns a brand-new request dict (instead of mutating or spreading the one it received) would otherwise discard the marker, letting the deployment hook re-run the guardrail a second time. * fix(guardrails): stop re-initializing DB guardrails on every poll (#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself. * chore(oss): litellm oss staging 150626 (#30463) * fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415) * fix(pricing): add GitHub Copilot MAI Code Flash pricing Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(pricing): cover GitHub Copilot MAI Code Flash pricing Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213) * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) #28990 added ownership recording for streaming /v1/responses via _wrap_responses_stream_for_container_ownership, which reads `getattr(stream_response, 'completed_response', None)` to extract the ResponsesAPIResponse. The unit test bypassed the Router, so it never exercised the production wrapping path. Through the Router (every proxy deployment), the stream is wrapped by FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set `self.completed_response = None` and __anext__ only forwarded chunks — the inner source iterator's terminal event never bubbled up to the attribute the ownership hook reads, so the hook silently recorded nothing and every follow-up /v1/containers/<id>/files call returned 403 for non-admin keys. This commit: - router.py: pre-resolves the responses-API terminal event tuple (response.completed / .incomplete / .failed) once per _aresponses_streaming_iterator call, and has the wrapper's __anext__ sniff each forwarded chunk's .type. First terminal event hit gets stored on the wrapper's completed_response. Iterator-agnostic — works for source_iterator AND any future wrapper. - common_request_processing.py: when _extract_completed_responses_response returns None we now warn instead of silently skipping. Reporter on #30210 lost a day to this exact silent skip; the warning surfaces future regressions of the same shape directly in operator logs. Fixes #30210 * fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments in FallbackResponsesStreamWrapper.__init__: router.py:2564 self.response = getattr(source_iterator, 'response', None) router.py:2565 self.model = getattr(source_iterator, 'model', None) router.py:2566 self.logging_obj = getattr(..., None) Those lines also exist on litellm_internal_staging and pass mypy there. Adding the typed terminal-event tuple above the class made the function body more narrowable, which surfaced the pre-existing mismatch — base class declares non-Optional types but the bridge path (LiteLLMCompletionStreamingIterator) legitimately omits these. Keep the None fallback and silence with type: ignore[assignment]. Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter which misleads operators when a non-code_interpreter stream aborts. Generalize to 'any tool container (e.g. code_interpreter)'. * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201) * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0 when they are absent from the raw entry (the price-unknown and free cases share the same representation). register_model then merges that result back into litellm.model_cost, which flips a sparse entry from 'no cost keys' (priced via model name) to 'cost keys = 0' (free). That defeats _is_cost_explicitly_configured (#24949) on re-registration: _is_model_cost_zero returns True, common_checks skips every tag / key / team / user / org budget check for the group, and over-budget traffic keeps returning 200. Spend keeps recording because cost calc still resolves by model name, so the symptom is silent and only triggers on the second register_model pass (router rebuild, /model/update, config sync). Mirror the existing litellm_provider-None guard one block above and pop the cost fields from the synthesized result when they are absent from the raw entry and not in the caller's value. Caller-provided zeros (genuinely free models, BYOK overrides) are preserved. Fixes #30198 * fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion Greptile #30201 review notes: - the `or`-chain in the raw-entry lookup treated an empty dict (a key with no fields) as falsy and fell through to the second arm — replace with explicit `is None` checks so a present-but-empty entry is still taken at face value. - the first assertion in `test_router_double_init_keeps_db_model_entry_sparse` used `in (None, 0)` which passes under the bug condition (cost = 0 matches the tuple); the strong follow-up assertion already covers every shape, so drop the dead branch. * fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426) * fix(bedrock mantle): use unique function-call id for responses->chat tool calls ... * fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved. * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241) * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) Router.get_deployment_credentials_with_provider re-validates a deployment's litellm_params through CredentialLiteLLMParams before handing them to file/batch/passthrough callers: return CredentialLiteLLMParams( **deployment.litellm_params.model_dump(exclude_none=True) ).model_dump(exclude_none=True) Any field NOT declared on CredentialLiteLLMParams gets silently dropped on the way through. azure_ad_token was undeclared, so Azure deployments using OAuth/M2M (azure_ad_token instead of a static api_key) silently lost their token at the files endpoint and the proxy returned: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, ... Declare azure_ad_token on CredentialLiteLLMParams alongside api_key / api_base / api_version so it rides through the round-trip. Static-key deployments stay unaffected (Optional, default None, dropped by exclude_none=True). Provider-callable (azure_ad_token_provider) is a separate concern and out of scope here. Fixes #30235 * fix(ui-types): regenerate schema.d.ts for new azure_ad_token field CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check auto-detected the new field and emitted the exact diff to apply. Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams, both get the new azure_ad_token marker next to it. * fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247) When the UI sends the callers own user_id (as it does for non-Admin global roles), _enforce_list_team_v2_access now nulls it out for org admins so _build_team_list_where_conditions scopes by organization_id only -- matching the legacy /team/list behavior and the documented intent. Fixes #30215 Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707) litellm_internal_staging already routes the cachedContents URL through get_vertex_base_url, fixing the multi-region 404 reported in #29571 — but carries no test coverage for the actual regression scenario (eu/us must resolve to the REP host aiplatform.{geo}.rep.googleapis.com). Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host assertions (including absence of the old broken {geo}-aiplatform host), plus regional (us-central1) and global no-regression checks. * fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245) * fix(proxy): close upstream LLM stream when client disconnects mid-stream When a streaming client disconnects, Starlette abandons the response body iterator without calling aclose(), so the proxy's connection to the upstream backend stays open until garbage collection, which may never come. The backend (e.g. vLLM) keeps generating into a dead pipe: small responses drain invisibly into TCP buffers while large ones block the backend on a full send buffer indefinitely (observed via lsof as an ESTABLISHED proxy->backend connection minutes after the client left) create_response now returns a StreamingResponse subclass that closes both its body iterator and the wrapped upstream-facing generator in a shielded finally. The upstream generator is closed directly rather than through a cascade because aclose() on a never-started generator skips its body, which would make the cascade a no-op when the client disconnects before the first chunk is sent. async_streaming_data_generator also gains the same shielded finally-aclose that async_data_generator in proxy_server.py already had, covering the Anthropic and Google SSE paths With this, killing a streaming client causes the backend to observe the abort within about a second and free its slot, while completed streams are unaffected. No flag is needed, unlike the non-streaming opt-in cancel in #30223: this only releases resources after the client is already gone and does not change any response a client can observe Fixes #30244 * fix(proxy): close upstream even when body iterator aclose raises BaseException Addresses the Greptile finding on #30245: the cleanup loop caught only Exception while the generator-level cleanup catches BaseException, so a CancelledError or GeneratorExit escaping body_iterator.aclose() would skip closing the upstream generator. Both sites now use the same scope and a regression test pins that the upstream is closed even when the body iterator explodes with a BaseException * fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection The response-level close added for #30244 only worked for SDK-based providers (e.g. openai), whose streams expose aclose all the way down. Providers served by base_llm_http_handler (hosted_vllm and most modern transformation-based providers) wrap a bare response.aiter_lines() generator in BaseModelResponseIterator, which had no aclose or close at all, and nothing retained the httpx response object; so CustomStreamWrapper.aclose() silently did nothing and the upstream connection stayed open. Verified with a vLLM-style mock: with hosted_vllm/ the backend streamed all 100 chunks to completion after the client disconnected, while openai/ aborted at chunk 6 BaseModelResponseIterator now carries an optional http_response and an aclose() that closes it; make_async_call_stream_helper attaches the response after building the iterator. With this, hosted_vllm aborts the backend within ~1.6s of the client dropping, and completed streams are unaffected --------- Co-authored-by: kursad <kursad.lacin@brado.net> * feat(anthropic): surface compaction usage iterations data (#27065) * feat(anthropic): surface compaction usage iterations data * style: apply black formatting to fix lint checks * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422) * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock * fix(usage): optimize test imports * feat: add fastCRW search provider (#30434) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider * libertai: update served endpoints backup + add mode/matrix tests Addresses review feedback: - Add libertai to litellm/provider_endpoints_support_backup.json, the file actually served by GET /public/supported_endpoints (the root provider_endpoints_support.json already had it). - Add tests asserting bge-m3 normalizes to mode='embedding' and that the served matrix lists libertai. embeddings stays false: the JSON-configured provider path only wires chat routing (OpenAILike embedding handler is reached only for literal openai_like/llamafile/lm_studio), matching the llamagate precedent; bge-m3 remains in the cost map for metadata. --------- Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> * feat(provider): add ModelScope as an OpenAI-compatible provider (#28460) * add ModelScope API support * add modelscope api support * update modelscope model list * add image-genetation support * update test and multimodal * fix: address PR review feedback for modelscope provider * update README * fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849) * fix(customer_endpoints): restrict /customer/daily/activity to admin-only * fix(customer_endpoints): check role before prisma_client guard * fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563) * fix(fallbacks): preserve fallback model in SDK fallback responses (#28260) * fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks * fix(fallbacks): gate x-litellm-* passthrough to trusted callers only The previous patch unconditionally let `x-litellm-*` keys bypass the `llm_provider-` prefix in `process_response_headers`. That function is also called on raw upstream-provider response headers (e.g. from `llm_http_handler.py`), so a malicious provider could return `x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker, bypassing the proxy model-override guard. Add a `preserve_litellm_internal_headers` flag (default False). Only `response_metadata.py`, which re-processes the already-built `_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes True. Raw provider header callsites keep the default False, so upstream `x-litellm-*` still gets the `llm_provider-` prefix. Adds a regression test for the spoofing case and renames the existing preserve test to make the trusted-path semantics explicit. * fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs * style(core_helpers): apply black formatting * fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): apply black formatting to modelscope chat transformation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): remove unused AllMessageValues import Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert: restore base_model_iterator.py to original PR state Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813. * fix(lint): add @override to modelscope image generation overrides Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913. --------- Co-authored-by: Joel Tony <github@jaytau.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: ztko <96878659+koztkozt@users.noreply.github.com> Co-authored-by: Nahrin <nahrin@nahrinoda.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Humphrey <a739376838@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: Dushyant Acharya <dushyantacharya873@gmail.com> Co-authored-by: Yuriy <yuriy.shuyskiy@gmail.com> Co-authored-by: Recep S <22618852+us@users.noreply.github.com> Co-authored-by: Moshe Malawach <moshe.malawach@protonmail.com> Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> Co-authored-by: Rongkun Yan <2493404415@qq.com> Co-authored-by: Varshith <kvarshithgowda@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> * ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules (#30516) * ci(lint): enforce blanket-noqa, dataclass-default, and unused-noqa rules Enable PGH004 (blanket-noqa), RUF008 (mutable-dataclass-default), RUF009 (function-call-in-dataclass-default-argument), and RUF100 (unused-noqa) in ruff.toml, and clean up every resulting violation. RUF008/RUF009 were already clean. PGH004/RUF100 surfaced ~335 stale or blanket noqas: blanket `# noqa` are now scoped to the rule they actually suppress (mostly T201), dead directives are removed, and inapplicable codes are trimmed (e.g. F401 dropped from `import *`). lint.external lists rules enforced outside this config (the strict-rule gate via ruff-strict.toml and upstream litellm's own ruff config) so RUF100 keeps the noqa directives that protect them instead of stripping coverage this config can't see. * ci(lint): trim RUF100 external list to load-bearing codes only Drop the 9 precautionary strict-gate codes (ANN001/002/003/401, B006, PLR0913, PLW0603, RUF012, TID251) that have zero `# noqa` references in the gated source. Keep only the 11 codes with live suppressions so RUF100 doesn't flag them as unused. Future strict-gate suppressions can re-add codes here (or fix the underlying issue) as needed. * ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379) * ci: enable ruff preview rules under the budgeted strict gate Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only, leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at zero. Enumerate the 118 firing codes explicitly with explicit-preview-rules so the gate is deterministic and stable across ruff upgrades rather than depending on preview auto-selecting the broad catalog. Grandfather the existing 58438 violations into ruff-strict-budget.json as per-rule baselines with headroom, so only net-new violations fail CI. The existing ten rules keep their hand-tuned slack; the new rules get slack 10 when the baseline is 50 or more and 3 otherwise. * ci: add ANN return-type rules to the budgeted strict gate Add ANN201/202/204/205/206 (missing return annotations) to the strict lane and grandfather the existing counts into ruff-strict-budget.json so the codebase ratchets toward explicit return types without breaking CI. * ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines Add two type-check gates, each grandfathering the current tree so only net-new violations fail CI, matching the ruff strict-budget ratchet. mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI invocation actually reads; the root [tool.mypy] is not picked up from the litellm/ working dir). The 4885 existing missing-annotation errors are captured in litellm/.mypy-baseline.txt and the run is piped through mypy-baseline filter so new untyped defs are rejected. basedpyright runs in strict mode over litellm/, with enableTypeIgnoreComments disabled so it only honors '# pyright: ignore' and never polices mypy's '# type: ignore'. The existing strict diagnostics are grandfathered into .basedpyright/baseline.json. Both tools are pinned in the dev group and uv.lock; the lint workflow and Makefile run them filtered through their baselines, with lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet. * ci: raise lint job timeout to 15m for the basedpyright strict pass * ci: pin pythonVersion 3.12 and regenerate baselines against merged base Merge litellm_internal_staging so the baselines cover code the CI merge includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is reproducible across interpreter versions (CI runs 3.12). * ci: regenerate basedpyright baseline against the frozen lint env The previous baseline was generated with optional provider deps (azure, google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors not in the baseline. Regenerate after uv sync --frozen so the baseline reflects the same dependency set the lint job sees. * ci: regenerate basedpyright baseline on python 3.12 frozen env The prior baseline still carried proxy-dev packages (e.g. prisma) that the lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import errors ungrandfathered. Regenerate in a python 3.12 venv synced to the frozen lock with default groups only, so the baseline matches exactly what CI sees. * ci: replace type-check baselines with per-file count budgets The mypy and basedpyright baselines were position-sensitive (and the basedpyright one was a 27MB file), so ordinary line shifts churned them. Replace both with a per-file count gate: scripts/type_check_gate.py reduces each tool's output to errors-per-file and checks it against a committed {file: max} budget, ignoring line and column numbers. A file fails only when it gains more errors than its ceiling; debt can't be shuffled between files because each file has its own cap and new files default to zero. Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are generated in the python 3.12 frozen lint env so they match CI. Drops the mypy-baseline dependency; basedpyright runs without its native baseline. ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update. * ci: add a small per-file slack to the type-check gate Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count before failing, so a basedpyright inference ripple in an unrelated file doesn't break the build over a couple of errors. Budgets still record exact counts; the tolerance is applied at check time. * ci: move type-check slack into the budget json and trim lint timeout Make slack declarative: the budget is now {"slack": N, "files": {path: count}} so the tolerance is tuned in JSON without editing the script, mirroring how ruff-strict-budget.json carries its slack. --update preserves the existing slack. Also drop the lint job timeout from 15m to 10m; the mypy and basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a comfortable margin. * ci: collapse fully-adopted ruff categories and drop inert preview flag ANN (all nine non-removed rules) and BLE (its only rule) were spelled out code-by-code; replace each with its category selector, which is exactly equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category selector and error when named explicitly). explicit-preview-rules was inert: every selected rule is stable and nothing is selected by category, so the flag had nothing to gate. Verified the strict-rule counts are identical before and after (62379 each, zero per-rule drift), so no budget change. * ci: drop redundant pyright dev dependency Nothing invokes bare pyright in the Makefile, the linting workflow, or scripts; the basedpyright gate added on this branch is the only type checker that runs. basedpyright is a superset fork that reads the same pyrightconfig.json and honors the same "# pyright: ignore" comments, so pyright==1.1.408 in the ci group was dead weight. Regenerated uv.lock under the same exclude-newer cutoff so the only change is removing pyright and its package stanza * ci: un-weaken mypy and error on Any in basedpyright mypy: enable warn_return_any, drop the valid-type silencer, and stop globally ignoring missing first-party imports via [mypy-litellm.*] ignore_missing_imports = False, which surfaced eight real broken litellm.* imports the blanket ignore was hiding; third-party imports stay ignored. The per-file budget moves 4888 -> 5799 (902 no-any-return, 1 valid-type, 8 import-not-found), all grandfathered so only net-new errors fail and the ceilings ratchet down basedpyright: error on reportExplicitAny and reportAny. The per-file budget moves 117033 -> 148946 (6931 explicit-Any, 24954 Any-typed expressions), grandfathered the same way * ci: add Any-discipline gate on changed lines under litellm/ Add scripts/check_any_discipline.py, a type-aware gate that fails when a changed line holds a value typed Any -- including the X | Any unions that mypy --strict / basedpyright accept (e.g. re.Match.group() -> str | Any, json.loads() -> Any, bare dict -> dict[Any, Any]). It reuses the repo's mypyc-compiled mypy 1.19 via a custom generic AST walker (mypyc precludes subclassing TraverserVisitor), loads litellm/mypy.ini for parity with lint-mypy, and uses a dedicated incremental cache (.mypy_cache_any) with mtime+hash invalidation to force re-checks. Scope is changed-lines-only so editing a legacy file never forces cleaning its existing Any debt; suppress a genuine typed/untyped boundary with # any-ok: <reason> (ANY002 requires the reason). Wire it into the Makefile (lint-any, lint, lint-dev), a parallel any-discipline CI job with its own actions/cache, .gitignore, and the CLAUDE.md / CONTRIBUTING.md docs. * ci: move Any-gate codes into the shared LIT namespace Renumber the Any-discipline checker into the LIT*** scheme owned by scripts/check_type_discipline.py (PR #30500) so the two checkers share one rule namespace and suppression convention: ANY001 -> LIT002 (Any-typed value; LIT002 was the retired/free slot) ANY002 -> LIT005 (any-ok without a reason; the shared suppression-reason code) ANY000 -> LIT000 (setup/build/read error; the shared error code) Messages and behavior are unchanged; LIT005's text already matches the "<token> requires a reason" shape used for cast-ok/guard-ok. * ci: gate mypy and basedpyright per error rule, not per file Switch the mypy/basedpyright budget gate from per-file error counts to per-rule-code totals, mirroring the {rule: {baseline, slack}} shape of ruff-strict-budget.json. A rule fails when its codebase-wide error count exceeds baseline + slack, so violations are tracked by category rather than by file location. scripts/type_check_gate.py now parses mypy from its text output (trailing [code]) and basedpyright from --outputjson (the JSON `rule` field), since basedpyright's wrapped text diagnostics mis-attribute the rule on continuation lines. Replace the *-file-budget.json files with freshly captured *-code-budget.json baselines and update the Makefile, CI, and CLAUDE.md accordingly. * docs: prefer Pydantic validation over any-ok suppression Point the Any-discipline guidance at validating Any with Pydantic (a model or TypeAdapter that returns a typed value or raises) and frame # any-ok as a last resort that should ideally never be used. * chore: remove extraneous comment * chore: make the CLAUDE.md more concise * chore: clean up bloated CONTRIBUTING.md additions * chore: make Makefile more concise * ci: add the lint-budget-update target CLAUDE.md references CLAUDE.md tells contributors to run make lint-budget-update, but the target was never defined. Add it as an aggregate that re-captures the ruff, mypy, and basedpyright budgets in one shot. * ci: recapture mypy and basedpyright budgets in the lint env The per-rule baselines were captured in a richer dependency env than the CI lint job's uv sync --frozen, so CI resolved fewer types and reported more errors than the budgets allowed (no-any-return 902 over cap 900, plus several basedpyright reportUnknown* rules). Regenerate both in the frozen env so they grandfather the true CI debt: mypy 5786 -> 5799 (no-any-return 890 -> 902, valid-type 1 restored), basedpyright 146213 -> 148942. * ci: check out PR head sha in lint and any-discipline jobs The default pull_request checkout uses refs/pull/N/merge, which folds the latest base commits into HEAD. The diff-based gates (ruff delta, Any discipline) then diff against the event's older base.sha and blame base's own new commits on this branch; staging's otel-v2 and streaming changes (#30326, #30485) tripped the Any gate on files this branch never touched. Checking out the PR head sha makes the gates diff the real branch tip against base, and pins the tree the mypy/basedpyright budgets were captured against so their counts stay deterministic as the base advances. * ci(lint): renumber Any-typed-value rule LIT002 -> LIT009 Free up LIT002 for the sibling type-discipline gate (check_type_discipline.py, #30500), which groups its mutable-collection family at LIT001 (annotation) and LIT002 (construction). This gate's Any-typed-value rule moves to LIT009 so the shared LIT namespace stays contiguous with no holes; LIT000 and LIT005 are unchanged. * style: rename lint-strict-budget -> lint-ruff-budget * ci: harden type-check gates against silent passes (greptile review) type_check_gate.py: refuse to certify a vacuous run. The CI pipe swallows the tool's exit code ('tool || true'), so a crashed mypy/basedpyright that emits nothing would parse to zero errors, breach no ceiling, and pass. is_vacuous_run() now fails when nothing was parsed but the budget expects errors. Also wrap basedpyright's json.loads in a JSONDecodeError handler that prints the offending output instead of dumping a raw traceback. check_any_discipline.py: ALL_LINES was None, which dict.get() also returns for a path absent from the line map, so a path-normalisation mismatch could let a violation on an unchanged file pass the scope filter. Make ALL_LINES a distinct sentinel object so 'whole file' and 'path missing' are unambiguous. Adds tests for all three. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Joel Tony <github@jaytau.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: ztko <96878659+koztkozt@users.noreply.github.com> Co-authored-by: Nahrin <nahrin@nahrinoda.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Humphrey <a739376838@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: Dushyant Acharya <dushyantacharya873@gmail.com> Co-authored-by: Yuriy <yuriy.shuyskiy@gmail.com> Co-authored-by: Recep S <22618852+us@users.noreply.github.com> Co-authored-by: Moshe Malawach <moshe.malawach@protonmail.com> Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> Co-authored-by: Rongkun Yan <2493404415@qq.com> Co-authored-by: Varshith <kvarshithgowda@gmail.com> * chore: satisfy strict-rule and any-discipline gates for the staging bundle The strict-rule budget and any-discipline gates added in #30379 flag the bundle's new lines: blind-except (BLE001), legacy typing imports (UP006/UP035/UP045), and values typed Any on changed lines (LIT009). Type-fix the cleanly-fixable cases (function signatures, payload dicts as dict[str, object], BudgetConfig.model_validate over **kwargs, direct KeyManagementSettings attribute access over getattr, Optional[X] -> X | None) and suppress the irreducible untyped boundaries (request/streaming dicts, cache reads, httpx responses, asyncio primitives, Pydantic model_dump navigation) with # any-ok and a short reason. Also fix two any-discipline gate false positives so legitimate code is no longer flagged: the synthetic Any in Coroutine/Generator send and yield protocol slots (the awaited/returned value is still checked), and the special-form Any of a TypedDict field's TempNode rvalue placeholder. * chore: extend basedpyright slack to the two rules #30563 left at default PR #30563 raised basedpyright slack to ~10% of baseline across the noisy reportUnknown*/reportAny family so staging bundles clear the per-rule gate, but it left reportArgumentType (slack 3) and reportPrivateUsage (slack 10) at their original tight values. This bundle pushes those two 10 and 1 over their caps respectively, so apply the same ~10% policy: reportArgumentType baseline 1863 -> slack 180, reportPrivateUsage baseline 1625 -> slack 160. No baselines move; only the slack on these two rules * fix: handle duplicate tool calls and stream tail disconnects * fix(proxy): mark stream completed before tail yields, not after [DONE] Clients routinely close the connection right after the final chunk or the terminating data: [DONE] frame. Setting stream_completed only after those trailing yields made the GeneratorExit from that close fall into the disconnect branch, recording false 499 client_disconnected metadata for a response that already delivered all content and fired success logging, and double-releasing the max_parallel_requests slot the success callback had already released. Restore stream_completed before the trailing raw-SSE, error, and [DONE] yields so terminal-marker closes are treated as the successful completions they are. The tool_use dedupe guard is kept. --------- Co-authored-by: apshada <49001649+apshada@users.noreply.github.com> Co-authored-by: Aarkin Karnik <56022539+Aarkin7@users.noreply.github.com> Co-authored-by: David Bochenski <david@goincremental.com> Co-authored-by: Cai Songrui <1922909737@qq.com> Co-authored-by: Martin Honermeyer <7229+djmaze@users.noreply.github.com> Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Anuj ojha <ojhaanuj224@gmail.com> Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com> Co-authored-by: Zekeriya Akgül <zkry.akgul@gmail.com> Co-authored-by: Thomas Menard <menardorama@gmail.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Mark Lopez <m@silvenga.com> Co-authored-by: Varshith <kvarshithgowda@gmail.com> Co-authored-by: Huynh Duc Tran <110240973+hdt12a1@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Samarth Maganahalli <samarth.maganahalli@rubrik.com> Co-authored-by: Dushyant Acharya <dushyantacharya873@gmail.com> Co-authored-by: Dushyant Acharya <dushyantacharya@Dushyants-MacBook-Pro.local> Co-authored-by: Thijmen Stavenuiter <thijmenstavenuiter@gmail.com> Co-authored-by: Vineeth Sai <vineethsai4444@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: rvishwas26 <rvishwas@athenahealth.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Joel Tony <github@jaytau.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: ztko <96878659+koztkozt@users.noreply.github.com> Co-authored-by: Nahrin <nahrin@nahrinoda.com> Co-authored-by: Humphrey <a739376838@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: Yuriy <yuriy.shuyskiy@gmail.com> Co-authored-by: Recep S <22618852+us@users.noreply.github.com> Co-authored-by: Moshe Malawach <moshe.malawach@protonmail.com> Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> Co-authored-by: Rongkun Yan <2493404415@qq.com>	2026-06-16 18:23:13 -07:00
Mateo Wang	be4fa702e7	ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) (#30500) ... * ci(lint): enforce type-discipline budget for casts and type guards Add a ratcheted gate that blocks net-new typing.cast() usage and bans TypeGuard/TypeIs outright, layered on the existing ruff-strict budget setup. - ruff-strict.toml: ban cast/TypeGuard/TypeIs (typing + typing_extensions) via flake8-tidy-imports banned-api (TID251) for a coarse import-level freeze. - ruff-strict-budget.json: bump TID251 baseline 2404 -> 2662 to absorb the ~258 pre-existing usages now matched by the new banned-api entries. - scripts/check_type_discipline.py: AST checker adding LIT006 (cast call sites, suppress with `# cast-ok: <reason>`) and LIT007 (TypeGuard/TypeIs annotations, suppress with `# guard-ok: <reason>`) for per-call-site granularity. - scripts/type_discipline_gate.py: baseline+slack gate with delta-vs-base, mirroring ruff_strict_gate.py. - type-discipline-budget.json: LIT006 baseline 1013 (slack 10), LIT007 0/0. - test-linting.yml: run the gate in CI against the PR base SHA. * ci(lint): enforce suppression-reason budgets and guard budgets against loosening - wire the **kwargs ban (LIT008) into the vendored type-discipline checker so it matches the budget that already referenced it - freeze LIT003/LIT004 (noqa / type-ignore without codes or reason) and LIT005 (*-ok suppression without a reason) at slack 0 so any net-new unexplained suppression trips the type-discipline gate - add scripts/budget_ratchet_check.py and a separate, non-gating budget-ratchet CI job that turns red when any *-budget.json ceiling is raised, a rule is dropped, or a budget file is deleted * ci(lint): ban mutable collections in annotations and all mutable construction Expand LIT001 from coarse builtins at interfaces to any mutable collection in any annotation (builtins, typing aliases, collections concretes, mutable ABCs) across signatures, class attributes, locals, and globals. Add LIT009 to flag mutable-collection construction (literals, comprehensions, constructors) so the unannotated seed-then-mutate pattern is caught too. Enumerate any-ok in LIT005 so its reason requirement holds even when only the stdlib checker runs. Budget LIT001 (21452) and LIT009 (25222) with slack 10 to ratchet down. * ci(lint): recommend pydantic at boundaries and add functional-refactor guidance Drop the msgspec mention from the cast banned-api messages so the recommended validation path matches the codebase's primary pattern (pydantic). Add a note to CLAUDE.md that lint / type-discipline failures should be resolved by refactoring to functional, immutable patterns rather than reaching for mutable structures or `# mutable-ok`. * style: make CLAUDE.md more concise * chore: update CLAUDE.md guidelines * ci(lint): renumber mutable construction LIT009 -> LIT002 next to LIT001 Group the mutable-collection family together: LIT001 (mutable collection in any annotation) and the construction rule now sit adjacent at LIT001/LIT002. The freed LIT009 slot is taken by the sibling Any gate (check_any_discipline.py, #30379), which moves its Any-typed-value rule LIT002 -> LIT009 in lockstep so the shared LIT namespace stays contiguous with no holes. Budget, gate docstring, and the checker's own docstring/messages are updated to match. * fix: numbering in CLAUDE.md * test(lint): test type-discipline checker, scope LIT007 to return types Add regression tests for check_type_discipline.py (every LIT rule, its suppression, and the comment scanner) and for budget_ratchet_check.py. Confine LIT007 to function return annotations, the only place TypeGuard/TypeIs are valid, so a runtime name that merely reads those identifiers is no longer flagged. Switch scan_comments to io.StringIO(source).readline, the standard readline that returns '' at EOF, dropping the iter(...).__next__ idiom. * fix(lint): best-effort worktree teardown so cleanup can't mask the real error base_counts ran `git worktree remove` through the raising `_run` in its finally, so a failed `git worktree add` (or a failure in the body) was masked by a second SystemExit from the cleanup. Tear the worktree down best-effort, like the sibling rmtree, so the original error propagates. * fix(lint): ratchet fails loudly on an unresolvable base; drop dead checker state Verify the merge-base ref resolves to a commit before trusting a missing-file result from git show, so an invalid or empty BASE_SHA now turns the budget-ratchet guard red instead of skipping every budget and passing vacuously Also drop the unused Comments.by_line field and the phantom --changed-only usage line from check_type_discipline's docstring, and cover the ref handling with tests * fix(lint): degrade malformed source to LIT000 instead of crashing the checker tokenize.generate_tokens raises IndentationError (a SyntaxError subclass) on a dedent mismatch, which escaped scan_comments' tokenize.TokenError handler and crashed the whole checker run, zeroing the gate for that invocation. Catch SyntaxError too so the file falls through to ast.parse and is reported as LIT000, matching the checker's graceful-degradation contract. Also add the trailing newline ruff-strict.toml lacked * perf(lint): skip the base worktree scan when no rule is over its ceiling cmd_check created a git worktree and re-scanned the base tree on every run, but a rule can only breach when its head count is already over baseline + slack; when none are, the base comparison cannot change the verdict. Short-circuit to OK in that case, which is every green PR, roughly halving the gate's work. Extract over_ceiling and cover it (and evaluate's drift-safety) with tests * fix(lint): exempt .dict()/.list()/.set() method calls from LIT002 _construction_kind matched dict/list/set as constructors via func.attr too, flagging common method calls like pydantic's model.dict() as mutable construction; 200 such false positives existed in litellm. Recognize dict/list/set construction only when unqualified while keeping the collections concretes (deque/defaultdict/...) matchable as attributes, since those are rarely method names. Ratchet the LIT002 baseline down 25222 -> 25022 to reflect the removed false positives * chore(lint): bump basedpyright ceilings to absorb staging base drift The basedpyright gate added in #30379 is a total-count check against basedpyright-code-budget.json and the linting workflow runs only on pull_request, so pushes to litellm_internal_staging never re-baseline it. Merging staging into this branch surfaced that drift: seven reportAny/reportUnknown* rules sit 10-149 errors above their committed ceiling even though this PR changes no files under litellm/, the only path basedpyright scans (pyrightconfig include is litellm). The new baselines match the counts CI measured on the merge commit, with the existing per-rule slack preserved * fix(lint): ratchet guard watches every budget file, not just two DEFAULT_BUDGETS only listed ruff-strict-budget.json and type-discipline-budget.json, so mypy-code-budget.json and basedpyright-code-budget.json were unguarded and their ceilings could rise with no signal, which is exactly the failure mode this guard exists to prevent. The gap became concrete when this PR bumped basedpyright-code-budget.json to absorb staging drift. All four budgets are now watched, so the budget-ratchet job surfaces that basedpyright bump for human review the same way it surfaces the TID251 raise. A regression test pins that every *-budget.json on disk is in DEFAULT_BUDGETS, failing loudly if a future budget escapes the ratchet * fix: add a lot more slack * fix(lint): restore LIT003 frozen slack to 0 The blanket slack bump set LIT003 (bare # noqa without codes or a reason) to a slack of 50, which contradicts the documented zero-tolerance invariant: the gate docstring and the PR description table both freeze LIT003/LIT004/LIT005 at slack 0 so any net-new unexplained suppression trips the gate. Slack 50 would let 50 new bare noqas through silently. The actual LIT003 count is 397, well under the 516 baseline, so restoring slack to 0 keeps the gate green while putting the freeze back. LIT004/LIT005/LIT007 were already correct at 0 * fix(lint): restore documented slack 10 for the buffered LIT rules The slack bump left LIT001/LIT002/LIT006/LIT008 at 2000/2500/100/100, 10-250x the "/ 10" the PR description table and the gate docstring document. That buffer was never needed: the gate already blames a rule only when its count exceeds the ceiling and grew vs the merge-base, so the violations the staging merge added in litellm/ sit in both head and base and are never charged to this PR. With slack back at the documented 10 the gate stays green, and the ceiling is tight again (LIT006 no longer waves through 99 net-new cast() calls). Baselines are unchanged; only the slack returns to its documented value * fix(lint): ratchet LIT003 baseline down to its actual count The LIT003 baseline was 516 while the current bare-noqa count is 397, leaving ~119 units of headroom that undercut the documented zero-tolerance freeze: the gate docstring claims any net-new bare noqa trips the gate, but with cap 516 a PR could add over a hundred first. Drop the baseline to the measured 397 so the freeze is exact (cap = 397 + slack 0), the same hard-zero-at-the-boundary shape LIT005 and LIT007 already use and pass in CI. PR table row updated to 397 / 0 * fix: increase slack * fix: increase slack * docs(lint): align gate docstring with buffered LIT003/LIT004 slack The budget now gives LIT003/LIT004 nonzero slack, so the gate's prose no longer claims they are frozen at slack 0; LIT005 remains the reasonless- suppression freeze and LIT007 the hard zero.	2026-06-16 16:59:21 -07:00
Mateo Wang	d0c2e87810	ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379) ... * ci: enable ruff preview rules under the budgeted strict gate Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only, leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at zero. Enumerate the 118 firing codes explicitly with explicit-preview-rules so the gate is deterministic and stable across ruff upgrades rather than depending on preview auto-selecting the broad catalog. Grandfather the existing 58438 violations into ruff-strict-budget.json as per-rule baselines with headroom, so only net-new violations fail CI. The existing ten rules keep their hand-tuned slack; the new rules get slack 10 when the baseline is 50 or more and 3 otherwise. * ci: add ANN return-type rules to the budgeted strict gate Add ANN201/202/204/205/206 (missing return annotations) to the strict lane and grandfather the existing counts into ruff-strict-budget.json so the codebase ratchets toward explicit return types without breaking CI. * ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines Add two type-check gates, each grandfathering the current tree so only net-new violations fail CI, matching the ruff strict-budget ratchet. mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI invocation actually reads; the root [tool.mypy] is not picked up from the litellm/ working dir). The 4885 existing missing-annotation errors are captured in litellm/.mypy-baseline.txt and the run is piped through mypy-baseline filter so new untyped defs are rejected. basedpyright runs in strict mode over litellm/, with enableTypeIgnoreComments disabled so it only honors '# pyright: ignore' and never polices mypy's '# type: ignore'. The existing strict diagnostics are grandfathered into .basedpyright/baseline.json. Both tools are pinned in the dev group and uv.lock; the lint workflow and Makefile run them filtered through their baselines, with lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet. * ci: raise lint job timeout to 15m for the basedpyright strict pass * ci: pin pythonVersion 3.12 and regenerate baselines against merged base Merge litellm_internal_staging so the baselines cover code the CI merge includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is reproducible across interpreter versions (CI runs 3.12). * ci: regenerate basedpyright baseline against the frozen lint env The previous baseline was generated with optional provider deps (azure, google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors not in the baseline. Regenerate after uv sync --frozen so the baseline reflects the same dependency set the lint job sees. * ci: regenerate basedpyright baseline on python 3.12 frozen env The prior baseline still carried proxy-dev packages (e.g. prisma) that the lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import errors ungrandfathered. Regenerate in a python 3.12 venv synced to the frozen lock with default groups only, so the baseline matches exactly what CI sees. * ci: replace type-check baselines with per-file count budgets The mypy and basedpyright baselines were position-sensitive (and the basedpyright one was a 27MB file), so ordinary line shifts churned them. Replace both with a per-file count gate: scripts/type_check_gate.py reduces each tool's output to errors-per-file and checks it against a committed {file: max} budget, ignoring line and column numbers. A file fails only when it gains more errors than its ceiling; debt can't be shuffled between files because each file has its own cap and new files default to zero. Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are generated in the python 3.12 frozen lint env so they match CI. Drops the mypy-baseline dependency; basedpyright runs without its native baseline. ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update. * ci: add a small per-file slack to the type-check gate Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count before failing, so a basedpyright inference ripple in an unrelated file doesn't break the build over a couple of errors. Budgets still record exact counts; the tolerance is applied at check time. * ci: move type-check slack into the budget json and trim lint timeout Make slack declarative: the budget is now {"slack": N, "files": {path: count}} so the tolerance is tuned in JSON without editing the script, mirroring how ruff-strict-budget.json carries its slack. --update preserves the existing slack. Also drop the lint job timeout from 15m to 10m; the mypy and basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a comfortable margin. * ci: collapse fully-adopted ruff categories and drop inert preview flag ANN (all nine non-removed rules) and BLE (its only rule) were spelled out code-by-code; replace each with its category selector, which is exactly equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category selector and error when named explicitly). explicit-preview-rules was inert: every selected rule is stable and nothing is selected by category, so the flag had nothing to gate. Verified the strict-rule counts are identical before and after (62379 each, zero per-rule drift), so no budget change. * ci: drop redundant pyright dev dependency Nothing invokes bare pyright in the Makefile, the linting workflow, or scripts; the basedpyright gate added on this branch is the only type checker that runs. basedpyright is a superset fork that reads the same pyrightconfig.json and honors the same "# pyright: ignore" comments, so pyright==1.1.408 in the ci group was dead weight. Regenerated uv.lock under the same exclude-newer cutoff so the only change is removing pyright and its package stanza * ci: un-weaken mypy and error on Any in basedpyright mypy: enable warn_return_any, drop the valid-type silencer, and stop globally ignoring missing first-party imports via [mypy-litellm.*] ignore_missing_imports = False, which surfaced eight real broken litellm.* imports the blanket ignore was hiding; third-party imports stay ignored. The per-file budget moves 4888 -> 5799 (902 no-any-return, 1 valid-type, 8 import-not-found), all grandfathered so only net-new errors fail and the ceilings ratchet down basedpyright: error on reportExplicitAny and reportAny. The per-file budget moves 117033 -> 148946 (6931 explicit-Any, 24954 Any-typed expressions), grandfathered the same way * ci: add Any-discipline gate on changed lines under litellm/ Add scripts/check_any_discipline.py, a type-aware gate that fails when a changed line holds a value typed Any -- including the X | Any unions that mypy --strict / basedpyright accept (e.g. re.Match.group() -> str | Any, json.loads() -> Any, bare dict -> dict[Any, Any]). It reuses the repo's mypyc-compiled mypy 1.19 via a custom generic AST walker (mypyc precludes subclassing TraverserVisitor), loads litellm/mypy.ini for parity with lint-mypy, and uses a dedicated incremental cache (.mypy_cache_any) with mtime+hash invalidation to force re-checks. Scope is changed-lines-only so editing a legacy file never forces cleaning its existing Any debt; suppress a genuine typed/untyped boundary with # any-ok: <reason> (ANY002 requires the reason). Wire it into the Makefile (lint-any, lint, lint-dev), a parallel any-discipline CI job with its own actions/cache, .gitignore, and the CLAUDE.md / CONTRIBUTING.md docs. * ci: move Any-gate codes into the shared LIT namespace Renumber the Any-discipline checker into the LIT*** scheme owned by scripts/check_type_discipline.py (PR #30500) so the two checkers share one rule namespace and suppression convention: ANY001 -> LIT002 (Any-typed value; LIT002 was the retired/free slot) ANY002 -> LIT005 (any-ok without a reason; the shared suppression-reason code) ANY000 -> LIT000 (setup/build/read error; the shared error code) Messages and behavior are unchanged; LIT005's text already matches the "<token> requires a reason" shape used for cast-ok/guard-ok. * ci: gate mypy and basedpyright per error rule, not per file Switch the mypy/basedpyright budget gate from per-file error counts to per-rule-code totals, mirroring the {rule: {baseline, slack}} shape of ruff-strict-budget.json. A rule fails when its codebase-wide error count exceeds baseline + slack, so violations are tracked by category rather than by file location. scripts/type_check_gate.py now parses mypy from its text output (trailing [code]) and basedpyright from --outputjson (the JSON `rule` field), since basedpyright's wrapped text diagnostics mis-attribute the rule on continuation lines. Replace the *-file-budget.json files with freshly captured *-code-budget.json baselines and update the Makefile, CI, and CLAUDE.md accordingly. * docs: prefer Pydantic validation over any-ok suppression Point the Any-discipline guidance at validating Any with Pydantic (a model or TypeAdapter that returns a typed value or raises) and frame # any-ok as a last resort that should ideally never be used. * chore: remove extraneous comment * chore: make the CLAUDE.md more concise * chore: clean up bloated CONTRIBUTING.md additions * chore: make Makefile more concise * ci: add the lint-budget-update target CLAUDE.md references CLAUDE.md tells contributors to run make lint-budget-update, but the target was never defined. Add it as an aggregate that re-captures the ruff, mypy, and basedpyright budgets in one shot. * ci: recapture mypy and basedpyright budgets in the lint env The per-rule baselines were captured in a richer dependency env than the CI lint job's uv sync --frozen, so CI resolved fewer types and reported more errors than the budgets allowed (no-any-return 902 over cap 900, plus several basedpyright reportUnknown* rules). Regenerate both in the frozen env so they grandfather the true CI debt: mypy 5786 -> 5799 (no-any-return 890 -> 902, valid-type 1 restored), basedpyright 146213 -> 148942. * ci: check out PR head sha in lint and any-discipline jobs The default pull_request checkout uses refs/pull/N/merge, which folds the latest base commits into HEAD. The diff-based gates (ruff delta, Any discipline) then diff against the event's older base.sha and blame base's own new commits on this branch; staging's otel-v2 and streaming changes (#30326, #30485) tripped the Any gate on files this branch never touched. Checking out the PR head sha makes the gates diff the real branch tip against base, and pins the tree the mypy/basedpyright budgets were captured against so their counts stay deterministic as the base advances. * ci(lint): renumber Any-typed-value rule LIT002 -> LIT009 Free up LIT002 for the sibling type-discipline gate (check_type_discipline.py, #30500), which groups its mutable-collection family at LIT001 (annotation) and LIT002 (construction). This gate's Any-typed-value rule moves to LIT009 so the shared LIT namespace stays contiguous with no holes; LIT000 and LIT005 are unchanged. * style: rename lint-strict-budget -> lint-ruff-budget * ci: harden type-check gates against silent passes (greptile review) type_check_gate.py: refuse to certify a vacuous run. The CI pipe swallows the tool's exit code ('tool || true'), so a crashed mypy/basedpyright that emits nothing would parse to zero errors, breach no ceiling, and pass. is_vacuous_run() now fails when nothing was parsed but the budget expects errors. Also wrap basedpyright's json.loads in a JSONDecodeError handler that prints the offending output instead of dumping a raw traceback. check_any_discipline.py: ALL_LINES was None, which dict.get() also returns for a path absent from the line map, so a path-normalisation mismatch could let a violation on an unchanged file pass the scope filter. Make ALL_LINES a distinct sentinel object so 'whole file' and 'path missing' are unambiguous. Adds tests for all three.	2026-06-16 12:07:46 -07:00
ryan-crabbe-berri	c90eb7e96f	feat: ruff strict-rule suppressions baseline gate (#30303) ... * feat: add ruff strict-rule suppressions baseline gate Introduce a stricter ruff rule set (typed params, no Any, complexity and arg-count caps, mutable-default and global-rebinding checks) grandfathered against the current tree and enforced as a budget rather than zero-tolerance ruff-strict.toml defines the 9 rules separately from ruff.toml so the existing ruff check stays green. scripts/ruff_suppressions.py builds the per-file, per-rule baseline in ruff-suppressions.json and gates CI by failing when the total grows past the baseline plus a 0.5% slack margin. The baseline ratchets down via `make lint-suppressions-update` after fixes * fix: surface per-file drift as a warning on a passing suppressions check Greptile flagged that cmd_check computed per-file regressions but only printed them on failure, so violations shifted between files (or a brand-new file under the slack) passed with a silent OK. Print them as a non-fatal warning on the pass path too; pass/fail behavior is unchanged * refactor: gate strict ruff rules on the delta vs base, not a frozen baseline The committed total-count baseline went stale against a moving base. CI lints the PR merged with the current staging tip, so violations merged by other PRs counted against this PR and tripped the budget even though nothing here touched them Replace it with a drift-proof gate. scripts/ruff_strict_gate.py runs ruff on the head, keeps only violations on lines this change adds relative to the merge-base, and fails when a rule exceeds its per-rule allowance in ruff-strict-budget.json (all 0 today). Because the base is measured live, base drift cancels out and only what the change introduces is gated. Drops ruff-suppressions.json and the old suppressions script * chore: allow 5 new ANN001/ANN003/ANN401 per change Give the three annotation-completeness rules a small per-change allowance so a large new module is not blocked over a few untyped params or kwargs, while the correctness and structural rules (B006, C901, PLR0913, PLW0603, RUF012, ANN002) stay at 0 * feat: add TID251 typing.Any/Dict import ban and widen annotation budgets Add TID251 (flake8-tidy-imports banned-api) to ruff-strict.toml, banning new imports of typing.Any and typing.Dict and steering new code toward structured types. It counts the import site, about one per file, so it is set non-blocking at 50 as a forward-looking signal Widen the annotation-completeness budgets so they nudge rather than block: ANN001 50, ANN401 50, ANN003 25. Correctness and structural rules stay at 0 * refactor: make the strict gate a drift-safe per-rule total ceiling Switch the gate from a per-change allowance to a hard ceiling on each rule's total count across the codebase. The ceiling is baseline + slack in ruff-strict-budget.json, with baseline captured from today's tree To stay drift-safe, the gate counts each rule on the head and on the merge-base (via a throwaway git worktree) and fails a rule only when its head total is over the ceiling and higher than the base, so base drift never blames a change that did not add to that rule. Annotation rules keep generous slack (ANN001 and ANN401 50, ANN003 25, TID251 50); structural and correctness rules are frozen at today's count. Add make lint-strict-budget-update to re-capture baselines * chore: give the structural strict rules a cushion of 3 To be liberal to start, B006, C901, PLR0913, PLW0603, RUF012, and ANN002 each get a slack of 3 instead of 0, so an occasional legitimate case is not hard-blocked. The annotation budgets are unchanged, and these ratchet down later * feat: ban more typing collection aliases and tighten annotation slack to 10 Add typing.List, typing.Set, typing.MutableSequence, and typing.MutableMapping to the TID251 banned-api list, steering new code toward tuple, Sequence, Mapping, frozenset, and frozen dataclasses. This raises TID251's baseline to 2404 Bring the three rules that were at slack 50 (ANN001, ANN401, TID251) down to 10 * docs: document the strict-gate ratchet and Any-avoidance in CLAUDE.md Add a line on running make lint-strict-budget-update to knock baselines down after fixes, and a line on validating untyped inputs in the caller rather than spending the Any budget * feat: make it a bit more strict --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 20:14:45 -07:00
Sameer Kankute	cfcdf8714a	feat: litellm oss 110626 (#30202) ... * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) (#29775) * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) Adds first-class support for the gpt-realtime-whisper streaming speech-to-text model, which uses the Realtime transcription session API rather than the file-based /audio/transcriptions path. Model registration: registers gpt-realtime-whisper and azure/gpt-realtime-whisper with audio-duration pricing (input_cost_per_second = 0.017/60, matching the published $0.017/minute input audio rate). REST endpoint: implements POST /v1/realtime/transcription_sessions (plus /realtime and /openai/v1 aliases) to mint an ephemeral transcription session for the WebRTC flow. Adds request/response types, OpenAI and Azure URL builders, a shared base handler (refactored from the client_secrets handler), the acreate_realtime_transcription_session SDK function, and route registration. The proxy encrypts the ephemeral key returned under client_secret.value and records the session type in the token so the follow-up /realtime/calls replays type=transcription rather than type=realtime. WebSocket: forwards intent=transcription through to the Azure handler (OpenAI already received it) with URL-encoding, so gpt-realtime-whisper opens a transcription session. Transcription-only sessions no longer trigger an erroneous response.create. Cost tracking: transcription sessions emit no response.done events; their usage arrives on conversation.item.input_audio_transcription.completed as {type: duration, seconds}. That usage is captured out-of-band (usage only, no transcript duplication) and billed by input_cost_per_second, with a token-billed fallback for token-priced transcription models. Adds tests for pricing math, URL builders, request/response types, the proxy route and SDK function, WebSocket intent forwarding, transcription-session streaming behavior, and the /realtime/calls session-type replay. * Address PR review: URL-encode all Azure WS query params; forward query_params through provider_config branch * Address PR review: session_type validation, model auth fix, cost perf, billing fallback, detail/docs cleanup * Improve test coverage: detection from backend, error paths, unknown usage type, resolved_model None * Backport realtime transcription websocket fixes * Enforce authorized realtime transcription model * Enforce realtime transcription model access * Enforce realtime resolved model scopes * Enforce WebRTC transcription model scope * Lazy evaluate debug log in pass-through endpoint (#30177) * Pass through debug lazy logging * fix(proxy): convert remaining eager pass-through debug logs to lazy formatting * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint (#30157) * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint The Parallel Search API moved from /v1beta/search (processor: base/pro, parallel-beta header) to /v1/search (mode: turbo/basic/advanced, no beta header). Request fields moved too: max_results, source_policy, and excerpt settings are now nested under advanced_settings, and source_policy uses include_domains/exclude_domains. The v1 response returns publish_date per result, which now maps to SearchResult.date instead of being hardcoded to None. The legacy processor param is mapped to the equivalent mode so existing callers keep working. * fix(parallel_ai): default mode to basic and simplify param handling The v1 API defaults to advanced mode when mode is omitted, while v1beta defaulted to the base processor. Without an explicit default, callers who pass no mode would be silently upgraded to a tier costing 2.25x more while litellm's cost map reports the basic-tier price. Sending mode=basic preserves the v1beta default and keeps cost tracking accurate. Also replaces the handled_params set with pop-as-consumed param handling so mapped params no longer need to be tracked in two places, and extends the tests to pin the default mode, processor=base mapping, mode-over-processor precedence, and top-level v1 param passthrough. * fix(parallel_ai): avoid double /v1 when api_base is already versioned A PARALLEL_AI_API_BASE like https://api.parallel.ai/v1 previously produced .../v1/v1/search. Strip a trailing /v1 before appending the search path and cover the api_base variants with a parametrized test. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(focus): add Mavvrik destination for FOCUS export (#29935) * fix: preserve responses streaming flag (#30189) * fix: preserve responses streaming flag * test: cover async responses streaming flag * fix(spend/daily-activity): stable offset pagination via id tiebreaker (#30164) (#30167) date alone is not a unique sort key for LiteLLM_DailyUserSpend or LiteLLM_DailyTeamSpend (many rows per date: api_key x model x model_group x provider x endpoint). Offset pagination over a non-unique sort landed on arbitrary boundaries, so a client paging through all results and summing per-page metrics (the Usage dashboard) got non-deterministic totals - sometimes inflated, sometimes deflated, different at different page_size values. Adding the row's UUID id (present on both tables) as a secondary sort gives every page a stable cursor. order=[{date desc}, {id asc}]. Fixes #30164 * fix(oci): inject a default maxTokens so omitted max_tokens doesn't truncate responses (#30018) * fix(oci): inject default maxTokens so omitted max_tokens doesn't truncate OCI GenAI applies a tiny server-side maxTokens default (~20 tokens) when the request omits it, so any call that doesn't send max_tokens comes back cut off mid-string with finishReason "length". MLflow judges never send max_tokens, so their JSON responses arrived as unterminated strings and json.loads failed in MLflow's gateway adapter. When no maxTokens/maxCompletionTokens target is set, inject DEFAULT_OCI_CHAT_MAX_TOKENS (env-overridable, defaults 4096), mirroring the Anthropic config's default-max-tokens behaviour. An explicit max_tokens still wins, and reasoning models still route to maxCompletionTokens. Used a fixed default rather than the catalog max_output_tokens because the catalog value is unreliable for some models (grok-4 reports max_output_tokens equal to its context window, not a real output cap, which would risk 400s). Adds TestOCIDefaultMaxTokens covering Cohere and generic injection, the explicit-override case, and the reasoning maxCompletionTokens branch. * test(oci): e2e regression that omitted max_tokens isn't truncated Real-proxy integration test asserting a chat completion that omits max_tokens completes with finish_reason "stop" instead of being cut off at OCI's ~20-token server default. Fails before the maxTokens-default injection (finish_reason "length", ~19 tokens), passes after. * test(oci): update cohere default-params test for injected maxTokens test_cohere_default_parameters asserted no maxTokens was injected, encoding the old behaviour where OCI's ~20-token server default truncated responses. Now that transform_request injects DEFAULT_OCI_CHAT_MAX_TOKENS, assert maxTokens equals that default while the other params (topK/topP/frequencyPenalty) stay pass-through with no hardcoded default. * fix(oci): make DEFAULT_OCI_CHAT_MAX_TOKENS a plain constant Drop the os.getenv override. The env knob was not requested and introducing a new env var forced a cross-repo dependency on litellm-docs (test_env_keys.py validates every referenced env var against the docs table there). A plain 4096 constant keeps the PR self-contained; callers who want a different limit pass max_tokens explicitly per request. * fix(oci): route all OpenAI commercial models to maxCompletionTokens OCI serves OpenAI models (gpt-4.1, gpt-5.1 through 5.5, o-series) that the litellm catalog doesn't track, so the supports_reasoning lookup returned False for them and the provider sent maxTokens, which the reasoning families reject with HTTP 400. With the injected default maxTokens this broke every request to those models, not just ones with an explicit max_tokens. Route the whole openai.* vendor prefix to maxCompletionTokens since OpenAI accepts max_completion_tokens on every chat model; the openai.gpt-oss-* open weights are served by OCI's own stack and keep maxTokens. Verified live against gpt-5.2, gpt-5, gpt-4o, gpt-4.1, gpt-oss-120b, llama-3.3, command-a and grok-3-mini * test(oci): hoist transformation imports and drop unused ones Makes the generic-chat test file ruff-clean: the per-test local imports of OCIChatConfig/OCIVendors shadowed the module-level import (F811) and left it unused (F401), and json plus three OCI type imports were never referenced * fix(oci): translate response_format json_schema to OCI's accepted shape (#29691) * fix(oci): translate response_format json_schema to OCI's accepted shape OCI GenAI rejected every json_schema response_format with HTTP 400 "Please pass in correct format of request", which broke structured-output callers such as MLflow LLM judges (they always send a json_schema). The provider forwarded OpenAI's raw json_schema body unchanged. For GENERIC models OCI's ResponseJsonSchema accepts only name/description/schema/isStrict, so OpenAI's `strict` key (and any other extra) 400s the request; the key must be renamed to isStrict and the body whitelisted. For Cohere models there is no JSON_SCHEMA type at all; the schema has to ride on JSON_OBJECT as {"type": "JSON_OBJECT", "schema": ...}. Cohere type values must also be the canonical uppercase TEXT/JSON_OBJECT. _normalize_response_format now branches by vendor and emits the exact shape each one accepts (verified live against OCI GenAI for Cohere, Meta, Gemini and Grok). Drops the unused, incorrect Cohere response-format pydantic models. Two existing tests asserted the broken behavior (lowercase type, raw jsonSchema on Cohere); they are rewritten to assert the corrected shape, and generic/Cohere json_schema regression tests are added. * fix(oci): raise early on json_schema response_format with no body A GENERIC model request with {"type": "json_schema"} and no json_schema object fell through to the JSON_OBJECT branch and emitted a bodyless {"type": "JSON_SCHEMA"}, which OCI rejects with an opaque HTTP 400. Raise a descriptive 400 at translation time instead. Cohere is unaffected since it always maps to JSON_OBJECT. * test(oci): gateway integration test for response_format json_schema Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): accept default n=1 on Cohere instead of hard-failing (#29705) * fix(oci): accept default n=1 on Cohere instead of hard-failing Cohere on OCI has no numGenerations field, so n was mapped to False and map_openai_params raised "param `n` is not supported on OCI" whenever a client sent n. But n=1 (and None) is the OpenAI default single-generation request, which every OCI model produces anyway, so standard clients that always send n=1 (such as the MLflow gateway) were rejected with a 500. Drop n=1/None silently for Cohere; only n>1 is genuinely unsupported and still raises (or drops under drop_params). Generic models are unaffected and keep numGenerations, including n>1. * docs(oci): explain why n is not advertised for Cohere despite tolerating n=1 * test(oci): gateway integration test for Cohere default n=1 Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): drop max_retries instead of hard-failing on OCI (#29727) max_retries is a litellm-level control param (litellm applies retries itself), not a generation param OCI accepts. The provider mapped it to False and raised "param `max_retries` is not supported on OCI" whenever it was present. The litellm proxy injects max_retries on every request, so any OCI call through the proxy 500'd unless drop_params was set. Drop max_retries silently in map_openai_params. Adds a unit test (Cohere and generic) and a gateway integration test that a plain request succeeds through a proxy without drop_params. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): rehydrate metadata JSONB text on ui_view_spend_logs (#29682) Fixes #29674. `/spend/logs/ui` raw-SQL path returns the JSONB metadata column as a string — prisma's query_raw skips the ORM-layer hydration. The UI reads metadata.status / metadata.error_information as object fields, so provider-failure rows look like successes. Fix: json.loads the metadata field right after query_raw, fall back to {} on malformed JSON. 3 existing error-code/error-message tests called json.loads on response.data[0]["metadata"] — they were leaning on the bug. Updated to read the dict directly. Plus 2 new regression tests (failure metadata roundtrip + invalid-json fallback). Reverting the fix makes both new tests fail with AssertionError: metadata should be dict, got <class 'str'>. * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) (#30020) * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) * fix: refund max_parallel_requests on disconnect from outer streaming generators The cancellation refund previously lived in async_post_call_streaming_iterator_hook, but that hook is nested inside the outer streaming generators and a nested async generator only receives GeneratorExit on garbage collection (non-deterministic). With only the v3 limiter enabled, /chat/completions also bypasses the hook entirely (needs_iterator_wrap() is false). Move the release into async_data_generator and async_streaming_data_generator, the generators Starlette closes on client disconnect, so the refund fires deterministically on every streaming route. Warn when no event loop is running, and document the window TTL refresh on the decrement * fix(mcp): propagate model into model_call_details for passthrough tool calls (#30122) * fix(mcp): propagate model into model_call_details for passthrough tool calls The @client decorator on call_mcp_tool creates the logging object via function_setup without a model kwarg, so model_call_details["model"] starts as None. execute_mcp_tool only set logging_obj.model as an instance attribute, which the spend-log writer never reads (it reads kwargs["model"] from model_call_details). MCP passthrough tools/call rows therefore persisted with model="" while list_tools rows showed "MCP: list_tools", degrading the Logs UI display and bucketing all MCP tool spend under an empty model in DailyUserSpend. Propagate the model into model_call_details alongside the existing attribute assignment so the StandardLoggingPayload and SpendLogs writer pick it up. Covers the /mcp passthrough, REST /mcp-rest/tools/call, and orchestrated paths (the latter already passed model into function_setup, so this is a no-op there). * test(mcp): trim regression test docstring * fix(mcp): surface upstream challenges for delegated OAuth (#30124) * fix(mcp): surface upstream challenges for delegated OAuth * docs(mcp): clarify delegated upstream auth comments * perf(benchmarks): add CPU timing metrics to streaming benchmark (#29980) * Add CPU timing metrics to streaming benchmark * Fix spacing around timing sample dataclass * fix(gemini): don't emit empty choices on metadata-only stream chunks (#29167) web_search + reasoning makes Gemini stream mid-chunks that carry only grounding/thought metadata — no content part, no finishReason. _process_candidates skips content-less candidates and the existing fallback only ran when finishReason was set, so choices stayed empty and the downstream streaming handler raised IndexError on choices[0]. Emit an empty-delta choice for content-less chunks regardless of finishReason. Fixes #28884 * fix(key): allow /key/update to clear budget_limits with [] or null (#30085) * Fix /key/update rejecting budget_limits clear requests with HTTP 400 Sending budget_limits: [] or null to /key/update returned HTTP 400, so once a key had budget windows the last one could never be removed. prepare_key_update_data only json.dumps'd budget_limits when the value was truthy, so [] and None passed through raw to the Prisma Json? column; jsonify_object only serializes dicts, and prisma-client-py has no DbNull sentinel for Json? writes, so Prisma rejected both shapes. Serialize the clear case explicitly as the JSON literal null, matching how memory_endpoints encodes metadata for the same column type. Truthy values keep the existing reset_at window initialization path. Fixes #30067. * Require admin access for budget_limits changes on /key/update Clearing budget_limits via [] or null is a budget mutation, but _validate_update_key_data only counted max_budget and spend as budget changes before deciding whether to skip _check_key_admin_access. A non-admin key owner or a team member with /key/update could therefore remove a key's per-window spend caps without admin authorization. Treat any explicit budget_limits value in the request (set, change, or clear) as a budget change so it gates through the same admin check as max_budget. model_fields_set is used because an explicit null is indistinguishable from an omitted field by value alone. * fix(proxy): persist guardrail info in spend logs for /v1/responses (#30092) Pre-call guardrail blocks on /v1/responses wrote guardrail_information as null in LiteLLM_SpendLogs because _handle_logging_proxy_only_error splits request_data by LoggedLiteLLMParams keys and litellm_metadata, where the Responses API stores request metadata including standard_logging_guardrail_information, was not among them. It fell into optional_params, so merge_litellm_metadata never saw it. Add litellm_metadata to LoggedLiteLLMParams so it routes into litellm_params the same way metadata does on the chat completions path Fixes #28971. * fix(proxy): handle non-standard SSE frames in Anthropic passthrough logging (#26000) Some third-party Anthropic-compatible providers emit non-standard SSE frames (OpenAI-style [DONE] sentinels, non-JSON keep-alive lines) in streaming responses. These caused json.JSONDecodeError in _build_complete_streaming_response, breaking the passthrough logging pipeline so the request was never logged or billed. Skip whole-line 'data: [DONE]' sentinels and catch JSONDecodeError per event. Matching the full line (not a substring) keeps a valid chunk whose text payload contains '[DONE]' from being dropped. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(newrelic): Add New Relic extension (#26989) * initial New Relic integration. * Minor fixes for basic observability. * Implemented basic support for the success path. Generates New Relic custom events needed by the AI Monitorin interface. * Supportability metric is sent on first request. * Emit supportability metric every hour instead of once a day. * Add the start/end times to the messages before sending them so that the start time and end time reflect the correct time and both are not set to 'now'. * Make use of `turn_off_message_logging` configuration that is available by default from CustomLogger. * Enabling New Relic agent to be wired when docker container starts if an environment variable is set. * If we cannot find trace information, send the AI events without the trace ID attached. * Use a fake trace_id if we cannot find one. * Implementing a configuration so that users can use litellm configuration to disable sending LLM messages to New Relic. There is a second method to do this via New Relic env var. * Mised file. * Cleaning up logic to turn off recording content via either the LiteLLM configuration or an env var. * Removing debugging. Fixed logic / comments around how often to send supportability metric. * Initial version of public doc for New Relic. * Use a proper name for the doc file. * Updating newrelic.md document. * Updating LiteLLM documentation for New Relic extension. * Moving New Relic imports into the methods to support unit tests. * Adding unit tests for the New Relic extension. * Updating linting and the unit tests that are not running in the CI environment. * Address reviewer feedback on New Relic integration. - Fix _record_error_metric to use app.record_custom_metric() instead of module-level newrelic.agent.record_custom_metric() so the call works outside of an active transaction context - Remove unreachable except ImportError block in _get_trace_context - Update stale "23 hours" comment to "27 hours" (matches 97200s threshold) - Remove commented-out debug code from _process_success - Fix docs typo: NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STOREDA -> NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STORED - Update TestRecordErrorMetric to verify app.record_custom_metric call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Reformating for the linter. * Addressing additional automated feedback. - Removed a legacy comment about the New Relic header - Reordered imports in one file - Switched another file to use the import at the top of the file instead of inline when used - Added unit tests for untested methods that were identified * Addressing new feedback. - Proper handling of time to floats. Created a util method and updated code to use it. - added the missing guard to ensure the app is enabled * Addressing feedback. - When an error occurs, still check if the periodic supportability metric should be emitted - Added a check to ensure the extension is ready in the error handler to match _process_success * Updating the NR event timestamps to more accurately reflect when the messages were generated. * Addressing feedback for potential better practice. * Addressing feedback on accessing default values. Added tests for most of these cases. * Adding a new catch exception block based on feedback. * Addressing feedback about a potential issue around a timestamp for the supportability metric. * Addressing minor feedback on length of generated, fallback traceId. * Addressing feedback. - A few more cases were found where the dictionary access might not return the correct value. - Handling cases where `traceparent` is not lower cased * Addressed feedback where the newrelic options might not apply correctly. * Addressing some feedback. * Addressing feedback. * Validating testing / formatting for our changes. * Updating linting, adding tests, defining data type for UI. * Configuration for the logging callback definition. * Adding a newrelic image for the UI to use. * Putting the New Relic callback in proper alphabetic order. * Copying the logo to a committed output directory so it shows up in a locally built container. * Adding missing definition of new env vars that were causing a build failure. * Addressing automated feedback from greptile. * Adding a few more unit tests to increase the code coverage just a bit more. * Additional unit tests to push coverage to almost 90%. * Adding a custom newrelic docker image build process. This removes the need to add the newrelic agent to the core litellm container or dependencies. * Clarifying message when the New Relic agent is not installed and someone is trying to use the newrelic extension. Either use the proper image when using docker, or install the agent manually when running from source. * Ensuring pip is available to install the New Relic agent. * Updating the definition and handling of traceId (no spanId). Clarifying behavior of env vars vs UI configuration for the newrelic extension. * Removing entries from the New Relic logger configuraiton UI as these values must be set as part of running the image. * Removing a stale doc file that has moved to the litellm-docs repo. Cleanup of Dockerfile to remove a LABEL that was incorrect. * Updating container image name to be the best guess for the new name. * Addressing feedback from greptile. - Added a comment around token_count=0 - Updated the boolean parser to allow a wider set of options which matches existing patterns in other parts of LiteLLM. * Removing option for a separate New Relic container image. The agreement is to handle this in the New Relic integration docs. * Updating error message when New Relic agent is not available. * Wiring in the test message from the LiteLLM callback UX. * Missed saving one of the file conflicts. * Fixed a lint error I introduced. Somehow, I dropped another string and now added it back. * Adding newrelic to the schema definition. * Added an admin check on the call before sending test message as mentioned by the AI code review. * Updating to use should_redact_message_logging(kwargs) as part of the logic to determine if message content should be sent to New Relic or not. This still uses the `record_content` property as well, but both have to be true in order for content to be included. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add Azure AI Foundry DeepSeek V3.1 and V4 Pro/Flash global pricing to cost map (#30134) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(logging): translate Responses bridge result to ModelResponse for spend logs (#28985) PR #29394 fixed the AnthropicResponse.model_validate crash for the streaming anthropic_messages -> OpenAI Responses bridge by unwrapping terminal events and returning the inner ResponsesAPIResponse. The spend_logs row lands and usage/cost are correct, but the row's response field stores the Responses API shape (output[...].content[...].text). The proxy UI Logs tab reads response.choices[0].message via parseMessages in prettyMessagesUtils.ts with no fallback for the Responses shape, so the OutputCard renders "No response data available" for every cross-routed call. The same shape mismatch affects every downstream consumer of spend_logs that assumes the canonical chat-completion shape This change keeps the unwrap from #29394 but routes the resulting ResponsesAPIResponse (and the bare-response non-streaming path) through LiteLLMResponsesTransformationHandler.transform_response, which is the same conversion already used by the chat-completion Responses bridge. Spend_logs now stores a ModelResponse with choices[0].message.content, so the UI and other consumers see the assistant text. On a translation failure (eg. empty output on an incomplete response) the handler falls back to a minimal ModelResponse carrying model and usage so the row still lands rather than being dropped as a Non-Blocking error Also corrects a stale comment in the Responses adapter that implied the call type was reclassified to acompletion; the code preserves anthropic_messages and the success handler translates back to ModelResponse for the row Fixes #28595 * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions (#30024) * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions The `/v1/messages` -> `/v1/chat/completions` streaming adapter (`AnthropicStreamWrapper`) silently dropped the first non-empty delta of every content block that started via a *transition* (e.g. text -> tool_use -> text, text -> thinking). When an upstream chunk both triggers a new content block (its type differs from the active block) and carries that block's first delta, the wrapper emitted `content_block_stop` -> `content_block_start` and then only re-queued the trigger chunk when it was an `input_json_delta` (bundled tool args). The synthesized `content_block_start` always carries an empty body, so the first `text_delta` / `thinking_delta` was lost — the client output started from the second token (e.g. "Hi, how can I help you?" rendered as ", how can I help you?", or text resuming after a tool call lost its first sentence). This is especially visible with Claude Code-style clients that consume Anthropic Messages streaming events strictly. Fix: re-queue the trigger chunk's translated delta whenever it carries non-empty content (text/thinking/signature/tool args), via a shared `_trigger_delta_has_content` helper used by both the sync and async paths. Empty trigger deltas are still suppressed so no spurious empty `content_block_delta` is introduced. Fixes #30014 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(anthropic-adapter): cover all _trigger_delta_has_content branches Add a direct parametrized unit test for the re-emit predicate so every delta type (text/input_json/thinking/signature), the empty-payload guards, and the malformed/non-delta cases are exercised independently of upstream chunk translation. Raises patch coverage for the new helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat: add opt-in healthy_only filter to GET /v1/models (#30130) * feat: add opt-in healthy_only filter to GET /v1/models Adds an opt-in `healthy_only=true` query parameter to GET /v1/models and GET /models that hides models whose backing deployments are all marked unhealthy by background health checks. - Add Router.async_get_fully_unhealthy_model_names(), mirroring the semantics of get_fully_blocked_model_names(): a model is hidden only when every backing deployment is unhealthy and the health state is not stale (fail open otherwise). - Reuses the existing DeploymentHealthCache populated by _run_background_health_check(), so no new health state is introduced. - No-op when allowed_fails_policy is set, mirroring _async_filter_health_check_unhealthy_deployments semantics. - team_public_model_name aliases are aggregated alongside model_name. - Hiding is presentation-only; default behavior is unchanged. Fixes #30128 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: address Greptile review notes - Note team-alias asymmetry vs get_fully_blocked_model_names - Debug-log when healthy_only is set but no health state is available Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Dedupe team soft budget alerts by team_id instead of token (#30097) _team_soft_budget_check sends type="soft_budget" alerts with event_group=TEAM, but SoftBudgetAlert.get_id always returned the request token. The alert cache key was therefore scoped per virtual key, so every active key in a team over its soft budget fired its own alert within budget_alert_ttl. Branch on event_group so team-level alerts dedupe by team_id, matching TeamBudgetAlert, while key and project level alerts keep per-token dedupe. Fixes #27398. * feat(bedrock guardrails): support contextual grounding qualifiers (request-side) (#30057) * test: add failing tests for Bedrock contextual grounding (request-side) Drive the request-side of Bedrock contextual grounding: callers tag message content blocks as grounding_source/query, the post_call hook assembles an ApplyGuardrail(OUTPUT) call carrying source + query + response(guard_content), and the bedrock converse transform must render the tags as prompt text instead of silently dropping them. Non-grounding payloads must stay byte-identical. * feat(bedrock guardrails): support contextual grounding qualifiers Bedrock contextual grounding scores a model response against a reference source and the user query, expressed via a per-content-block `qualifiers` array on ApplyGuardrail. The guardrail hook previously sent plain text only, so grounding could not be driven through it even though the response-side contextualGroundingPolicy parsing already existed. Callers now tag message content blocks `{"type":"grounding_source"}` / `{"type":"query"}` (mirroring the existing `guarded_text` marker). On the generate path the bedrock converse transform renders them as plain text; at post_call the hook harvests them from the request and assembles one ApplyGuardrail(OUTPUT) call carrying grounding_source + query + the response (as guard_content). Requests without these tags produce a byte-identical payload, so existing behaviour is unchanged. * Feat(guardrail): Adding support for custom Ovalix guardrail (#21887) * Feat(guardrail): Adding support for custom Ovalix guardrail * Internal CR comments fixes * greptileai comments fixes * fix conflict * fixes * fix sha256 * clarify Ovalix actor-id hash is for normalization, not PII protection * fix(github_copilot): normalize per-event item_id in /responses streaming (#30072) GitHub Copilot's native /v1/responses stream assigns a different item_id to every event of a single output item (output_item.added, the part.added / delta / done events, and output_item.done). Spec-strict clients like the Vercel AI SDK key streaming parts by item_id and abort with "reasoning part <id> not found" / "text part <id> not found" when a delta references an unregistered id. Override transform_streaming_response in GithubCopilotResponsesAPIConfig to anchor every event of an output item to the id from its output_item.added. Copilot accepts that id paired with the final encrypted_content on the next turn, so multi-turn replay is unaffected. Fixes #30071 * feat: add /model/block and /model/unblock endpoints (#30125) * feat: add /model/block and /model/unblock endpoints Add dedicated proxy-admin POST /model/block and /model/unblock endpoints over the existing blocked flag on LiteLLM_ProxyModelTable, mirroring the /key/block and /key/unblock pattern. Calling a model whose deployments are all blocked now returns a clear 403 "Model is blocked" instead of a generic no-deployment error, including direct-dispatch route types (e.g. eval) via a pre-route guard. Includes audit-log entries for block/unblock and unit tests. Closes #29742 Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * chore: regenerate dashboard API types for model block/unblock endpoints Regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts from the proxy OpenAPI spec (npm run gen:api) so it includes the new endpoints. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: widen router block-helper param type and add direct unit tests Type the _are_all_deployments_blocked deployments parameter to match its callers (DeploymentTypedDict) so mypy passes, and add tests/test_litellm/test_router_block_helpers.py with direct unit tests for the three block helper methods so router_code_coverage recognizes them. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: restore type-ignore on messages arg after black reflow Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * refactor: raise model-block 403 in proxy layer, not SDK Router Keep the SDK Router's documented behavior for blocked deployments (filtered -> "no healthy deployment") and move the 403 PermissionDeniedError into the proxy layer (route_llm_request), where model blocking is an admin concept. This avoids a backwards-incompatible 403 for SDK users who set blocked=True on their own deployments, per maintainer review. Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: add week unit support to get_next_standardized_reset_time (#30100) * fix: add week unit support to get_next_standardized_reset_time The function handled d/h/m/s/mo units but silently fell through to the default next-midnight branch for the w (week) unit. This was inconsistent: _extract_from_regex already accepted w in its character class, and duration_in_seconds already returned value * 604800 for it. Add the missing elif unit == 'w' branch that delegates to _handle_day_reset with value * 7, which reuses the existing Monday- alignment logic for 1w and the generic N-day-from-midnight path for larger multiples. Add test_week_based_resets covering 1w from a Wednesday (expects next Monday) and 2w from a Monday (expects 14 days forward at midnight). Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * test: exercise relative week semantics with non-Monday base dates + add docstring Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * fix: black formatting and remove undocumented MAVVRIK_FOCUS_FREQUENCY env var * fix: black formatting with correct version and sync schema.d.ts for healthy_only param * fix: resolve mypy errors and add transcription_sessions to JSON schema endpoint enum * fix: restore MAVVRIK_FOCUS_FREQUENCY guard and exclude it from docs key scan * fix: address Greptile P2 comments - move constant, use UTC datetime, skip redundant team lookup * revert: restore original team lookup logic in can_key_call_resolved_model --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: nina-hu <nina.huuu@gmail.com> Co-authored-by: Sahith Jagarlamudi <104647530+s-jag@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com> Co-authored-by: alex107ivanov <30668368+alex107ivanov@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com> Co-authored-by: Teo Xian Zhong Augustine <35527068+auggie246@users.noreply.github.com> Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: Saksham Maggo <122939011+SakshamMaggo@users.noreply.github.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Kelvin <leikaiwei@outlook.com> Co-authored-by: Josh Bonczkowski <josh.bonczkowski@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: M. Dennis Turp <mdturp@pm.me> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Piotr Minkina <piotrminkina@users.noreply.github.com> Co-authored-by: Martín Alcalá Rubí <martin@tryolabs.com> Co-authored-by: T. Kobayashi <13004314+nix-tkobayashi@users.noreply.github.com> Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: Shalom <shalom@ovalix.io> Co-authored-by: codgician <15964984+codgician@users.noreply.github.com> Co-authored-by: FugoP <kim@pomsora.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-11 22:30:26 -07:00
Yassin Kortam	012d9f6c0a	feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker (#30211)	2026-06-11 10:34:26 -07:00
ryan-crabbe-berri	0d120de785	chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) ... * chore(hooks): enforce Conventional Commits and Conventional Branches Adds opt-in local git hooks plus a CI PR-title check: - .githooks/commit-msg validates commit subjects against Conventional Commits 1.0.0 (feat|fix|docs|style|refactor|perf|test|build|ci| chore|revert)(scope)!: subject. Merge/revert/fixup!/squash!/amend! messages pass through; --no-verify still works. - .githooks/pre-push validates branch names against Conventional Branches (feature|bugfix|hotfix|release|chore)/desc. Bypasses main, litellm_internal_staging, dependabot/*, gh-readonly-queue/*. Tag pushes and deletions are skipped. - scripts/install_git_hooks.sh sets core.hooksPath=.githooks and is wired up as 'make install-hooks'. Opt-in — not chained into install-dev. - .github/workflows/conventional-commits.yml validates PR titles via amannn/action-semantic-pull-request pinned to v6.1.1's SHA. This is the actual gate since squash-merge uses the PR title as the commit subject. - tests/test_litellm/test_git_hooks.py exercises both hooks via subprocess for accept / reject / bypass / git-generated-message cases. - CONTRIBUTING.md documents the conventions, the install step, the bypass list, and the --no-verify escape hatch. Resolves LIT-3306 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(hooks): address Greptile review on PR #28703 Resolves two findings from the automated code review: 1. CONTRIBUTING.md: shrink the new Conventional Commits / Branches section to a 2-line pointer at docs.litellm.ai. Per the team convention, the full documentation lives in the litellm-docs repo — see BerriAI/litellm-docs#208 for the companion change that adds the section to docs/extras/contributing_code.md. 2. .githooks/commit-msg: tighten the subject regex to also reject an uppercase first letter in the description. CI's subjectPattern is ^(?![A-Z]).+$ so the previous local hook would accept 'feat: Add thing' which would then fail the PR-title check. The local hook is now the strictly tighter of the two gates. Test cases extended to cover both the new rejection and the digit/symbol-start cases that remain allowed. Resolves LIT-3306 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: trigger ci after branch rename * fix(ci): rerun pr title check when bypass label changes amannn/action-semantic-pull-request only honors ignoreLabels if the workflow retriggers on labeled/unlabeled events; without them a red check stays red after a maintainer applies the bypass label. Also point the CONTRIBUTING.md workflow comments at the conventions section, which now sits above the Development Workflow section. --------- Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MBP.localdomain> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-11 10:00:23 -07:00
Mateo Wang	20e453f698	feat(cli): per-agent lite claude / codex / opencode commands that wrap coding agents through the proxy (#29850) ... * feat(cli): add `litellm-proxy run -- <agent>` to wrap coding agents through the proxy Wraps Claude Code, Codex, OpenCode, and any other coding agent so all of its LLM traffic routes through a LiteLLM proxy, with the agent-vault style of "just works" DX: one `run -- <agent>` command, auto SSO login when interactive, env-key "agent mode" for containers/CI, and a fail-fast key check against the proxy so bad credentials error immediately instead of deep inside the agent. The wrapped binary is detected by name to pick the right variables. Claude Code gets ANTHROPIC_BASE_URL (the bare proxy root, so it appends /v1/messages) and ANTHROPIC_AUTH_TOKEN, with any stray ANTHROPIC_API_KEY cleared so the proxy token wins. Codex and OpenCode get OPENAI_BASE_URL (proxy + /v1) and OPENAI_API_KEY. Unrecognized commands get both sets so they work either way. `litellm-proxy claude-code` remains as a shortcut for `run -- claude`. The core logic is split into dependency-injected helpers (agent_profile, build_agent_env, verify_proxy_key, run_agent) so env wiring, the preflight, and the launch handoff are unit-tested without monkeypatching, alongside CliRunner tests for auth resolution, agent mode, and auto-login. Mutation-tested the env profiles, preflight, and agent-mode branch to confirm the tests fail when the behavior is broken. https://claude.ai/code/session_0154VpLXW7mMvk5wfbgPRJa6 * Make each coding agent its own litellm-proxy command Replace the `run -- <agent>` interface and the `claude-code` shortcut with top-level commands generated per known agent, so launching is just `litellm-proxy claude`, `litellm-proxy codex`, or `litellm-proxy opencode`, with everything after the agent name forwarded straight to it. This drops the ceremony of `run --` and cuts typing. The `--model`/`--small-fast-model` wrapper flags are gone; pass the agent's own model flag instead, or export the model env vars (the wrapper preserves what you already have set), which keeps the surface minimal and avoids intercepting flags the agent owns. Rename the module to agents.py to match. * fix(cli): route `litellm-proxy codex` through the proxy via a custom provider Codex ignores OPENAI_BASE_URL (it always dials api.openai.com over the Responses WebSocket transport), so the OpenAI env profile alone left `litellm-proxy codex` talking to OpenAI directly instead of the proxy. Point Codex at the proxy with a custom provider passed as `-c` config overrides, and force the HTTP/SSE Responses transport with supports_websockets=false since the proxy does not speak the Responses WebSocket protocol. The provider reads its key from OPENAI_API_KEY, which the agent env already exports. The overrides are injected ahead of the user's args so they precede Codex's subcommand. Claude Code and OpenCode are unaffected; they honor the exported env vars. Adds regression tests for the per-agent launch args and the injection ordering. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Rename litellm-proxy CLI command to lite The proxy management CLI was invoked as litellm-proxy, which is a lot to type for an everyday command. Rename the console script entry point to lite and update the in-CLI usage examples, help text, error messages and docs to match. * fix(sso): stop CLI auth success page from hanging on "Closing..." The CLI opens the SSO success page with webbrowser.open, so the tab is not script-opened and the browser refuses window.close(). The countdown would end on "Closing..." and the tab would sit there forever. Drop the countdown and just show "You can now close this window and return to your terminal." from the start, while still attempting window.close() once so the tab auto-closes in the rare case the browser allows it. Add a regression test asserting the manual-close instruction is always present and the misleading countdown/"Closing..." text is gone. * fix(cli): reattach controlling terminal after SSO login, keep litellm-proxy alias When the first `lite claude` has to log in via browser SSO, completing the login could leave stdin detached from the terminal, so a TUI agent like Claude Code would start in non-interactive mode and exit with "Input must be provided". The wrapper now reopens the controlling terminal onto stdin just before handoff when the session started interactively; piped or redirected input is detected up front and left alone, so agent-mode and non-interactive use are unchanged. Also keep the `litellm-proxy` console script as an alias for `lite` so existing scripts and CI that invoke `litellm-proxy` keep working; both names map to the same CLI. * feat(install): make the curl installer need only curl, not a pre-existing Python The installer now lets uv provision a managed Python 3.13 when no suitable interpreter is found, instead of aborting. The minimum is also bumped from 3.9 to 3.10 to match the package's requires-python (>=3.10), so a system Python 3.9 is no longer selected only for uv tool install to reject it. * feat(cli): add thin litellm[cli] install path (install-cli.sh + brew) for the lite CLI On a developer laptop the `lite` CLI only needs `lite login` and running coding agents through a proxy, but the sole install path was `litellm[proxy]`, which drags in the whole server tree (fastapi, uvicorn, boto3, polars, cryptography, litellm-enterprise). The CLI's heavy imports are all guarded, so it runs on the base SDK plus just rich, pyyaml and requests. Add a `cli` extra carrying exactly those three, a `scripts/install-cli.sh` curl one-liner that installs `litellm[cli]`, and a `BerriAI/homebrew-litellm` tap formula with a release runbook under `packaging/homebrew/`. The installer passes no `--python`, so uv honours litellm's requires-python and provisions a managed interpreter, skipping a too-old (3.9) or too-new (3.14+) system Python instead of failing to resolve. A pyproject thin-contract test asserts the `cli` extra keeps the deps the CLI imports and never leaks a server-only dependency from `proxy`, so the laptop install cannot silently re-bloat * fix(install): let uv pick the Python via --python-preference system Both installers detected a system Python with a floor-only check and forced it with `uv tool install --python <interp>`. On a host whose only Python is outside litellm's requires-python (a too-old 3.9 or, increasingly, a too-new 3.14) that forced an incompatible interpreter and the resolve failed. Drop the detection and pass `--python-preference system`: uv reuses a compatible system Python when present and downloads a managed one otherwise, always honouring requires-python * test(router): filter aiohttp unclosed-session gc noise in test_async_fallbacks test_async_fallbacks asserts the last three captured log records are the router's fallback messages. Under the litellm_router_testing job (pytest -k router -n 4) many router tests share the module-level in_memory_llm_clients_cache (max 200, ttl 3600s). Older cached OpenAI/Azure clients get evicted while their aiohttp ClientSession is still open, and when the gc reclaims them aiohttp emits "Unclosed client session"/"Unclosed connector" through the asyncio logger. Those records land in caplog mid-test and push the expected router logs out of the last-three window, so the assertion flips to failing non-deterministically. These warnings are async cleanup noise, not router debug logs, so filter them out exactly like the existing leaked-task warnings before asserting order. The assertion on the three router fallback messages is unchanged. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-06-10 13:52:26 -07:00
Yassin Kortam	d5d6b26a72	fix: improve bedrock streaming hot path perf (#28720)	2026-05-28 11:31:37 -07:00
Yassin Kortam	2eab9ee2c0	perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths (#28289) ... * perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths - Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path - Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config - Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories - Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config) - Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default) - Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each) - Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk - Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch - Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk - Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params) - Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels - Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches - Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution * perf: address greptile review for anthropic streaming hot path - Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta events from different block indexes are observed without an intervening flush. Anthropic sends blocks strictly sequentially, but defensive bail prevents silent text-merging if the protocol ever interleaves. - Replace leaf-class `__dict__` check for `async_post_call_streaming_hook` in `_callback_capabilities` with a function-identity comparison that walks the MRO. A vendor base class can carry the override and the registered class can add nothing else; before this PR the hook was unconditionally invoked, so an inherited-override miss would silently drop the hook on the streaming path. - Add unit tests for both behaviors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(mypy): narrow model_name to str in cost-injection branch The hoisted cost_injection_active flag in chunk_processor encodes the `bool(model_name)` requirement but mypy can't track that invariant through the local, so the per-chunk `_process_chunk_with_cost_injection( chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed non-None local inside the cost-injection branch so mypy narrows correctly without changing runtime behavior. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-23 12:15:59 -07:00
Yassin Kortam	a6494e6fe3	perf: eliminate per-request callback scanning on proxy hot path (#27858) ... - Introduce `_CallbackCapabilities` dataclass and `ProxyLogging._callback_capabilities()` static method that inspects `litellm.callbacks` once and caches capability flags keyed on (list length, member ids); invalidates automatically when the callback list mutates without per-request iteration overhead - Replace O(n) `litellm.callbacks` walks in `async_pre_call_hook`, `during_call_hook`, `async_post_call_streaming_iterator_hook`, `async_post_call_streaming_hook`, and `post_call_response_headers_hook` with fast-path exits when no relevant callbacks are registered - Add `needs_iterator_wrap()` and `needs_per_chunk_streaming_hook()` instance methods to decouple iterator-level wrapping from per-chunk hook execution; avoids `get_response_string` materialization per chunk when no guardrail or chunk-hook callback is active - Introduce `_fast_serialize_simple_model_response_stream()` using `orjson` for common single-choice text streaming chunks, bypassing the full Pydantic serializer; falls back to `model_dump_json` for tool calls, logprobs, usage, and provider-specific fields - Add early-return in `_restamp_streaming_chunk_model` when downstream model already matches the requested model, avoiding unnecessary string comparisons on every chunk - Fix stale zero-cost cache bug in `_is_model_cost_zero`: move the per-router `_zero_cost_cache` dict onto the `Router` instance and clear it in `_invalidate_model_group_info_cache` so in-place pricing updates via `upsert_deployment` immediately resume budget enforcement - Add `scripts/benchmark_chat_completions_perf.py`: standalone async benchmarking tool with a mock OpenAI provider, LiteLLM proxy process management, non-streaming RPS, streaming TTFT, and full-stream latency measurements with repeat/median run support - Add comprehensive unit tests covering capability detection, cache invalidation, fast-path correctness, zero-cost cache regression, and the no-callback streaming fast path Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>	2026-05-14 09:28:31 -07:00
ryan-crabbe-berri	be84d5cd7d	ci: add manually-triggered mutation testing workflow (#27576) ... * ci: add manually-triggered mutation testing smoke workflow Adds a workflow_dispatch-only GitHub Actions workflow that runs mutmut against a single source/test pair (router_settings_endpoints) to validate the tooling end-to-end before scaling. The workflow reinstalls litellm non-editable so the mutants/ sandbox is not shadowed by the editable .pth on sys.path, and sets PYTHONPATH so the trampolined sandbox copy wins over site-packages. mutmut itself is pulled in via uv run --with so it does not appear in uv.lock or affect the shared dev environment. Includes a temporary push: trigger scoped to this branch so we can iterate before the workflow file lands on the default branch — to be removed before merging (workflow_dispatch only requires the file on the default branch to surface the manual trigger button). * ci(mutation): disable rerun and xdist plugins for mutmut runs mutmut's in-process pytest.main() call hits `INTERNALERROR: no option named 'filtered_exceptions'` from pytest-retry's pytest_configure hook. Reruns are also wrong for mutation testing — a "failed" mutant test that gets retried would mask which mutants are killed vs. survive. Disable retry, rerunfailures, and xdist via pytest_add_cli_args in [tool.mutmut]. * ci(mutation): uninstall pytest-retry before mutmut runs `-p no:retry` (and similar names) didn't match pytest-retry's entry-point name, so the plugin still loaded and crashed during mutmut's "Running clean tests" phase. Uninstalling the package is surgical and doesn't depend on guessing the entry-point name. * ci(mutation): emit per-survivor diffs to run-page summary + artifact The previous artifact only contained `mutmut results` text (which in mutmut 3.x lists survivor names but not the actual mutations). Adds: - `mutmut export-cicd-stats` to produce mutmut-cicd-stats.json with the killed/survived/total scoreboard. - `mutmut show <name>` per surviving mutant to capture each mutation as a unified diff. - A `mutmut-report.md` that combines summary + run-progress tail + per-survivor diffs, written to both the artifact and $GITHUB_STEP_SUMMARY (visible on the run page, no download needed). - Corrected artifact paths: stats files live under mutants/, not the project root. - The trampolined source file from the sandbox so survivors can be inspected even outside `mutmut show`. * ci(mutation): document intended manual weekly cadence in trigger comment * ci(mutation): generate ACH-style report with embedded function bodies Replaces the inline bash markdown generation with a Python script that: - Groups survivors by function (one section per function, function body shown once per section, surviving mutants nested as subsections) - Embeds each enclosing function's source via Python AST (so the agent has full context, not just a 3-line `mutmut show` diff) - Inlines the existing test file(s) listed in [tool.mutmut].tests_dir - Writes an ACH-style task description at the bottom following the prompt template from arXiv 2501.12862 Output goes to mutation-report.md (artifact) and the head of the file is appended to $GITHUB_STEP_SUMMARY for at-a-glance visibility. * fix(mutation report): correctly parse function names with leading underscores mutmut's mutant-name prefix is x_ (single underscore), so a function named _foo produces mutants x__foo__mutmut_N. The previous regex \.x__(.+)__mutmut_ ate the function's leading underscore as part of the prefix. Changed to \.x_(.+)__mutmut_ so leading underscores are preserved in the captured function name; verified for normal, leading- underscore, and dunder-method names. * feat(mutation report): full Meta ACH-style rendering with MUTANT delimiters For each surviving mutant, parse the mutmut sandbox trampoline file and render the mutated function as it appears in the source — with the differing lines wrapped in `# MUTANT START` / `# MUTANT END` comments, matching the format from Meta's ACH paper (arXiv 2501.12862, Table 1). Renames the function header back to its original name so the agent sees the function as it would appear in the file. Falls back to the unified diff if the trampoline lookup fails. Handles replace, insert, and delete diff ops; uses difflib's SequenceMatcher to find the differing line ranges. The unified diff is preserved in a collapsible <details> block as secondary context. * ci(mutation): scope to whole management_endpoints folder, drop temp push trigger Final scope before merge: - paths_to_mutate / tests_dir broadened from one file to the entire management_endpoints source/test folders - Trigger is now `workflow_dispatch` only — the temporary push: block used during workflow iteration is removed - timeout-minutes bumped from 60 to 350 (just under the GH-hosted job cap of 360); whole-folder mutation against ~15 files / ~7.5k LOC can take a few hours - Artifact path for the trampoline files glob-expanded to cover all files under mutants/litellm/proxy/management_endpoints/ * fix(mutation report): warn when multiple functions in a file share a name Addresses the Greptile review concern: ast.walk's first-match-wins behavior could embed the wrong function body when a file defines the same name in multiple places (e.g., a module-level helper and a class method). mutmut's mutant identifier does not carry class context, so we can't always determine which definition was mutated. find_function_in_file now returns the start line of every matching definition; render() surfaces a "Note: N functions named X" warning in the report when there is more than one match. The first match is still embedded as the body — the warning tells the reader to verify manually instead of silently using the wrong context. Smoke-tested against the existing artifact: single-match files render unchanged. * Fix mutation report anchors * Fix mutation report TOC anchors --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-05-11 15:19:57 -07:00
harish-berri	a67b7a7e87	Refactor Bedrock response stream shape handling (#27257) ... * Refactor Bedrock response stream shape handling - Introduced a module-level constant `BEDROCK_RESPONSE_STREAM_SHAPE` to cache the response stream shape, eliminating the need for per-instance caching in `BedrockEventStreamDecoderBase`. - Updated relevant methods to utilize the new constant, improving performance by avoiding redundant loading of the shape. - Added tests to ensure the shape is loaded correctly at import time and is consistent across different modules. - Added a new mock server script for testing Bedrock pass-through functionality. * Refactor response parsing for Bedrock and SageMaker - Improved code readability by formatting the parsing method calls in `AWSEventStreamDecoder` for both Bedrock and SageMaker response stream shapes. - Added blank lines for better separation of code blocks in `invoke_handler.py` and `common_utils.py` to enhance maintainability. * Enhance error handling for Bedrock and SageMaker response stream shape loading - Wrapped the loading logic in `_load_bedrock_response_stream_shape` and `_load_sagemaker_response_stream_shape` with try-except blocks to gracefully handle exceptions. - Added logging to warn when the response stream shape cannot be pre-loaded, ensuring the module imports cleanly. - Updated tests to verify that loading failures return `None` instead of propagating exceptions. * Implement error handling for missing response stream shapes in Bedrock and SageMaker - Added checks in `_parse_message_from_event` methods to raise appropriate errors when `BEDROCK_RESPONSE_STREAM_SHAPE` or `SAGEMAKER_RESPONSE_STREAM_SHAPE` is None, ensuring clearer error reporting. - Updated logging messages to reflect the unavailability of event-stream decoding for both Bedrock and SageMaker. - Enhanced unit tests to verify that the correct exceptions are raised when the response stream shapes are not loaded.	2026-05-06 17:39:38 -07:00
Yassin Kortam	950074eea2	fix: atomic TPM rate limit (#27001) ... Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>	2026-05-05 16:58:07 -07:00
Krrish Dholakia	b6fc75b3ce	Merge branch 'litellm_internal_staging' into litellm_adaptive_routing	2026-04-20 15:28:08 -07:00
Krrish Dholakia	386f334fee	Prompt Compression - add it to the proxy (#25729) ... * refactor: new agentic loop event hook simplifies how to create logic for tool based multi llm calls * fix: compress - make it work on anthropic input as well * fix(compress.py): working prompt compression for claude code ensures claude code messages can run through proxy easily * docs: add agentic loop hook guide * docs: add agentic_loop_hook to sidebar * fix: fix multiple arguments error * fix: fix tool call loop for compression on streaming /v1/messages * fix: fix linting errors * fix: fix ci/cd errors * feat(litellm_pre_call_utils.py): use claude code session for litellm session id allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation * fix: suppress incorrect mypy warning rE: module * revert: drop PR's changes to litellm/proxy/_experimental/out/ Restores the 34 HTML files under _experimental/out/ to their pre-PR paths (X/index.html -> X.html). All renames are R100 (content unchanged); no other files are touched. * fix: address greptile review comments on PR #25729 - Skip ``kwargs["tools"] = []`` injection when compression is a no-op — Anthropic Messages rejects empty tool arrays on requests that did not originally declare tools. - Move agentic-loop safety guards (fingerprint cycle / max depth) out of the per-callback try/except so they propagate instead of being swallowed by the generic exception handler. Extracted _check_agentic_loop_safety. - Gate generic ``x-<vendor>-session-id`` capture behind the LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to preserve backwards compatibility; explicit x-litellm-* headers are unaffected. - Fix monkeypatch target in pre-call-hook test to patch the actual module-level binding (litellm.integrations.compression_interception.handler.compress). - Add regression tests for empty-tools skip and opt-in session capture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag Generic x-<vendor>-session-id header capture is a new feature and only runs *after* the explicit x-litellm-trace-id / x-litellm-session-id checks, so it does not change behavior for any existing caller that was already using the LiteLLM headers — no backwards-incompatibility to gate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor(compress): replace input_type with CallTypes call_type Drop the bespoke ``CompressionInputType`` literal and use the existing ``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()`` now takes ``call_type: Union[CallTypes, str]`` (default ``CallTypes.completion``) — no new concept to learn, and the enum is already the way the rest of the codebase talks about request shapes. Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions shape) and ``anthropic_messages`` (Anthropic structured content blocks). Updated: compress(), the compression_interception handler, tests, docs, and the two eval scripts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-04-20 15:08:00 -07:00
Krrish Dholakia	924fa6a3bc	feat: commit new adaptive routing	2026-04-18 21:29:39 -07:00
Krrish Dholakia	dd4a1d2be2	feat: add adaptive routing to litellm ... allow model routing to improve based on conversation signals ensures router is picking best model for task	2026-04-18 16:35:17 -07:00
Ishaan Jaffer	e8461b5b97	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
ishaan-berri	cb8fc480e6	Merge pull request #25732 from harish876/health-check-oom ... Optimize database query to prevent OOM errors during health checks	2026-04-15 18:13:11 -07:00
harish876	d20c70f24c	Optimize database query which fetches latest model_id, model_name pairs and dedupes them in memory. ... Current fix includes - Updates test case - Optimized query with docstring. The change leverages deduplication and sorting logic from SQL - Added a bench script to differentiate peak memory usage before and after	2026-04-15 00:54:37 +00:00
ishaan-berri	0e43050a01	Merge pull request #25650 from BerriAI/litellm_dev_04_13_2026_p1 ... feat: add litellm.compress() — BM25-based prompt compression with ret…	2026-04-14 12:24:47 -07:00
Krrish Dholakia	26c7412339	feat: add litellm.compress() — BM25-based prompt compression with retrieval tool (#25637) ... * feat: add litellm.compress() for BM25-based context compression Adds a compress() utility that reduces context size for LLM calls using BM25 relevance scoring (with optional semantic embeddings via litellm.embedding()). Messages below a token threshold pass through unchanged; messages above are scored, ranked, and the lowest-relevance ones replaced with stubs. Originals are cached and a retrieval tool is injected so the model can recover dropped content on demand. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(compress): truncate high-scoring messages instead of fully stubbing them When a relevant message was too large to fit in the token budget it was replaced with a stub, leaving the LLM with no real content to work with. Now the highest-scoring overflow message is truncated (first 70% + last 30% of words) to fill the remaining budget, so the LLM always receives actual content rather than just a retrieval pointer. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(bm25): add prefix expansion so query terms match inflected doc tokens "cook" now matches "cooking", "auth" matches "authentication", etc. Without this, short query terms scored 0 against longer inflected forms in documents, causing the wrong message to be kept. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add routing correctness test and eval harness for litellm.compress() - test_simple_compression: parametrized test verifying BM25 routes the right message based on query ("How to cook?" keeps cooking, "Fix auth" keeps auth content) - eval_compression.py: end-to-end eval harness comparing baseline vs compressed model performance on HumanEval-style coding problems Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat(eval): add SWE-bench Lite compression eval harness Uses princeton-nlp/SWE-bench_Lite_bm25_27K which bundles ~27k tokens of BM25-retrieved repo context per problem — large enough to meaningfully stress litellm.compress() without Docker or GitHub API calls. Proxy eval metrics (no test runner needed): - has_diff: model produced a valid unified diff - file_overlap: fraction of gold-patch files in generated patch - exact_file_match: generated patch touches exactly the right files Run: python tests/eval_swe_bench.py --model gpt-4o --problems 10 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(eval): robust dataset loading + sys.path fix for worktree imports - Add HuggingFace API fallback so the SWE-bench loader doesn't need the `datasets` library (avoids pyarrow/numpy binary compat issues) - Insert repo root into sys.path so compression module resolves from worktrees - Use direct import of litellm_compress to avoid __getattr__ issues Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * improve compression quality: line-based truncation, multi-message budget, 70% default target - Switch truncate_message from word-based to line-based splitting to preserve code structure (function boundaries, indentation) - Allow multiple messages to be truncated instead of burning entire budget on one overflow message - Raise default compression target from 50% to 70% of trigger for better quality/cost tradeoff - Add --compression-target CLI arg to SWE-bench eval harness - Move tests to canonical locations (tests/test_litellm/, scripts/) - Add docs page and sidebar entries for compress() Eval results (5 problems, Opus, trigger=10k): Hunk overlap delta improved from -0.417 to -0.221 Content similarity now matches baseline (+0.006) Cost savings: 72% Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add SWE-bench performance results to compress() docs Include benchmark table from Opus eval (5 problems, trigger=10k) showing 72% cost savings with file-level quality fully preserved. Add metric explanations and eval runner examples. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(eval): use tolerance-based hunk overlap metric The exact line-number matching was too brittle — LLM-generated patches often target the right code region but with slightly offset line numbers. Switch to hunk-level overlap with a 10-line tolerance window so nearby edits count as matches. This better reflects actual patch quality. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add compression_interception callback for LiteLLM Proxy Add a proxy callback that automatically compresses incoming /v1/messages payloads above a configurable token threshold, runs the retrieval tool loop server-side, and returns the final response. This brings compress() support to proxy deployments (e.g. Claude Code via /v1/messages). - New callback: litellm/integrations/compression_interception/ - Proxy config: compression_interception_params in litellm_settings - Support for input_type param in compress() (openai vs anthropic) - Docs: proxy setup instructions with YAML config example - Tests: 139-line unit test suite for the interception handler Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Revert "feat: add compression_interception callback for LiteLLM Proxy" This reverts commit 72bd5cb152. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-04-13 12:23:54 -07:00
stuxf	a6c30b30bf	build: migrate packaging, CI, and Docker from Poetry to uv (#25007) ... * build: migrate packaging metadata to uv * ci: move automation and local tooling to uv * docker: migrate image builds and runtime setup to uv * docs: update install and deployment guidance for uv * chore: align auxiliary scripts and tests with uv * test: harden test_litellm isolation * fix: keep release and health check images self-contained * build: pin uv tooling and health check deps * test: isolate bedrock image request formatting from suite state * test: cover sandbox executor requirements flow * ci: fix circleci no-op command steps * ci: fix circleci publish workflow parsing * fix: stabilize remaining uv migration CI checks * ci: increase matrix test timeout headroom * fix: restore published docker and license coverage * fix: restore proxy runtime build parity * fix: restore proxy extras parity and venv migrations * ci: persist uv path across circleci steps * fix: keep psycopg binary in default test env * docker: preserve prisma cache across stages * test: run local proxy checks through uv python * build: restore runtime deps moved into ci * build: refresh uv lock after upstream merge * fix: restore module import in test_check_migration after merge The conflict resolution imported only the function but the test body references check_migration as a module throughout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching - Move google-generativeai, Pillow, tenacity back to ci group (they are lazily imported and bloat the base SDK install needlessly) - Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant in Docker where system Node.js is already installed via apk) - Remove all nodejs-wheel node replacement and venv npm patching blocks from Dockerfiles since the wheel is no longer installed - Add --no-default-groups to CodSpeed benchmark workflow so the benchmark environment matches the old minimal pip install footprint - Apply standard uv two-phase Docker pattern: copy metadata first, install deps (cached layer), then copy source and install project - Replace CircleCI enterprise no-op with proper uv sync command Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: regenerate uv.lock after removing nodejs-wheel-binaries Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): use cache/restore instead of cache to prevent cache poisoning The old workflow used actions/cache/restore (read-only). The uv migration changed it to actions/cache (read-write), which zizmor flags as a cache poisoning risk. Restore the safer read-only variant. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert The setup-uv action enables caching by default, which zizmor flags as a cache poisoning risk. Disable it since we already use a read-only cache/restore step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): disable setup-uv cache in publish workflow Silences zizmor cache-poisoning alert. Publishing workflow runs infrequently on protected branches so caching adds no real benefit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(test): remove duplicate verbose_logger mock in test_check_migration The logger was patched twice — first via mocker.patch() then via mocker.patch.object(autospec=True). The second call fails because autospec cannot inspect an already-mocked attribute. Remove the redundant first patch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): free disk space before Docker build in test-server-root-path The Dockerfile.non_root build ran out of disk on the CI runner. Remove Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-09 11:46:23 -07:00
Yuneng Jiang	51af6fedb3	[Infra] Harden supply chain: remove unused scripts, add pip binary-only install ... Remove ci_cd/publish-proxy-extras.sh (dead, unreferenced PyPI publish script) and .pre-commit-config.yaml (pulls external repos from GitHub on git commit). Add --only-binary :all: to scripts/install.sh to prevent execution of malicious setup.py during pip install. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-02 14:13:57 -07:00
Yuneng Jiang	5f63873dca	[Infra] Pin all Docker build dependencies to exact versions ... Pin every dependency across all Docker builds so upgrades are intentional. Verified by building all 3 production images and diffing pip freeze against known-good v1.83.0-nightly baselines — zero version drift. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-04-01 00:05:39 -07:00
Ishaan Jaff	8e61b32b8e	[Staging] - Ishaan March 17th (#23903) ... * feat(xai): add grok-4.20 beta 2 models with pricing (#23900) Add three grok-4.20 beta 2 model variants from xAI: - grok-4.20-multi-agent-beta-0309 (reasoning + multi-agent) - grok-4.20-beta-0309-reasoning (reasoning) - grok-4.20-beta-0309-non-reasoning Pricing (from https://docs.x.ai/docs/models): - Input: $2.00/1M tokens ($0.20/1M cached) - Output: $6.00/1M tokens - Context: 2M tokens All variants support vision, function calling, tool choice, and web search. Closes LIT-2171 * docs: add Quick Install section for litellm --setup wizard (#23905) * docs: add Quick Install section for litellm --setup wizard * docs: clarify setup wizard is for local/beginner use * feat(setup): interactive setup wizard + install.sh (#23644) * feat(setup): add interactive setup wizard + install.sh Adds `litellm --setup` — a Claude Code-style TUI onboarding wizard that guides users through provider selection, API key entry, and proxy config generation, then optionally starts the proxy immediately. - litellm/setup_wizard.py: wizard with ASCII art, numbered provider menu (OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama), API key prompts, port/master-key config, and litellm_config.yaml generation - litellm/proxy/proxy_cli.py: adds --setup flag that invokes the wizard - scripts/install.sh: curl-installable script (detect OS/Python, pip install litellm[proxy], launch wizard) Usage: curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh litellm --setup * fix(install.sh): remove orange color, add LITELLM_BRANCH env var for branch installs * fix(install.sh): install from git branch so --setup is available for QA * fix(install.sh): remove stale LITELLM_BRANCH reference that caused unbound variable error * fix(install.sh): force-reinstall from git to bypass cached PyPI version * fix(install.sh): show pip progress bar during install * fix(install.sh): always launch wizard via $PYTHON_BIN -m litellm, not PATH binary * fix(install.sh): use litellm.proxy.proxy_cli module (no __main__.py exists) * fix(install.sh): suppress RuntimeWarning from module invocation * fix(install.sh): use Python bin-dir litellm binary to avoid CWD sys.path shadowing * fix(install.sh): use sysconfig.get_path('scripts') to find pip-installed litellm binary * fix(install.sh): redirect stdin from /dev/tty on exec so wizard gets terminal, not exhausted pipe * fix(install.sh): warn about git clone duration, drop --no-cache-dir so re-runs are faster * feat(setup_wizard): arrow-key selector, updated model names * fix(setup_wizard): use sysconfig binary to start proxy, not python -m litellm * feat(setup_wizard): credential validation after key entry + clear next-steps after proxy start * style(install.sh): show git clone warning in blue * refactor(setup_wizard): class with static methods, use check_valid_key from litellm.utils * address greptile review: fix yaml escaping, port validation, display name collisions, tests - setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys - setup_wizard.py: add _styled_input() with readline ANSI ignore markers - setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture - setup_wizard.py: validate port range 1-65535, initialize before loop - setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai - setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict - setup_wizard.py: skip model_list entries for providers with no credentials - setup_wizard.py: prompt for azure deployment name - setup_wizard.py: wrap os.execlp in try/except with friendly fallback - setup_wizard.py: wrap config write in try/except OSError - setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite) - setup_wizard.py: add .gitignore tip next to key storage notice - setup_wizard.py: fix run_setup_wizard() return type annotation to None - scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh) - scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch) - scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat - scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies - tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape * style: black format setup_wizard.py * fix: address remaining greptile issues - Windows compat, YAML quoting, credential flow - guard termios/tty imports with try/except ImportError for Windows compat - quote master_key as YAML double-quoted scalar (same as env vars) - remove unused port param from _build_config signature - _validate_and_report now returns the final key so re-entered creds are stored - add test for master_key YAML quoting * fix: add --port to suggested command, guard /dev/tty exec in install.sh * fix: quote api_base in YAML, skip azure if no deployment, only redraw on state change * fix: address greptile review comments - _yaml_escape: add control character escaping (\n, \r, \t) - test: fix tautological assertion in test_build_config_azure_no_deployment_skipped - test: add tests for control character escaping in _yaml_escape * feat(ui): remove Chat UI page link and banner from sidebar and playground (#23908) * feat(guardrails): MCPJWTSigner - built-in guardrail for zero trust MCP auth (#23897) * Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers * Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present. * Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings. * Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers * Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present. * Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings. * feat(guardrails): add MCPJWTSigner built-in guardrail for zero trust MCP auth Signs outbound MCP tool calls with a LiteLLM-issued RS256 JWT so MCP servers can trust a single signing authority instead of every upstream IdP. Enable in config.yaml: guardrails: - guardrail_name: mcp-jwt-signer litellm_params: guardrail: mcp_jwt_signer mode: pre_mcp_call default_on: true JWT carries sub (user_id), act.sub (team_id, RFC 8693), tool-level scope, iss, aud, iat/exp/nbf. RSA-2048 keypair auto-generated at startup unless MCP_JWT_SIGNING_KEY env var is set. Adds /.well-known/jwks.json endpoint and jwks_uri to /.well-known/openid-configuration so MCP servers can verify LiteLLM-issued tokens via OIDC discovery. * Update MCPServerManager to raise HTTPException with status code 400 for extra headers in OpenAPI-backed servers. Adjust tests to verify the correct status code and exception message. * fix: address P1 issues in MCPJWTSigner - OpenAPI servers: warn + skip header injection instead of 500 - JWKS Cache-Control: 5min for auto-generated keys, 1h for persistent - sub claim: fallback to apikey:{token_hash} for anonymous callers - ttl_seconds: validate > 0 at init time * docs: add MCP zero trust auth guide with architecture diagram * docs: add FastMCP JWT verification guide to zero trust doc * fix: address remaining Greptile review issues (round 2) - mcp_server_manager: warn when hook Authorization overwrites existing header - __init__: remove _mcp_jwt_signer_instance from __all__ (private internal) - discoverable_endpoints: copy dict instead of mutating in-place on OIDC augmentation - test docstring: reflect warn-and-continue behavior for OpenAPI servers - test: update scope assertions for least-privilege (no mcp:tools/list on tool-call JWTs) * fix: address Greptile round 3 feedback - initialize_guardrail: validate mode='pre_mcp_call' at init time — misconfigured mode silently bypasses JWT injection, which is a zero-trust bypass - _build_claims: remove duplicate inline 'import re' (module-level import already present) - _types.py: add TODO comment explaining jwt_claims is forward-compat plumbing for a follow-up PR that will forward upstream IdP claims into outbound MCP JWTs * feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes Addresses all missing pieces from the scoping doc review: FR-5 (Verify + re-sign): MCPJWTSigner now accepts access_token_discovery_uri and token_introspection_endpoint. When set, the incoming Bearer token is extracted from raw_headers (threaded through pre_call_tool_check), verified against the IdP's JWKS (JWT) or introspected (opaque), and only re-signed if valid. Falls back to user_api_key_dict.jwt_claims for LiteLLM JWT-auth mode. FR-12 (Configurable end-user identity mapping): end_user_claim_sources ordered list drives sub resolution — sources: token:<claim>, litellm:user_id, litellm:email, litellm:end_user_id, litellm:team_id. FR-13 (Claim operations): add_claims (insert-if-absent), set_claims (always override), remove_claims (delete) applied in that order. FR-14 (Two-token model): channel_token_audience + channel_token_ttl issue a second JWT injected as x-mcp-channel-token: Bearer <token>. FR-15 (Incoming claim validation): required_claims raises HTTP 403 when any listed claim is absent; optional_claims passes listed claims from verified token into the outbound JWT. FR-9 (Debug headers): debug_headers: true emits x-litellm-mcp-debug with kid, sub, iss, exp, scope. FR-10 (Configurable scopes): allowed_scopes replaces auto-generation. Also fixed: tool-call JWTs no longer grant mcp:tools/list (overpermission). P1 fixes: - proxy/utils.py: _convert_mcp_hook_response_to_kwargs merges rather than replaces extra_headers, preserving headers from prior guardrails. - mcp_server_manager.py: warns when hook injects Authorization alongside a server-configured authentication_token (previously silent). - mcp_server_manager.py: pre_call_tool_check now accepts raw_headers and extracts incoming_bearer_token so FR-5 verification has the raw token. - proxy/utils.py: remove stray inline import inspect inside loop (pre-existing lint error, now cleaned up). Tests: 43 passing (28 new tests covering all FR flags + P1 fixes). * feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes (core) Remaining files from the FR implementation: mcp_jwt_signer.py — full rewrite with all new params: FR-5: access_token_discovery_uri, token_introspection_endpoint, verify_issuer, verify_audience + _verify_incoming_jwt(), _introspect_opaque_token() FR-12: end_user_claim_sources ordered resolution chain FR-13: add_claims, set_claims, remove_claims FR-14: channel_token_audience, channel_token_ttl → x-mcp-channel-token FR-15: required_claims (raises 403), optional_claims (passthrough) FR-9: debug_headers → x-litellm-mcp-debug FR-10: allowed_scopes; tool-call JWTs no longer over-grant tools/list mcp_server_manager.py: - pre_call_tool_check gains raw_headers param to extract incoming_bearer_token - Silent Authorization override warning fixed: now fires when server has authentication_token AND hook injects Authorization tests/test_mcp_jwt_signer.py: 28 new tests covering all FR flags + P1 fixes (43 total, all passing) * fix(mcp_jwt_signer): address pre-landing review issues - Remove stale TODO comment on UserAPIKeyAuth.jwt_claims — the field is already populated and consumed by MCPJWTSigner in the same PR - Fix _get_oidc_discovery to only cache the OIDC discovery doc when jwks_uri is present; a malformed/empty doc now retries on the next request instead of being permanently cached until proxy restart - Add FR-5 test coverage for _fetch_jwks (cache hit/miss), _get_oidc_discovery (cache/no-cache on bad doc), _verify_incoming_jwt (valid token, expired token), _introspect_opaque_token (active, inactive, no endpoint), and the end-to-end 401 hook path — 53 tests total, all passing * docs(mcp_zero_trust): rewrite as use-case guide covering all new JWT signer features Add scenario-driven sections for each new config area: - Verify+re-sign with Okta/Azure AD (access_token_discovery_uri, end_user_claim_sources, token_introspection_endpoint) - Enforcing caller attributes with required_claims / optional_claims - Adding metadata via add_claims / set_claims / remove_claims - Two-token model for AWS Bedrock AgentCore Gateway (channel_token_audience / channel_token_ttl) - Controlling scopes with allowed_scopes - Debugging JWT rejections with debug_headers Update JWT claims table to reflect configurable sub (end_user_claim_sources) * fix(mcp_jwt_signer): wire all config.yaml params through initialize_guardrail The factory was only passing issuer/audience/ttl_seconds to MCPJWTSigner. All FR-5/9/10/12/13/14/15 params (access_token_discovery_uri, end_user_claim_sources, add/set/remove_claims, channel_token_audience, required/optional_claims, debug_headers, allowed_scopes, etc.) were silently dropped, making every advertised advanced feature non-functional when loaded from config.yaml. Add regression test that asserts every param is wired through correctly. * docs(mcp_zero_trust): add hero image * docs(mcp_zero_trust): apply Linear-style edits - Lead with the problem (unsigned direct calls bypass access controls) - Shorter statement section headers instead of question-form headers - Move diagram/OIDC discovery block after the reader is bought in - Add 'read further only if you need to' callout after basic setup - Two-token section now opens from the user problem not product jargon - Add concrete 403 error response example in required_claims section - Debug section opens from the symptom (MCP server returning 401) - Lowercase claims reference header for consistency * fix(mcp_jwt_signer): fix algorithm confusion attack + add OIDC discovery 24h TTL - Remove alg from unverified JWT header; use signing_jwk.algorithm_name from JWKS key instead. Reading alg from attacker-controlled headers enables alg:none / HS256 confusion attacks. - Add _oidc_discovery_fetched_at timestamp and _OIDC_DISCOVERY_TTL = 86400 (24h). Without a TTL the cached discovery doc never refreshes, so IdP key rotation is invisible. --------- Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com> * fix(ci): stabilize CI - formatting, type errors, test polling, security CVEs, router bug, batch resolution Fix 1: Run Black formatter on 35 files Fix 2: Fix MyPy type errors: - setup_wizard.py: add type annotation for 'selected' set variable - user_api_key_auth.py: remove redundant type annotation on jwt_claims reassignment Fix 3: Fix spend accuracy test burst 2 polling to wait for expected total spend instead of just 'any increase' from burst 2 Fix 4: Bump Next.js 16.1.6 -> 16.1.7 to fix CVE-2026-27978, CVE-2026-27979, CVE-2026-27980, CVE-2026-29057 Fix 5: Fix router _pre_call_checks model variable being overwritten inside loop, causing wrong model lookups on subsequent deployments. Use local _deployment_model variable instead. Fix 6: Add missing resolve_output_file_ids_to_unified call in batch retrieve non-terminal-to-terminal path (matching the terminal path behavior) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * chore: regenerate poetry.lock to sync with pyproject.toml Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix: format merged files from main and regenerate poetry.lock Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(mypy): annotate jwt_claims as Optional[dict] to fix type incompatibility Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): update router region test to use gpt-4.1-mini (fix flaky model lookup) Replace deprecated gpt-3.5-turbo-1106 with gpt-4.1-mini + mock_response in test_router_region_pre_call_check, following the same pattern used in commit 717d37cc5b for test_router_context_window_check_pre_call_check_out_group. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * ci: retry flaky logging_testing (async event loop race condition) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): aggregate all mock calls in langfuse e2e test to fix race condition The _verify_langfuse_call helper only inspected the last mock call (mock_post.call_args), but the Langfuse SDK may split trace-create and generation-create events across separate HTTP flush cycles. This caused an IndexError when the last call's batch contained only one event type. Fix: iterate over mock_post.call_args_list to collect batch items from ALL calls. Also add a safety assertion after filtering by trace_id and mark all langfuse e2e tests with @pytest.mark.flaky(retries=3) as an extra safety net for any residual timing issues. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): black formatting + update OpenAPI compliance tests for spec changes - Apply Black 26.x formatting to litellm_logging.py (parenthesized style) - Update test_input_types_match_spec to follow $ref to InteractionsInput schema (Google updated their OpenAPI spec to use $ref instead of inline oneOf) - Update test_content_schema_uses_discriminator to handle discriminator without explicit mapping (Google removed the mapping key from Content discriminator) Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * revert: undo incorrect Black 26.x formatting on litellm_logging.py The file was correctly formatted for Black 23.12.1 (the version pinned in pyproject.toml). The previous commit applied Black 26.x formatting which was incompatible with the CI's Black version. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * fix(ci): deduplicate and sort langfuse batch events after aggregation The Langfuse SDK may send the same event (e.g., trace-create) in multiple flush cycles, causing duplicates when we aggregate from all mock calls. After filtering by trace_id, deduplicate by keeping only the first event of each type, then sort to ensure trace-create is at index 0 and generation-create at index 1. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>	2026-03-18 15:09:01 -07:00
Ishaan Jaff	1f412bc6d8	[Feat] Add Tool Policies for AI Gateway (#22732) ... * fix: fix ui render * fix: fix minor bugs * refactor: use prisma functions instead of raw sql (safer) * fix(add-new-tiles-to-tool-policies): allow developer to see what's available * feat: ensure tool allowlist runs correctly for tool names + mcp's * refactor: more ui improvements * feat: working key tool blocking * feat(tools): show tool logs * refactor: backend code improvements * refactor: improve log viewer for tools * fix: address PR review feedback for tool access control - Add missing blocked_tools column to root schema.prisma (schema drift) - Invalidate ToolPolicyRegistry after policy mutations so changes take effect immediately - Remove dead code: unused get_effective_policies, get_tool_policies_cached, and helpers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: race condition in permission resolution and remove duplicate allowlist check - Use atomic update_many with object_permission_id=None to prevent concurrent requests from creating orphaned permission rows and losing tool blocks - Remove duplicate allowed_tools enforcement from guardrail (already enforced in auth layer via check_tools_allowlist) - Move inline uuid import to module level Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * update to account for userAgent * UI - Add ToolDetails * input/output policy * LiteLLM_PolicyAttachmentTable * LiteLLM_PolicyAttachmentTable * fix: add _enqueue_tool_registry_upsert * fix: tool mgmt endpoints * tool mgmt endpoints * Update tests/test_litellm/proxy/db/test_tool_registry_writer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update tests/test_litellm/proxy/db/test_tool_registry_writer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update tests/test_litellm/proxy/db/test_tool_registry_writer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: sync root schema.prisma and fix test_tool_registry_writer for input/output policy - Migrate root schema.prisma LiteLLM_ToolTable from call_policy to input_policy/output_policy, add missing user_agent and last_used_at columns (now consistent with litellm/proxy/schema.prisma and litellm-proxy-extras) - Fix SpendLogToolIndex comment across all three schema files - Fix all call_policy references in test_tool_registry_writer.py: swapped update_tool_policy arguments, wrong get_tools_by_names return type assertions, _mock_tool_row setting call_policy instead of input_policy Addresses Greptile review feedback on PR #22732. Made-with: Cursor --------- Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>	2026-03-03 20:22:20 -08:00
Krish Dholakia	67f90254ed	feat(guardrails): team-based guardrail registration and approval workflow (#22459) ... * feat(guardrails): team-based guardrail registration and approval workflow Add team-based guardrail submission system where teams can register Generic Guardrail API guardrails for admin review. Includes: - POST /guardrails/register endpoint for team-scoped submissions - Admin review endpoints (list/get/approve/reject submissions) - Team Guardrails tab in the UI dashboard - extra_headers support for forwarding client headers to guardrail APIs - Prisma schema migration for status, submitted_at, reviewed_at fields - Documentation for team-based guardrails and static/dynamic headers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(guardrails): address review feedback - SSRF, silent failure, redundant query - Validate api_base URL scheme (http/https only) and hostname in register_guardrail to prevent SSRF via team submissions - Return warning field in approve response when in-memory initialization fails so admins know the guardrail won't work until next sync cycle - Eliminate redundant DB query in list_guardrail_submissions by fetching all team guardrails once and deriving both filtered list and summary counts from the single result set Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(guardrails): add pending_review status guard to reject endpoint Prevent rejecting already-active or already-rejected guardrails, which would create a DB/memory inconsistency (active in memory but rejected in DB). Now mirrors the approve endpoint's status check. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-02 22:06:49 -08:00
Krish Dholakia	12c4876891	Agents - assign tools (#22064) ... * feat(proxy): add max_iterations limiter for agent session loops (#22058) Adds a new proxy hook that enforces a per-session cap on the number of LLM calls an agentic loop can make. Callers send a session_id with each request, and the hook counts calls per session, returning 429 when the configured max_iterations limit is exceeded. - Uses Redis Lua script for atomic increment (multi-instance safe) - Falls back to in-memory cache when Redis unavailable - Follows parallel_request_limiter_v3 pattern - Configurable via key metadata: {"max_iterations": 25} - Session counters auto-expire via TTL (default 1hr) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * feat: add new code execution dataset * feat(agent_endpoints/): allow giving agents keys * fix: ui fixes * feat: allow assigning mcp servers to agents * fix: eliminate duplicate DB queries in MCP agent auth and N+1 in agent listing (#22110) - Extract _get_agent_object_permission helper so _get_allowed_mcp_servers_for_agent and _get_agent_tool_permissions_for_server share a single DB fetch instead of each independently querying the same agent row (was 1+N queries per MCP request) - Use include={"object_permission": True} on find_many in get_all_agents_from_db to eagerly load permissions in one query instead of N+1 - Use include={"object_permission": True} on create/update/find_unique in all agent CRUD operations, removing attach_object_permission_to_dict follow-up calls Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-25 11:44:30 -08:00
Ryan Crabbe	e0ddb2a525	fix: guard print_aggregate against empty latencies	2026-02-23 10:06:10 -08:00
Ryan Crabbe	95d9514054	fix: add auth headers and empty latencies guard to benchmark script	2026-02-23 09:58:40 -08:00
Ryan Crabbe	94b76ea9ad	feat: add network_mock transport for benchmarking proxy overhead without real API calls ... Intercepts at httpx transport layer so the full proxy path (auth, routing, OpenAI SDK, response transformation) is exercised with zero-latency responses. Activated via `litellm_settings: { network_mock: true }` in proxy config.	2026-02-21 17:52:39 -08:00
Alexsander Hamir	7f81dea8b3	Add custom auth header support and increase default prompt size to 100k chars (#19436)	2026-01-20 13:25:12 -08:00
Alexsander Hamir	270b41b0f4	Simplify file comments (#19382)	2026-01-19 17:01:38 -08:00
Alexsander Hamir	0cd7763d5f	Add health check scripts and parallel execution support (#19295) ... - Add health_check_client.py for monitoring model availability - Add health_check_client_README.md with usage documentation - Add health_check_requirements.txt for dependencies - Add run_parallel_health_checks.ps1 (PowerShell version) - Add run_parallel_health_checks.sh (Bash version) - Organize all scripts under scripts/health_check/ directory	2026-01-19 08:38:38 -08:00
Xiaohan Fu	07fe9e8604	implement failopen option default to True on grayswan guardrail (#18266) ... * implement failopen option default to True * introduce a config to set the timeout limit (default to 30)	2026-01-06 15:17:05 +05:30