1653 commits

Author	SHA1	Message	Date
mateo-berri	ae81625ee6	feat(anthropic): add Claude Opus 5 ... Registers claude-opus-5 across the cost maps and provider lists so the model prices, reports its real 1M/128K limits, and advertises its capabilities instead of falling through the generalization patterns at zero cost. Adds the first-party entry plus the Bedrock (base, global, us, eu, au, jp), Vertex AI, and Azure AI variants. Pricing matches Opus 4.8 at $5/$25 per MTok with the usual 1.1x regional premium on the cross-region inference profiles, and fast mode is priced at 2x through provider_specific_entry on the first-party entry only. Two fields deliberately differ from Opus 4.8: prompt_cache_min_tokens drops to 512, and bedrock_output_config_effort_ceiling is omitted because Bedrock accepts output_config.effort="max" for Opus 5.	2026-07-24 10:43:49 -07:00
mateo-berri	e411d637b3	feat(gemini): day-0 pricing for gemini-3.6-flash and gemini-3.5-flash-lite	2026-07-21 08:50:40 -07:00
Mateo Wang	1e741094fe	Merge pull request #33807 from BerriAI/litellm_vertex_azure_midsys ... fix(vertex,azure): model-aware mid-conversation system for Claude /v1/messages	2026-07-20 20:48:47 -04:00
mateo-berri	23b5b7d199	fix(vertex,azure): model-aware mid-conversation system for Claude /v1/messages ... Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic Messages contract, which was verified live to be byte-identical to api.anthropic.com: a leading role:"system" entry in messages is rejected on every model ("messages.0: use the top-level 'system' parameter"), and a mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5 but 400s on Claude 4.7 and older ("role 'system' is not supported on this model"). This is the same contract Bedrock Invoke already handles model-aware (PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude Code session on an older Vertex/Azure Claude model hard-400s on its reminder turns, and the only thing sparing 4.8+/5 was that nothing was hoisted Extract Bedrock's model-gated normalization into the shared AnthropicMessagesConfig base as _normalize_system_role_messages and call it from the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the leading run of system entries and keep mid-conversation reminders in place so the top-level system prefix stays byte-identical and the prompt cache is preserved; unflagged models hoist every system entry so the request returns a completion instead of a 400 Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5 cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system fallback rule, so without the explicit flag those models would be treated as unsupported and hoist every reminder, collapsing the prompt cache (the exact customer regression). A per-provider test guards this so future 4.8+/5 entries cannot silently miss the flag Closes the Vertex/Azure gap from the customer RCA	2026-07-20 16:48:29 -07:00
Tin Chi Lo	d966122249	fix(fireworks_ai): correct glm-5p2 prompt-cache read price to $0.14/1M ... glm-5p2 (and its fireworks_ai/glm-5p2 alias) carried cache_read_input_token_cost of 2.6e-07, the GLM 5.1 rate; the entry was seeded from the wrong row. Fireworks' standard serverless rate for GLM 5.2 is $0.14/1M = 1.4e-07, so every prompt-cache hit was billed at nearly double the real rate. Corrects the value in both the canonical map and the bundled backup. The existing fireworks cost-calculator test now reads the cached rate from the map instead of hardcoding it, so it tracks the shipped value.	2026-07-17 17:57:36 -07:00
devin-ai-integration[bot]	00e0dd1bc1	fix(pricing): mark realtime-only gpt-realtime models as mode realtime (#33728) ... The gpt-realtime family (OpenAI and Azure) only serves /v1/realtime and is rejected by /v1/chat/completions with "This is not a chat model", but the cost map tagged them mode=chat. Retag them mode=realtime (a value already used by gemini-live and handled by the health-check realtime handler) and add realtime to the ModelInfoBase mode literal. Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-17 17:39:19 +00:00
Tin Chi Lo	ba70189e32	fix(router): resolve prompt cache minimum per model instead of a flat 1024 ... MINIMUM_PROMPT_CACHE_TOKEN_COUNT was a flat 1024 described as "minimum number of tokens to cache a prompt by Anthropic". Anthropic's minimum cacheable prefix is per-model and ranges from 512 to 4096, and it can differ per platform for the same model, so one constant is wrong in both directions is_prompt_caching_valid_prompt gates PromptCachingDeploymentCheck, which is what optional_pre_call_checks: ["prompt_caching"] turns on. When it believes a prompt is cacheable, async_filter_deployments pins routing to whichever deployment previously served that prefix. For a prompt between 1024 and 4096 tokens on Opus 4.6, Opus 4.5 or Haiku 4.5, litellm judged it cacheable and constrained routing while the provider never cached it, so the pin cost load balancing for nothing. In the other direction Fable 5 caches from 512 tokens, so a 512 to 1024 token prefix was refused a pin it had earned The minimum now resolves from prompt_cache_min_tokens in the model cost map, which keeps it current with new models and lets the Bedrock override for Fable 5 fall out of the existing per-entry keys with no special casing. MINIMUM_PROMPT_CACHE_TOKEN_COUNT stays as a global escape hatch when explicitly set, and as the fallback for models the cost map has no entry for async_filter_deployments only ever receives the model group alias, never a model name, so it resolves the threshold from healthy_deployments instead. A group may mix models with different minimums, so it takes the max: a prompt is only treated as cacheable when it clears every member's minimum, because an unnecessary pin is the defect being fixed while a missed pin only forfeits an optimization Gemini context caching shares this gate and has the same defect; its entries are left unset so they keep today's behavior, tracked separately in LIT-4525	2026-07-16 19:10:23 -07:00
yuneng-jiang	f1f33f560f	Merge pull request #33335 from BerriAI/litellm_oss_daily_2026_07_10 ... chore(ci): merge daily internal staging branch	2026-07-15 13:06:30 -07:00
mateo-berri	0a9ac87538	feat(bedrock_mantle): add GPT-5.6 sol/terra/luna to model cost map ... Register bedrock_mantle/openai.gpt-5.6-{sol,terra,luna} with mode=responses, /v1/responses in supported_endpoints, and use_openai_responses_path so the data-driven gate routes them through BedrockMantleResponsesAPIConfig on the openai/v1 Mantle base path. Without these entries the models fall through to chat-completions emulation, which the Mantle endpoint rejects. Pricing and context window sourced from the AWS Bedrock pricing page and the GPT-5.6 model cards (272K context, OpenAI first-party rates with the 1.1x in-region US uplift, 90% cached-input discount, 1.25x cache write).	2026-07-15 10:45:39 -07:00
yuneng-jiang	6372ca32c1	Revert "chore(ci): sync litellm_internal_staging into daily OSS branch (#33337)" (#33339) ... This reverts commit 90f495f8dc.	2026-07-14 19:32:25 -07:00
yuneng-jiang	90f495f8dc	chore(ci): sync litellm_internal_staging into daily OSS branch (#33337) ... * feat(router): add LLM-based classifier option to complexity router (#32169) * feat(router): add LLM-based classifier option to complexity router Adds classifier_type: "heuristic" | "llm" to complexity_router_config. When set to "llm", the router calls a configured model (e.g. a small model like haiku) via structured output to pick the complexity tier, falling back to the existing regex/keyword scorer on any error, empty response, or unparseable output. * feat(ui): add classifier_type option to complexity router UI, fix edit flow Adds an "Advanced: Classification Method" section to ComplexityRouterConfig with a heuristic/LLM toggle, revealing a classifier model picker and timeout when LLM is selected. Also fixes the auto router edit modal, which never rendered the complexity router UI at all (it only handled the semantic router), and the "Edit Auto Router" button visibility check, which was gated on auto_router_config and never matched complexity router deployments. * fix(router): attribute classifier calls to caller, raise default timeout Forwards the original request's litellm_metadata into the classifier's acompletion call. Without it, the proxy's cost-tracking gate sees no user_api_key/team_id/user_id and silently drops spend logging and budget accounting for every classifier call, letting an authenticated user rack up unaccounted provider spend via repeated requests. Also raises the default classifier timeout from 400ms to 3000ms (400ms undershoots real LLM latency and would silently degrade to the heuristic scorer on most requests) and corrects the module/class docstrings, which still claimed zero external API calls after the llm classifier path was added. * fix(ci): resolve ruff strict-budget and frontend-lint failures - Use PEP 585 generics (dict/tuple/list) in the new aclassify/_classify_with_llm signatures instead of typing.Dict/Tuple/List, and suppress BLE001 on the intentionally broad except in aclassify's fallback path with a reason. - Fix prettier formatting in ComplexityRouterConfig.tsx. - Regenerate eslint-metrics.json (was stale after the classifier UI changes). * fix(ci): regenerate stale eslint-metrics.json * fix(router): strip parent budget reservation from classifier metadata The classifier's internal acompletion call previously forwarded the parent request's full litellm_metadata, including its budget reservation (user_api_key_budget_reservation / user_api_key_auth). That reservation belongs to the routed completion the classifier is deciding on, not to the classifier call itself, so it's now stripped while key/team attribution fields are still forwarded for spend logging. * fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map (#32840) * fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map * test: use apac regional profile for cost-map fallback test since jp now has an entry * fix(responses): preserve reasoning_tokens through chat->responses usage translation (#32837) * fix(responses): preserve reasoning_tokens through chat->responses usage translation Remove the unconditional else-branch that wrote reasoning_tokens=0 whenever completion_tokens_details.reasoning_tokens was None or absent. Also change OutputTokensDetails.reasoning_tokens from int=0 to Optional[int]=None so that re-instantiation without explicit reasoning_tokens no longer silently zeroes out the field, and remove the same hardcoded zero from the mock_responses_api_response initializer. * test(responses): update assertions to match Optional[int] reasoning_tokens default * fix(responses): preserve explicit reasoning_tokens=0 in usage translation Align the reasoning_tokens guard with the is-not-None guards used for text_tokens and image_tokens: a provider-reported zero passes through while an absent value stays omitted. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke * feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule * fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param to accept dict (#32835) * fix(responses-api): raise APIError on in-stream error events; widen ErrorEventError.param - BaseResponsesAPIStreamingIterator._maybe_raise_for_error_event inspects each chunk and raises litellm.APIError for type=error and type=response.failed events so callers see an exception instead of a benign stream chunk - rate_limit* codes map to 429; client error codes (invalid_request_error, context_length_exceeded, etc.) map to 400; all other codes default to 500; raw integer codes are never used as-is as HTTP status codes - ErrorEventError.param widened from Optional[str] to Optional[Union[str, Dict]] to prevent Pydantic ValidationError on dict-typed param payloads silently dropping error events before any type inspection * test(responses-api): add streaming iterator error event tests to CI-covered path * test(responses-api): cover response.failed, dict-error, null-error, and sync iterator paths * test(responses-api): set completion_start_time on mock logging objects for internal staging _process_chunk * fix(responses-api): map insufficient_quota to 429, derive failed-response log status from error code, and record failed-stream usage for spend accounting insufficient_quota moves out of the 400 bucket; OpenAI returns HTTP 429 for it and the non-streaming exception mapping treats 429 as RateLimitError, so the in-stream mapping now agrees _handle_logging_failed_response previously hardcoded APIError(status_code=500), so a rate-limited response.failed was logged to integrations as 500 while the caller saw 429; it now shares the same error-code-to-status mapping via _error_event_fields and _status_code_for_error_code usage carried on a response.failed event is now stashed as combined_usage_object with its computed cost on the logging object before failure handlers run, reusing the mid-stream-interruption spend recovery path (_failure_handler_helper_fn, proxy post_call_failure_hook, _ProxyDBLogger), so failed streams count their billed tokens instead of logging zero cost dedupe: TestMaybeRaiseForErrorEvent in tests/llm_responses_api_testing duplicated tests/test_litellm/responses/test_streaming_iterator_error_events.py, which is the canonical mirrored location and CI-covered via test-unit-responses-caching-types; the duplicate class is removed * fix(responses-api): wrap retriable in-stream errors in MidStreamFallbackError and map error type field to status Mirror chat streaming semantics from _handle_stream_fallback_error: 429 and 5xx in-stream error events now raise MidStreamFallbackError carrying the mapped APIError so the router's FallbackResponsesStreamWrapper triggers mid-stream fallback and cooldown; non-retriable 4xx still raise APIError directly. Status mapping now reads both the OpenAI error type and code fields, so type-classified client errors (e.g. invalid_request_error with code invalid_prompt) map to 400 instead of falling through to 500. * fix(responses-api): accumulate streamed output text so mid-stream fallback continues instead of restarting MidStreamFallbackError was always raised with generated_content="", so the router's stream_with_fallbacks treated every mid-stream error as pre-first-chunk and retried with the original input, streaming duplicated content to clients that had already received partial output. The iterators now accumulate response.output_text.delta text (mirroring chat's response_uptil_now) and pass it as generated_content, letting the router build a continuation input via _build_responses_continuation_input. * test(responses-api): pin in-stream token limit error to raised APIError --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(prometheus): skip budget metric DB lookups when gauges are NoOpMetric (#32834) adds a top-level guard in _increment_remaining_budget_metrics that returns early when all four budget gauges are NoOpMetric (excluded from prometheus_metrics_config), and per-entity guards in each _set_*_budget_metrics_after_api_request helper for partial disabling. eliminates four async DB/cache round-trips per successful LLM request when budget metrics are disabled. Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(anthropic): strip @version suffix in _model_map_lookup_candidates (#32833) vertex_ai/claude-opus-4-8@default (and sibling @default models) were misclassified as non-adaptive because _model_map_lookup_candidates only stripped provider prefixes but never the @<suffix> portion. The lookup produced candidates like ["vertex_ai/claude-opus-4-8@default", "claude-opus-4-8@default"], neither of which exists in model_cost, so _is_adaptive_thinking_model returned False. LiteLLM then sent thinking.type=enabled to a @default Vertex AI endpoint that requires thinking.type=adaptive, resulting in a 400. _strip_version_suffix now removes @<suffix> from each candidate, adding the bare model name (e.g. "claude-opus-4-8") to the lookup chain. Also adds supports_adaptive_thinking: true to the three @default model_cost entries that were missing it as belt-and-suspenders. Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> * fix(datadog): split log batches proactively under intake payload limits (#32860) * fix(datadog): split log batches proactively under intake payload limits * fix(datadog): size intake chunks with exact wire serialization * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface (thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5) get the effort translated to a legacy thinking budget_tokens. Models with no reasoning support have thinking/effort dropped under drop_params. And because adaptive thinking carries no budget while the legacy form must satisfy Anthropic's max_tokens > budget_tokens rule, the translated budget is capped below max_tokens, dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass through untouched. This matters because clients like Claude Code speak native Anthropic /v1/messages and send the adaptive interface unconditionally, regardless of the routed model. The native passthrough previously only capability-gated the OpenAI-style reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so a pre-4.6 model rejected it with "This model does not support the effort parameter" and the request failed. Claude Code already gets drop_params auto-set, so its requests now succeed. * test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests Addresses review feedback on the max_tokens-too-small branch. Previously a thinking-capable model whose max_tokens could not fit the minimum thinking budget had thinking silently dropped regardless of drop_params, while a residual output_config field in the same call still raised when drop_params was off. Gate both consistently on drop_params: raise a clear error (naming max_tokens for the undersized case) when drop_params is off, drop otherwise. Claude Code gets drop_params auto-set, so it still succeeds. Adds tests for the undersized-max_tokens raise, the residual output_config raise, and the no-adaptive-interface passthrough on a non-adaptive model. * fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts The previous raise-when-not-drop_params behavior broke existing bedrock and vertex messages tests: those providers already silently strip unsupported output_config for pre-4.6 models (issue #22797) with no drop_params required, and the shared parent transform raising pre-empted that. It also conflicted with the goal of keeping requests working rather than failing them. Make the reshape silent: translate effort to legacy thinking for thinking-capable models, drop thinking for non-reasoning models, and remove only the consumed effort key from output_config, leaving any residual (e.g. format) for provider subclasses (bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the review note about inconsistent drop_params handling by making every path uniform. Updates the tests to assert the silent behavior and residual output_config preservation. * fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5) Greptile caught a real bug: the early-return guard treated supports_output_config as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises supports_output_config (it accepts output_config.effort) but is not adaptive, so it rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking block raw, reproducing the exact failure the fix is meant to prevent. thinking:{type:adaptive} and output_config.effort are independent capabilities. Only early-return for adaptive-thinking models. For a model that supports output_config.effort but is not adaptive, keep the native effort and drop only the unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code payload now returns 200 instead of 400. Adds regression tests for Opus 4.5 with and without adaptive thinking. * fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models Claude Opus 4.5 advertises supports_output_config but not adaptive thinking, so the early-return guard forwarded thinking.type=adaptive raw and Anthropic rejected it. The guard now only skips true adaptive models; effort-only requests on effort-capable models still pass through untouched. The _map_reasoning_effort call is wrapped to surface unrecognized effort values as a clean 400, matching _translate_reasoning_effort_to_anthropic * fix(anthropic): fall back to legacy thinking when effort level unsupported Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code defaults to xhigh on newer models, so preserving that level raw gets rejected by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model and fall through to the budget translation for unsupported levels * fix(anthropic): keep effort-only requests untouched for provider normalization The xhigh fall-through consumed effort-only requests on effort-capable models, breaking bedrock invoke's own normalization which clamps xhigh to the model's ceiling after the base transform runs (test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict the fall-through to requests that carry adaptive thinking; effort-only requests pass through so provider subclasses keep owning level clamping --------- Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> * test(models): assert capability fields on regional Azure gpt-5.6 entries (#32875) * feat(auto_router): keyword tier overrides and semantic keyword matching for the complexity router Add deterministic keyword-to-tier overrides and optional embedding-based (semantic) keyword matching to the complexity router, and surface both in the Add Auto Router UI behind a Router Type selector: "Auto-Router v2 [Recommended]" (complexity tiers + keyword overrides + semantic matching, the default) and "Semantic Router [to be deprecated]" (the existing utterance-based router, unchanged). Keyword-to-tier overrides resolve to the highest tier matched rather than the first keyword matched, so match order no longer affects the routing decision. Backend: - config: KeywordTierRule model plus keyword_tier_rules, semantic_keyword_matching, embedding_model, and match_threshold on ComplexityRouterConfig, with a validator requiring an embedding model and rules when semantic matching is on - complexity_router: evaluate keyword rules before scoring; lexical matches escalate to the most-severe matched tier (order-independent), and semantic mode reuses LiteLLMRouterEncoder + SemanticRouter to match paraphrases by cosine similarity, falling back to the scorer when nothing matches - model management: clear complexity_routers on cache reload so config edits take effect Frontend: - Add Auto Router tab restores the Router Type radio (Auto-Router v2 recommended by default, Semantic Router still available) and sends keyword_tier_rules plus the semantic settings on the recommended path, instead of flattening keywords into custom_technical_keywords - client-side guard blocks submit when semantic matching is enabled without an embedding model or without any keyword tier rules, mirroring the backend validator - moved the "How Classification Works" explainer below Custom Technical Keywords and above Keyword Tier Overrides - remove the Test Connection action from the recommended flow, which can't build a valid pre-save payload for a router (leaves a TODO for a JSON preview / config test follow-up) Tests cover lexical escalation, semantic matching via the real library with injected embeddings, the semantic config guard, config validation, the reload-clear regression, and the frontend payload builder * fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882) Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache. * feat(team): let org admins reach PATCH /team/{team_id} like POST /team/update Wire the coarse route gate so PATCH /team/{team_id} is reachable by exactly the roles that can call POST /team/update: proxy admins, org admins of the team's own organization, and JWT admins. Regular internal users and view-only proxy admins stay blocked, matching the existing endpoint Because the team id lives in the path rather than the body, the org-context resolver now also reads it from path_team_id for the bare /team/{team_id} route, so an org admin's organization is resolved and injected the same way it already is for POST /team/update. /team/{team_id} is added to management_routes rather than the role-agnostic self_managed_routes; the latter would have opened POST /team/new to any authenticated user through the shared /team/{team_id} path pattern * feat(ui): root the gateway breadcrumb in the AI Gateway selector The AI Gateway select (ViewSwitcher) now sits at the root of the DashboardHeader breadcrumb instead of on the right, so the top bar reads [AI Gateway select] > Page to match the redesign. It keeps the same dropdown, including the Chat / Chat UI options. When no plugins are registered and Chat UI is disabled there is nothing to switch between, so the breadcrumb falls back to the static section crumb rather than rendering a dangling leading separator * fix(complexity_router): build semantic route index once under concurrent cold-start Concurrent first requests each hit asyncio.to_thread to build the SemanticRouter index, firing duplicate embedding calls for the static route utterances. Guard the lazy build with a per-router asyncio.Lock (double-checked) so the index is constructed exactly once regardless of how many callers race in cold. Adds a regression test asserting ten simultaneous cold-start requests build the index the same number of times as a single request, and reworks the fake embedding router to count builds by how often a route utterance is embedded (robust to which embedding path the library uses) while still recording sync-call thread ids for the off-event-loop assertion. * feat(ui): always show the gateway selector with a discoverable Chat entry The AI Gateway selector now always renders at the breadcrumb root, even with no plugins and Chat UI disabled, so the Chat feature stays discoverable. The Chat entry is always listed: clickable when enabled, and disabled with an "Admins can enable in Settings" hint when it is off. Since the selector is now unconditional, the useViewSwitcherVisible hook and the section-crumb fallback added in the previous commit are removed * fix(proxy): guard delete_model router eviction on auto_router/ prefix delete_model popped the auto_routers/complexity_routers registries by the deleted deployment's model_name without checking it was actually an auto_router/* deployment. Deleting a regular DB model that merely shares a name with a config-defined router therefore evicted that router, which add_deployment never restores, leaving it unroutable until a proxy restart. This is the same cross-tenant DoS clear_cache was hardened against; mirror its auto_router/ prefix guard here. Extracts _deployment_name_and_model to read model_name and litellm_params.model from the deployment (delete_deployment returns the raw model_list dict at runtime despite its Deployment annotation), and adds a regression test asserting a same-named config router survives deletion of an unrelated regular model. * refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds * feat(fallback-generalizations): widen adaptive-thinking gate to any claude family at major 5+ * fix(fallback-generalizations): tolerate legacy remote rule schema and keep register_model cache-pricing inheritance * fix(fallback-generalizations): let exact cost-map entries beat capability rules across lookup-candidate ladders * fix(team): bound json merge patch recursion depth apply_json_merge_patch recurses into nested objects, which the repo's recursive_detector code-quality check flags because unbounded recursion over caller-supplied JSON has caused CPU/stack issues before. Cap the recursion at a depth far above any realistic team-metadata shape and reject deeper patches with a ValueError so a pathologically nested body fails closed instead of overflowing the stack, then register the function in the detector's ignore list alongside the other depth-bounded JSON walkers * fix(auth): tolerate request objects without path_params in common_checks The PATCH /team/{team_id} org-context wiring reads request.path_params to resolve the team id from the path. A real Starlette Request always exposes path_params, but common_checks is exercised with lightweight request doubles that don't, which raised AttributeError. Read it defensively so a missing or null path_params falls back to no path team id, matching the "not a bare team route" outcome; real requests are unaffected * fix(mcp): re-register DCR client when proxy origin no longer matches its registered redirect_uri A dynamically registered (RFC 7591) OAuth client persisted onto the MCP server row is bound to the redirect_uri it was first registered with, but that binding was never recorded. After the proxy's public origin changed, every authorize paired the reused client with the new callback and the IdP rejected it permanently. The DCR persist now records redirect_uris alongside the client identity. The admin register path treats a positive mismatch between the recording and the current callback as stale and re-registers a replacement client; rows without a recording (pre-existing installs and admin-configured clients) are grandfathered so upgrades never re-mint client_ids or orphan refresh tokens. The persist also writes client_secret and token_endpoint_auth_method explicitly as None when absent so the credential blob merge cannot pair a re-registered public client with the previous client's secret. Public register routes and non-admin callers keep existing behavior. Closes #32473 * fix(mcp): emit one operator warning per DCR re-registration event The stale-redirect path logged three warnings for a single re-registration: the staleness probe plus the reuse skip in both register_client_with_server and the persist race guard. The reuse-skip message is a mechanical consequence of the probe's decision, so it now logs at debug; the actionable warning that names both bindings and the re-authentication impact is emitted once by _persisted_dcr_redirect_uri_is_stale * ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI * refactor(ui): use TanStack Pacer debounce for the team keys search Replace lodash/debounce in TeamVirtualKeysTable with useDebouncedValue from @tanstack/react-pacer, matching the sibling VirtualKeysTable and PaginatedKeyAliasSelect which already debounce their key-alias search that way. Pacer is already a dependency, so this drops the odd-one-out lodash usage and keeps the search-debounce pattern consistent across the key tables. * fix(fallback-generalizations): cover bare Claude majors in baseline and routing, require claude- prefix in adaptive gate * fix(mcp): strip scheme default port from get_request_base_url netloc * feat(ui): typed openapi-fetch foundation (fetchClient) + first typed caller (useCustomers) (#29884) * feat(ui): add the typed openapi-fetch client (fetchClient) as the dashboard fetch foundation Introduces fetchClient (openapi-fetch) bound to schema.d.ts, used inside ordinary TanStack Query hooks so path/query/body types come from the proxy's OpenAPI spec. A small runtime registry feeds the client the base URL and auth header name (registered by networking) and the session token (published by AuthContext), so call sites carry no token plumbing; auth-header injection and ApiError mapping live in openapi-fetch middleware reusing deriveErrorMessage/ApiError from client.ts, and non-2xx maps to a thrown ApiError so query functions just read .data. The base URL default resolves from NEXT_PUBLIC_BASE_URL so a request still targets the right origin if it fires before networking registers its getter. AuthContext clears accessToken alongside the token on logout so no query fires unauthenticated after the session ends. Foundation only; callers migrate one at a time, each fully typed, in follow-up changes. * feat(ui): migrate useCustomers to the typed fetchClient Converts useCustomers from allEndUsersCall to fetchClient.GET("/customer/list"); the response is typed as LiteLLM_EndUserTable[] from the schema, so the hand-written Customer/CustomersResponse types are deleted. They were also inaccurate (allowed_model_region was string but is "eu"|"us", and a budget_id the table has no field for). No cast; the schema type flows to the one consumer. First caller on the new pattern. * fix(ui): route typed-client errors through the session-expiry handler The typed fetchClient middleware threw ApiError without invoking the handleError side effect that the legacy createApiClient wires via onError, so a migrated caller hitting an expired key no longer triggered the auto-logout. Add an error-handler seam to runtime.ts, register handleError from networking.tsx alongside the base-url/header getters, and call it in the middleware before throwing so both clients behave the same. Regression test asserts the handler fires with the derived message on non-2xx and stays silent on success * fix(ui): point the customers EndUser type at CustomerResponse The /customer/list response model was renamed to CustomerResponse on staging; the merged branch still aliased EndUser to LiteLLM_EndUserTable, so the exported type and its test mock had drifted from what the schema actually returns. CustomerResponse is also the accurate shape (it types allowed_model_region as 'eu' | 'us' and carries budget_id) * chore(ui): refresh eslint-metrics baseline after staging merge The recorded baseline predated the litellm_internal_staging merge, so its no-explicit-any and no-large-inline-object-arg counts were higher than the merged tree actually has. Regenerate via npm run lint:metrics so the gate reflects current reality * refactor(ui): source the typed client token from the session cookie, not AuthContext The typed client read its bearer from a runtime value that AuthContext pushed via setAuthToken, but migrated hooks gate enabled on useAuthorized, which decodes the cookie directly. Two independent derivations of the same cookie with different timing: on first load the query fires (useAuthorized sees the token) before AuthContext's async effect publishes it, so the first request goes out unauthenticated and only succeeds on a React Query retry. Make the token a registered getter like the base-url and header-name getters, reading the same cookie useAuthorized decodes, so the client's token and the gate can't diverge. Revert the AuthContext changes entirely; nothing is pushed from React state anymore. * test(e2e): cover Langfuse logging.yaml P0 logs_spend cells (#32857) * test(e2e): cover Langfuse logging.yaml P0 logs_spend cells Team, user/key, and org-scoped dynamic Langfuse callbacks drive real chat traffic and assert calculatedTotalCost matches StandardLogging response_cost and proxy spend. Also assert tool calls and applied guardrails land on the trace. Missing env or proxy is a hard failure, never a skip * test(e2e): use langfuse_otel callback for Langfuse spend coverage Team and key dynamic logging attach callback_name=langfuse_otel (OTLP to Langfuse) instead of the classic langfuse SDK. Match generations named litellm_request by prompt marker and user_api_key_alias * test(e2e): require Langfuse spend assert; drop AGENTS.md Guardrail path no longer soft-gates logs_spend. Non-stream responses must return positive x-litellm-response-cost; remove tests/e2e/AGENTS.md * test(e2e): fail when Langfuse spend is missing on guardrail path Always run logs_spend assertions for tool_permission; require positive x-litellm-response-cost on non-stream and positive /spend/logs spend * test(e2e): do not fall back to unmatched spend log rows poll_proxy_spend_for_key returns None when response_id or positive-spend filters match nothing, instead of silently using rows[0] * fix(complexity_router): use max aggregation for semantic keyword route scoring SemanticRouter defaults to mean aggregation across a route's utterances. Since each tier's route holds one utterance per configured keyword, a real semantic match on one keyword was averaged together with the tier's other, unrelated keywords and dragged below match_threshold — e.g. a MEDIUM tier with keywords [beep, boop, new york] never fired for a genuine "new york" paraphrase, because mean(sim_to_beep, sim_to_boop, sim_to_new_york) landed well under the threshold even though sim_to_new_york alone cleared it. Pass aggregation="max" so a tier matches if the query is close enough to any one of its keywords, not the average of all of them. Verified against live Voyage embeddings: raw cosine similarity for "new york" vs a paraphrase was 0.54 (above a 0.5 threshold), but the route scored 0.28 under mean aggregation and never matched; max aggregation fixes it. Adds a regression test with a tier holding one matching and two unrelated keywords, asserting the tier still fires; fails without aggregation="max". * refactor(auth): resolve PATCH team org-context from the route template Replace the request.path_params read (and its defensive getattr guard) with the route template. A real Starlette request always exposes path_params, but common_checks runs on lightweight request doubles that don't, so reading it directly forced a getattr workaround that only existed to tolerate those doubles. Instead, match the route template (/team/{team_id}) to identify the RESTful update route and take the team id from the last path segment. This drops the path_params dependency entirely, and because the template distinguishes the PATCH route from its single-segment siblings (/team/new, /team/list, ...), it also avoids a spurious team lookup those routes would otherwise trigger if we matched the resolved path shape alone. * chore(ui): remove eslint-metrics.json lint-count snapshot The eslint-metrics.json snapshot duplicated the violation counts already enforced by eslint-budgets.json. Keeping it current added a CI drift check, a pre-commit regenerate-and-flag step, and a standalone npm run lint:metrics script, none of which caught anything the budget gate did not, yet all of which failed noisily whenever the snapshot went stale. This drops the file and that machinery while leaving eslint-budgets.json as the actual ratchet gate * fix(complexity_router): preserve user_api_key_auth in sub-call metadata Removing user_api_key_auth entirely from classifier/embedding sub-call metadata (as _BUDGET_RESERVATION_METADATA_KEYS previously did) prevented _filter_deployments_by_model_access_groups from scoping those sub-calls to the caller's authorized access groups. An access-group-scoped caller could therefore reach embedding/classifier deployments outside their group. Only strip user_api_key_budget_reservation, which is the actual budget- reservation state that must not reach sub-calls. user_api_key_auth is now kept so access-group filtering works correctly for both the embedding path and the LLM classifier path. * test(e2e): drop vertex from pipecat tool smoke (#32925) Exclude vertex_ai from pipecat tool smoke; raw-ws tool_call_round_trip remains the Vertex source of truth. Also remove the Playwright key models dropdown suite so stage is not blocked by that UI harness * fix(complexity_router): sanitize budget reservation inside forwarded user_api_key_auth * fix(complexity_router): review hardening - blank keywords, router registry eviction, edit-modal controls - config: KeywordTierRule now strips and drops blank/whitespace keywords (a stray "" makes _keyword_matches match every prompt, silently forcing that tier for all traffic); still requires at least one real keyword to remain - frontend build_complexity_router_config: trim keywords and drop rules left empty so an unfilled "Add keyword rule" row no longer ships a rule the backend rejects with a 400 in the heuristic (non-semantic) flow, where the client-side semantic guard doesn't run - proxy clear_cache / delete_model: the auto_router/ prefix also covers quality_router/ and adaptive_router/, so pop the model_name from all four router registries (no-op where absent) instead of only auto/complexity; otherwise a DB quality_router's stale entry made reload raise "already exists" and abort, and adaptive left a leak - frontend ComplexityRouterConfig: only render the Keyword Tier Overrides and Semantic keyword matching sections when their change handlers are provided, so the edit-auto- router modal (which omits them) no longer shows interactive-but-dead controls * fix(anthropic): thread real provider through capability probes instead of pinning anthropic * docs(anthropic): note the two provider params' roles in _map_reasoning_effort * fix(anthropic): override custom_llm_provider in provider config subclasses so capability probes use the right namespace * feat(mcp): admit dcr_bridge oauth_delegate clients via a single envelope bearer * fix(mcp): admit dcr_bridge envelopes under the live key, not a frozen identity The bridge envelope sealed only user_id/server_id, and admission fabricated a UserAPIKeyAuth(user_id=...) with no object_permission, team_id, org_id, or key identity. Downstream MCP permission checks read the missing restrictions as unrestricted, so a caller holding a valid envelope for a restricted key could reach tools and servers that key was never granted, and a revoked key kept working until the envelope expired. Bind the hashed authorizing key into the envelope identity and reload the live UserAPIKeyAuth by it at admission via get_key_object, failing closed with a 401 when the key is missing, blocked, or expired. Authorization is resolved fresh per request instead of frozen at mint time, so current key/team/org and tool restrictions plus revocation are enforced. * fix(mcp): enforce team block and alias-priority token injection on bridge admission Two follow-ups on the envelope admission arm flagged in review. Team revocation bypass: _reload_admitted_key checked only the key's own blocked/expires, so blocking a key's team left every envelope minted under it live until expiry. Reload the team and reject a blocked team, mirroring common_checks, so a team block revokes its envelopes immediately. Caller-overridable upstream token: egress resolves the per-server auth header alias-first, but injection keyed under server_name, so for a server with a distinct alias a caller-forwarded x-mcp-{alias}-authorization sat at the higher-priority slot and paired the admitted identity with an attacker's upstream credential. Inject under alias-first so the sealed token owns the slot egress resolves. * fix(mcp): route bridge admission through the centralized policy gate and mirror the SCIM owner check * fix(mcp): import assert_never from typing_extensions for Python 3.10 * fix(mcp): surface real status from bridge admission policy gate instead of flattening to 401 Over-budget rendered 401 (should be 429), model-access and other typed failures collapsed to 401, and a transient DB outage was masked as an auth error. Mirror UserAPIKeyAuthExceptionHandler: budget maps to 429, a sub-check's own HTTPException/ProxyException keeps its status, a DB outage is a retryable 503, and only a genuinely unresolvable failure stays the fail-closed 401. * fix(rate-limit-v3): populate x-ratelimit-* remaining/limit values in standard_logging_object for streaming (LIT-4333) (#32711) Streaming requests return from common_request_processing before async_post_call_success_hook runs, so response._hidden_params.additional_headers never gets the v3 x-ratelimit-{descriptor_key}-{remaining|limit}-{rate_limit_type} entries. Prometheus / logging callbacks that read those values from standard_logging_object.hidden_params.additional_headers then see nothing; combined with the pre-existing gap that Prometheus reads from that same slot (LIT-2577 / PR #28816), per-key remaining RPM/TPM cannot be monitored for streaming traffic at all. Fix in three parts: - Stash the pre-call RateLimitResponse in the metadata channels the async success-logging callback inherits, alongside the existing top-level entry the non-streaming path reads. - Add async_logging_hook to the v3 handler. It fires in a distinct earlier loop inside async_success_handler (all callbacks' async_logging_hook complete before any async_log_success_event starts), so mirroring the pre-call snapshot into standard_logging_object.hidden_params.additional_headers and response._hidden_params.additional_headers here guarantees every downstream success callback sees the values regardless of registration order. Non-streaming keeps the existing async_post_call_success_hook write and this hook re-populates the same values idempotently. - Extract the shared `_merge_ratelimit_statuses_into_additional_headers` helper the non-streaming path already had inlined so both callsites emit the identical key shape. * fix(proxy): skip None model_name in clear_cache router eviction set * fix(mcp): map a DB outage during bridge key reload to a retryable 503 get_key_object's raw transport error propagated uncaught out of _reload_admitted_key as an opaque 500; classify it via the shared _raise_503_if_db_unavailable helper (also used by the live-policy gate) so a database outage is a retryable 503, while a key-not-found ProxyException stays the fail-closed 401. * test: remove live OpenAI fine-tuning job-creation test blocked by platform wind-down (#32933) OpenAI is winding down self-serve fine-tuning and the org can no longer create fine-tuning jobs (403 training_not_available; the CI key surfaces it as a 500 server_error), so test_create_fine_tune_jobs_async fails on every batches_testing run since 2026-07-11 and reruns never clear it. The request contract stays covered by the mocked create/list/cancel/ retrieve tests in the same file, and the deleted test's unique standard_logging_object assertions now run inside test_mock_openai_create_fine_tune_job. * refactor(anthropic): consolidate the provider fallback into a _resolved_provider property * feat: add lite auth print-token for Claude Code apiKeyHelper support (#32846) * feat: add silent CLI token refresh for apiKeyHelper support lite auth print-token prints a valid proxy credential for use as Claude Code's apiKeyHelper, transparently refreshing it first if the cached JWT is stale. This unblocks MDM-managed apiKeyHelper deployments (managed via `lite auth print-token`) that need silent mid-session credential rotation without restarting the client. Refresh capability is backed by a virtual key minted with an empty model list and cli_refresh metadata, kept strictly separate from the actual (short-lived, real-model-scoped) call credential -- so a leak of the credential that flows through every LLM request and subprocess env var can't also self-renew. The refresh flow is single-use: /sso/cli/refresh mints a fresh JWT + refresh token pair and blocks the presented refresh token immediately, so a replay can't mint a second pair from it. Server: /sso/cli/refresh (rotate) and /sso/cli/logout (revoke) endpoints. lite login now also stores a refresh token; lite logout revokes it server-side instead of only clearing the local file. * fix: allow non-admin users to hit CLI refresh routes; resolve apiKeyHelper base_url from token.json Found via a live end-to-end test against a real proxy + real Claude Code session: /sso/cli/refresh and /sso/cli/logout were unreachable for any non-proxy-admin caller, since Depends(user_api_key_auth) pulls in a route-RBAC gate that 403s any route not on an explicit allowlist. That made the feature unusable for actual end users, who authenticate as internal_user. Add both routes to internal_user_routes; the handlers already do their own fine-grained check (metadata.cli_refresh) same as /key/block does today. Also: `lite auth print-token` required an explicit --base-url/ LITELLM_PROXY_URL matching the stored token's origin, defaulting to localhost:4000 otherwise. But apiKeyHelper is configured bare (no flags), so this always mismatched a real deployment. Track whether --base-url was explicitly passed (via click's ParameterSource) and, if not, resolve the server from token.json directly instead of the CLI default. * test: mock refresh-token minting in test_cli_poll_key_tolerates_missing_user_row Landed on litellm_internal_staging after this branch's refresh-key minting change; needs the same mock as the other cli_poll_key tests since minting now runs unconditionally whenever a JWT is generated. * fix(ci): update test_cli_auth.py for refresh_token contract, regenerate schema.d.ts _poll_for_authentication now always includes "refresh_token" in its returned dict, and _handle_team_selection_during_polling returns a dict instead of a bare JWT string -- test_cli_auth.py predates this branch's refresh-token work and still asserted the old shapes. schema.d.ts regenerated via `npm run gen:api` to pick up the new /sso/cli/refresh and /sso/cli/logout routes (plus unrelated drift from other PRs merged since it was last generated). * fix(ci): apply CI's own schema.d.ts diff (enterprise routes I can't generate locally) Local `npm run gen:api` only sees OSS routes -- this machine's litellm_enterprise editable install points at a now-deleted temp directory, so it silently drops enterprise-only routes from the spec. Applied the exact diff CI's own generation produced instead of re-running the generator locally. * fix: close refresh-token race, fail closed on DB down, fix logout base_url Addresses Greptile review findings on the CLI refresh-token PR: - cli_refresh_token minted a new JWT + refresh token BEFORE blocking the presented one. Two concurrent requests bearing the same refresh token could both pass auth and both mint fresh pairs, yielding four live credentials from one consumed token. Now the presented token is consumed atomically first via update_many (only succeeding if it flips blocked from False/None to True); the loser gets count=0 and is rejected before anything is minted. - When prisma_client is None, refresh silently returned a new JWT without ever being able to mark the presented token consumed, leaving it valid indefinitely. Now fails closed with a 500 instead. - `lite logout` sent its revocation POST to ctx.obj["base_url"], which defaults to localhost:4000 when --base-url isn't passed -- the same bug print_token had before the base_url_explicit fix, just missed here. Now resolves the same way: trust the stored token's origin unless the caller explicitly overrode --base-url. * fix(ci): satisfy ruff format and narrow token_data type in logout * fix(security): never trust refresh-token metadata for authorization Addresses a real privilege-escalation path Veria flagged: cli_refresh_token read team_id, team_alias, and max_budget straight off the presented token's own metadata and used them to authorize the new JWT. Since any authenticated user can self-mint a virtual key with arbitrary metadata via the ordinary /key/generate endpoint, a self-forged key with {"cli_refresh": true, "team_id": "<any-team>", "max_budget": 999999999} would sail through _require_cli_refresh_token's only check (metadata.cli_refresh == True) and get a JWT scoped to a team the caller never belonged to, with a budget it never had -- full cross-team / budget bypass, and a removed team member could keep refreshing team-scoped sessions indefinitely. Metadata's team_id is now treated as an untrusted UX hint only: honored solely if the CALLER (identified by the authenticated key's own user_id, not client input) is a current member per a fresh get_user_object lookup. team_alias and max_budget are never read back from metadata at all -- team_alias comes from a live get_team_object lookup and max_budget is recomputed with the exact same capping logic the initial SSO login poll uses. _mint_cli_refresh_token no longer accepts or stores team_alias/max_budget, only the team_id hint. Added regression tests proving: a forged/stale team_id is dropped (falls back to no team, not silently honored), and a forged max_budget in metadata never reaches the issued JWT. * fix(ci): catch HTTPException specifically instead of bare Exception (BLE001) * fix: un-consume refresh token if minting the replacement fails Greptile flagged a real reliability gap: cli_refresh_token blocks the presented token atomically, then does several more DB calls before returning a replacement (user lookup, team lookup, JWT mint, new refresh-key mint). Since this endpoint exists specifically for fully unattended apiKeyHelper operation, a single transient failure in that window (DB hiccup, etc.) permanently stranded the user: their old token was already dead and no new one was issued, with no recovery path short of a full interactive browser re-login. Wrap that window in try/except; on any failure, best-effort revert the consumed token back to usable (blocked=False) before re-raising, so a retry can succeed. Standard compensating-action pattern since generate_key_helper_fn doesn't take an injectable transaction, so wrapping the whole thing in a real DB transaction isn't practical here. * fix(security): refresh key had unrestricted model access, not none Critical bug: _mint_cli_refresh_token used models=[] intending "no LLM access", but that's backwards in this codebase. Per _check_model_access_helper: `len(filtered_models) == 0 and len(models) == 0` -> all_model_access = True. An empty models list on a key with no team_id means UNRESTRICTED access to every model, not zero access. The CLI refresh token -- meant to be usable for nothing but silently exchanging itself for a new JWT -- was actually a fully unrestricted API key for its entire 90-day lifetime, completely undermining the whole point of keeping it separate from the short-lived call credential. Fixed with two independent layers: allowed_routes hard-restricts the key to exactly /sso/cli/refresh and /sso/cli/logout (the real enforced boundary, checked in the shared user_api_key_auth dependency for every route); models is set to an unmatchable sentinel string as defense-in-depth in case any code path only consults the models field. Added an end-to-end regression test that exercises the actual model-access-control function against a key shaped like the minted refresh token, rather than only asserting on what arguments were passed to the key-generation call -- the latter kind of test is exactly what let the original bug ship, since asserting `models == []` is equally consistent with "no access" and "unrestricted access" without checking what the access-control code actually does with that shape. Also: the compensating-rollback added for reliability un-blocked a consumed refresh token even when the underlying user no longer exists. That's a permanent, intentional rejection, not a transient failure -- un-blocking it would let a stale refresh token become valid again for a different account if the user_id is ever reused/re-registered. Moved the user-existence check outside the rollback-on-failure block so it stays permanently blocked. * refactor: rotate CLI refresh tokens via regenerate_key_fn instead of hand-rolled consume/rollback The refresh token is already a plain litellm virtual key, so rotation can delegate to the same atomic DB update /key/regenerate uses instead of a bespoke update_many + compensating-rollback dance. This makes silent CLI refresh an Enterprise feature, same as regular key regeneration. * refactor: replace CLI stateless JWT + refresh-key pair with one self-rotating virtual key The CLI previously minted two credentials on login: a stateless self-signed JWT for LLM calls, and a separate DB-backed refresh-only key (scoped away from ever calling an LLM) just to authorize minting a new JWT. Collapse this into a single real virtual key, used directly as the LLM bearer token and re-presented to /sso/cli/refresh to rotate its own secret in place. This also means the CLI session key now shows up in the Admin UI's Keys page and can be revoked/regenerated like any other key, rather than being an invisible, unmanageable stateless token. * refactor: drop silent CLI refresh, key just expires and requires re-login /sso/cli/refresh only ever benefited Enterprise deployments (regenerate_key_fn's gate), while everyone else already fell through to "re-run lite login" on failure. Cut the endpoint, the rotation logic, and the client-side refresh path entirely; print-token now just prints the cached key until it hits its LITELLM_CLI_JWT_EXPIRATION_HOURS duration, then fails fast telling the user to log in again. Session key itself is unaffected: still a real, revocable virtual key visible in the Keys UI, `lite logout` still revokes it directly. * fix(ci): regenerate schema.d.ts after removing /sso/cli/refresh route * revert: go back to stateless JWT, keep only lite auth print-token The virtual-key redesign (revocable, Keys-UI-visible credential) wasn't needed just to support print-token, and cost real server-side surface (a mint path, a logout-revoke endpoint, migrated tests/docs) for a property this repo doesn't need yet. Reverting cli_poll_key/_types.py/schema.d.ts back to the original stateless-JWT design; the only durable addition from this whole effort is `lite auth print-token` (reads the cached credential, prints it while fresh, fails with a clear message once it's past LITELLM_CLI_JWT_EXPIRATION_HOURS) plus the base_url_explicit plumbing it needs. `lite logout` goes back to clearing the local file only, since a stateless JWT can't be revoked server-side. * refactor: move CLI token freshness check to cli_token_utils, drop unnecessary renames Addresses review: the freshness check is a pure token-shape/timestamp util, not command logic, so it belongs alongside the other SDK-level CLI token helpers (load_cli_token, get_litellm_gateway_api_key) rather than in commands/auth.py. Also reverted a few incidental jwt_token/ session_key variable and string renames that weren't load-bearing. * fix(mcp): run the route gate on bridge admission so allowed_routes are enforced The envelope arm reloaded the identity and ran _run_centralized_common_checks but skipped RouteChecks.should_call_route, which the standard pipeline runs between the builder and common_checks. Because the centralized checks treat MCP as an inference route and never re-check allowed_routes, a key barred from MCP routes could mint an envelope at the token endpoint (not itself an MCP route) and replay it against MCP. Run the route gate before admitting, and clear the request-scoped budget_reservation, matching the wrapper's sequence; a disallowed route now surfaces the gate's own 403. * fix(proxy): reserve budget for tiered pricing Ensure tier-only models reserve their estimated request cost so concurrent requests cannot bypass exhausted budgets. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): bill tier-only deployments instead of $0 Route cost calculation to the deployment's router_model_id entry when it carries tiered_pricing but no flat per-token rate, so models like dashscope/qwen3.7-plus are billed via their tier table rather than the pricing-stripped shared alias. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(ui): convert activity metrics charts to shadcn/recharts (#32726) * refactor(ui): convert activity metrics charts to shadcn/recharts Swap the seven tremor AreaChart/BarChart sites in activity_metrics.tsx to the shared shadcn/recharts wrappers and switch CustomLegend/CustomTooltip to the ported versions in shared/charts. Chart props, colors, formatters, and legend behavior are unchanged; tests now assert on real recharts SVG output instead of tremor mocks. * fix(ui): restore tremor No data placeholder for empty AreaChart data * test(ui): scope activity metrics chart assertions to card titles instead of render order * feat(ui): extend topnav border across the sidebar header (#32920) Pin the sidebar header to the same 56px height as the dashboard topnav and give it a matching bottom border, so the two borders sit flush and read as one continuous line. Revert to auto height when the rail is collapsed so the stacked logo and toggle are not clipped. * fix(cost): coerce string tiered-pricing costs and share tier helper YAML-parsed tier costs can arrive as strings (e.g. "4e-07"), which broke arithmetic in the graduated tiered-pricing calculation. Coerce per-token costs to float in both the in-range and remaining-tokens paths. Move the tiered-cost helper out of the Dashscope module into a provider-neutral home so the proxy budget reservation no longer depends on a provider-specific module. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(anthropic): clarify the Opus 4.5 branch in adaptive-effort translation Add an inline comment explaining that the effort-capable non-adaptive branch in _translate_adaptive_effort_for_non_adaptive_model exists for models like Claude Opus 4.5 that accept output_config.effort but reject adaptive thinking, and why effort-only requests pass through while adaptive requests with an unsupported effort level fall through to the legacy translation. * fix(anthropic): translate raw adaptive thinking for chat completions on pre-4.6 models Clients that pass thinking={"type": "adaptive"} directly (not via the reasoning_effort alias) on the /chat/completions interface had it forwarded unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the translation already applied on the native /v1/messages passthrough (#32867): translate to legacy thinking={type: enabled, budget_tokens}, capped below max_tokens, dropping thinking when max_tokens can't fit even the minimum budget. Hoists the shared budget-capping helper onto AnthropicConfig so both paths use one implementation. * fix(proxy): reserve tiered budget all-or-nothing across all deployments Alibaba Model Studio (Dashscope) tiered pricing is all-or-nothing: the tier is selected by a request's total input tokens and every token, input and output, is billed at that one tier's rate. The reservation path used graduated slicing and, worse, picked the output tier from the output-token count, so a long-context request with a large output allowance reserved far less than the provider charges and could slip past a depleted budget. Select the tier from input tokens and apply its rates to all input and output tokens. Reservation also read tiered pricing from only the first deployment in a model group. A caller could hit an alias whose cheaper deployment was listed first and exceed the budget once routed to a costlier sibling. Estimate against every eligible deployment's pricing and reserve the maximum. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(complexity_router): log the cause of each routing decision The complexity router's info log didn't say what drove a routing decision. Literal and semantic keyword matches logged an identical "keyword rule fired" line (no way to tell which mechanism fired), and the scorer's line carried no consistent marker tying it to the same question. Emit one greppable line per decision naming the cause: literal_keyword_match, semantic_keyword_match, or complexity_scorer. The hook already knows which ran (the config's semantic_keyword_matching flag distinguishes lexical from semantic; the override-vs-scorer branch distinguishes keyword match from scorer), so this is label-only: no behavior change, no new types, no added latency. Adds regression tests asserting each decision path logs its cause; they fail if a label is swapped or the cause= marker is dropped. * feat(ui): add redesigned sidebar account menu (#32931) * feat(ui): add redesigned sidebar account menu Introduce SidebarAccountMenu, a sidebar-only account/logout menu built on shadcn Popover/Switch/Badge/Separator/Button, and wire it into leftnav in place of the shared UserDropdown. The panel has a LiteLLM header with the bouncing moon and a clickable version tag, Tier/Role/Email/User ID rows with copy actions, the five display toggles, and Logout. UserDropdown is left untouched so the control-plane / chat navbar keeps its existing menu. The version tag links to the same release notes page as the navbar tag, and the bouncing icon reuses the existing header animation gated by the Hide Bouncing Icon toggle. * test(ui): point account-menu e2e specs at the migrated sidebar menu The sidebar account menu moved from an antd Dropdown to a Base UI popover (SidebarAccountMenu), so the login, logout, proxy-logout-url, and internal user identity specs were still waiting on antd-era locators (.ant-dropdown, the popupRender wrapper class, the user-dropdown-panel test id, and a menuitem-role Logout). Point them at the new panel test id (sidebar-account-menu-panel) and the button-role Logout instead. The logout behavior is unchanged since both menus call the same useLogout handler. * fix(bedrock-converse): translate adaptive thinking for pre-4.6 models Follow-up to #32867 (native /v1/messages) and the /chat/completions commit earlier on this branch, extending the same adaptive-thinking translation to the Bedrock Converse path. Clients like Claude Code send thinking={type: "adaptive"} on every request. When routed via Bedrock Converse to pre-4.6 models (claude-haiku-4-5, claude-sonnet-4-5), this was forwarded as-is and rejected by the model. Mirrors the translation already applied on the /chat/completions and /v1/messages paths: map to legacy thinking={type: enabled, budget_tokens}, capped below max_tokens. Also fixes the missing custom_llm_provider arg in the chat completions path's call to AnthropicConfig._map_reasoning_effort. * fix(mcp): run proxy-wide pre-DB gates on bridge envelope admission The envelope arm bypasses user_api_key_auth, so it never ran pre_db_read_auth_checks (request-size and body-safety limits, the IP allowlist, and the general_settings route allowlist) that the normal MCP admission path runs before any key lookup. A caller blocked by IP or a disallowed proxy route could be admitted through an envelope where the same principal on the normal path is rejected. Run those gates before the envelope crypto, mirroring the pipeline's pre-DB ordering; a blocked IP or route surfaces its own 403. * fix(anthropic): pass resolved provider to adaptive-thinking check The rebase onto staging changed _is_adaptive_thinking_model to require custom_llm_provider (no default), so the one-arg call in the raw adaptive thinking branch raised TypeError at runtime for any /chat/completions caller sending thinking={type: adaptive}. Use self._resolved_provider, matching the reasoning_effort branch just below. Caught by Greptile. * test(bedrock-converse): cover adaptive-thinking drop when max_tokens too small Adds the regression test for the warning-drop branch in the Converse adaptive-thinking translation, mirroring the chat completions path's test_raw_adaptive_thinking_dropped_when_max_tokens_too_small. * fix(guardrails): filter Add-Guardrail mode dropdown per provider (#32712) * fix(guardrails): filter Add-Guardrail mode dropdown per provider The GET /guardrails/ui/add_guardrail_settings endpoint returned every GuardrailEventHooks value in one flat supported_modes list, so the Admin UI rendered pre_mcp_call as a selectable Mode for every guardrail. Saving Content Filter or Tool Permission with pre_mcp_call then failed with a 400 because those guardrails' server-side supported_event_hooks list excludes it. Expose each guardrail's supported hooks as a get_supported_event_hooks classmethod on CustomGuardrail (mirrors the existing get_config_model pattern) and have the endpoint iterate guardrail_class_registry to build a supported_modes_by_provider map. The UI Mode dropdown filters by that map when the selected provider is known and falls back to the global list otherwise. __init__ now sources its own supported_event_hooks list from the classmethod so the two sides can't drift. Also register BedrockGuardrail, ToolPermissionGuardrail, lakera, lakera_v2, and presidio in guardrail_class_registry so they participate in the map (they were previously only in guardrail_initializer_registry and had no class-registry entry). Behavior change: guardrails that previously had no supported_event_hooks declared (aim, javelin, azure/text_moderation, cato_networks, crowdstrike_aidr, headroom, hiddenlayer, lasso, noma, onyx, prompt_security, qualifire, repelloai, zscaler_ai_guard, aporia_ai, lakera_ai, lakera_ai_v2, mcp_jwt_signer, model_armor, presidio) now validate the configured mode at instantiation. Existing configs where the mode was silently a no-op will fail at proxy startup with a clear validation error rather than running as a broken guardrail. Resolves LIT-4226 * fix(guardrails): add LITELLM_STRICT_GUARDRAIL_MODES escape hatch, preserve current mode in edit form Address Greptile P1 (startup break) and P2 (edit form UX): LITELLM_STRICT_GUARDRAIL_MODES defaults to true (raise on unsupported event_hook, unchanged behavior for the guardrails validated pre-PR). Setting it to false logs a warning and continues, giving deployments an opt-out while they fix configs that now surface as errors instead of silently no-op'ing. Regression test covers both modes. Edit form now surfaces the currently-saved mode even when it is not in the filtered per-provider list, so a legacy row (e.g. content_filter saved with pre_mcp_call before this fix) no longer disappears from the dropdown; the option renders with a 'not supported by <provider>' note so the user knows to pick another. * fix(guardrails): correct audited hook lists, prune stale modes on provider switch, clean form lint Audited every get_supported_event_hooks classmethod against the hooks each guardrail's own tests exercise and its handler methods. Five were too narrow and their tests caught it in CI: rubrik gains pre_call, presidio gains during_call and pre_mcp_call, prompt_security, onyx and qualifire gain during_call. The remaining classes match either their original __init__ declarations or their exercised modes exactly. Cursor review fixes: the Add form now drops selected modes the new provider does not support when the user switches providers, so a pre_mcp_call selection cannot ride along into a provider that rejects it at save; the edit form handles list-shaped stored modes instead of treating mode as always a string. Extracted shared toModeArray and getSupportedModesForProvider helpers into guardrail_info_helpers so both forms use one implementation, typed the remaining any usages in both forms, removed nested ternaries, and committed the ratcheted-down eslint metrics and pruned suppressions * fix(proxy): reserve tiered output at the higher reasoning rate Some tiered Dashscope models price reasoning output above standard output (output_cost_per_reasoning_token > output_cost_per_token). The reservation charged all output at the standard rate, so a reasoning-heavy request reserved too little and concurrent calls could exceed the budget before reconciliation. The reasoning share is unknown before the request runs, so reserve every output token at the higher of the two configured rates, for both tiered and flat pricing. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(ui): convert user agent and per-user usage charts to shadcn/recharts (#32725) * refactor(ui): convert user agent and per-user usage charts to shadcn/recharts Swap the tremor BarChart import for the shared shadcn/recharts wrapper in user_agent_activity.tsx (DAU/WAU/MAU charts) and per_user_usage.tsx (usage distribution histogram). All chart props are unchanged; the wrapper exposes the same tremor prop surface with matching defaults. Extend user_agent_activity.test.tsx and add per_user_usage.test.tsx with parity assertions on the real recharts SVG output: bar series per category, stacked x positions, resolved fill colors, axis bucket labels, legend text, and value formatter output on axis ticks. Remove the dead ResizeObserver polyfill in user_agent_activity.test.tsx now that the scoped global mock in tests/setupTests.ts renders charts, which also lowers the no-explicit-any metric by one. * test(ui): harden bar x-position parsing against recharts path format * feat(ui): working Test Connection for the complexity auto router The consolidated auto-router tab dropped the Test Connection button because the shared prepareModelAddRequest helper returns an empty array for an auto router (it has no model_mappings), so the caller crashed destructuring result[0].litellmParamsObj. That is the crash in #31590 and the open PR #31794. #31794 only silenced the crash by pointing the test at auto_router/complexity_router, which is not a provider model, so the /health/test_connection health check (a real litellm.ahealth_check completion) would still error. Bring the button back and make it meaningful: an auto router dispatches to saved model groups, so Test Connection now probes those directly. It builds a deduped target list from the configured tiers (tiers sharing a model group collapse to one probe) plus the embedding model when semantic keyword matching is on, then runs a live /health/test_connection against each and shows per-target pass/fail. This never touches prepareModelAddRequest, so the original destructure crash cannot recur. Scope is the recommended complexity router only; the to-be-deprecated semantic router is untouched. No backend changes. Supersedes #31794. Resolves #31590. * refactor(ui): extract a shared CopyButton and fix the sidebar copy confirmation (#32945) * fix(ui): show sidebar copy confirmation only on a successful write The sidebar account menu's copy button switched to the checkmark synchronously, before the clipboard write settled, so it confirmed a copy that never happened when navigator.clipboard was undefined on non-secure origins or when writeText rejected. The handler now guards navigator.clipboard, awaits the write, and flips to the checkmark only on success Also updates the header accent emoji in the same menu * refactor(ui): extract a shared CopyButton for the sidebar account menu The copy-icon-to-checkmark pattern was hand-rolled in several places, including the sidebar account menu whose private copy button held the false-confirmation bug. Extract a single canonical CopyButton into components/shared, built on the Button primitive with a guarded and awaited clipboard write so the checkmark appears only on a real success, and have SidebarAccountMenu consume it The success and failure-mode coverage now lives in the shared component's own test; the sidebar test keeps one case asserting the email row is wired to it * fix(ui): probe auto-router tiers via real proxy routing, not /health/test_connection Live testing showed the first cut was broken: /health/test_connection merges {...configParams, ...requestParams}, so passing the public model_group name as the request model overrode the resolved provider model and every tier failed with "LLM Provider NOT provided". The frontend only has the public group name, not the underlying litellm_params, so it cannot build the request that endpoint needs. Switch to testing each model group the way production actually routes it: send a minimal request to /v1/chat/completions (or /v1/embeddings for the embedding model) by public group name through the shared apiClient. The router resolves the group, credentials, and provider itself, so a green row means the tier is genuinely reachable. Verified live: voyage embedding returns 200, a tier with a bad key returns the real provider auth error. Also address Greptile feedback: rows now update progressively as each probe settles instead of all at once, and TIER_ORDER is derived through a `satisfies Record<keyof ComplexityTiers, null>` guard so adding a tier without listing it is a compile error. * fix(completion): forward aws credential kwargs into litellm_params so the responses bridge keeps WIF auth Chat-completions requests to responses-only Bedrock Mantle models are bridged to the Responses API, but completion() forwarded only aws_bedrock_project_id into get_litellm_params, so aws_role_name, aws_web_identity_token, aws_session_name and the other SigV4 credential kwargs never reached sign_request and botocore fell back to the default credential chain ("Bedrock Mantle auth failed: no Bearer token and no usable AWS credentials"). Forward the whole AWS credential kwarg family, extracted from the OPTIONAL_KWARGS_KEYS set get_litellm_params already supports. * refactor(ui): convert entity usage and usage page charts to shadcn/recharts (#32729) * refactor(ui): convert entity usage and usage page charts to shadcn/recharts Swap the tremor BarChart/DonutChart render sites in EntityUsage, SpendByProvider, TopKeyView, TopModelView, KeyModelUsageView and UsagePageView to the shared shadcn/recharts wrappers. Convert the two sole-chart Daily Spend cards and the KeyModelUsageView card to the shadcn Card primitives. Close the donut parity gap with strictly additive optional DonutChart props: showLabel/label render a center total (tremor showed valueFormatter(sum) by default) and startAngle/endAngle forward to the Pie so both provider donuts keep tremor's clockwise-from-12 layout. Defaults preserve the previous wrapper behavior. DailyData and two site-local row types move from interface to type alias so they satisfy the wrappers' Record<string, unknown> constraint; interfaces lack implicit index signatures. Tests now assert on real recharts output: bar/sector counts, cyan fills, axis labels, donut center totals, and the TopKeyView bar-click drill-down into the key info modal. The dead tremor chart mocks in UsagePageView.test.tsx are removed and lint metrics/suppressions are regenerated for the dropped tremor imports. * fix(ui): compute donut center label only when shown and assert Model Usage renders as a card title * refactor(e2e): bucket rate limits, budgets, and spend tracking under quota_management * fix(e2e): name the route the spend_calculate registry cell actually exercises * refactor(e2e): move budgets and spend_tracking suites under quota_management * test(e2e): cover key rpm/tpm rate limiting, window reset, and pacing headers * test(e2e): assert the tpm block at its exact token crossing instead of a call-count heuristic * test(e2e): fail the rpm reset test when the limiter resets early * test(e2e): name the tpm window deadline's latency margin * refactor(e2e): model the tpm spend loop's two outcomes as values * test(e2e): source the ratelimit suite's model from E2E_CHEAP_ANTHROPIC_MODEL * feat(guardrails): add pre_mcp_call support to Content Filter (#32936) * feat(guardrails): add pre_mcp_call support to Content Filter * test(guardrails): cover canonical MCP key gate under pre_mcp_call mode * fix(guardrails): scan MCP arguments per value and gate mixed-mode scans by call type * fix(guardrails): cap MCP argument scan depth and register the walker with the recursion detector * test(guardrails): update LIT-4226 UI settings tests for content filter pre_mcp_call support * fix(guardrails): use builtin generics in MCP scan annotations to satisfy strict-rule budget Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * fix(bedrock): allow bedrock-mantle:CreateInference in the web identity session policy * fix(ui): drop max_tokens from the auto-router connection probe max_tokens=1 makes reasoning models (o1/o3/...) return a 400 "max_tokens reached" because reasoning tokens count against the cap, so a reachable reasoning tier showed a false failure in Test Connection. Live-verified: o3 400s with the cap and succeeds without it. Extract the request shape into a pure buildModelGroupTestRequest and cover it with a test asserting the chat body carries no max_tokens (or max_completion_tokens), so this regression is caught in unit tests instead of only against a live reasoning model. * test(main): assert the responses bridge forwards static aws keys as well as web identity params * bump: litellm-proxy-extras 0.4.75 -> 0.4.76 (#32957) * feat(proxy): add expires filter to GET /key/list (#32953) * feat(proxy): add expires filter to GET /key/list Add an opt-in expires query param to GET /key/list so callers can fetch only expired or only active keys without paginating every page and filtering client-side. 'expired' matches keys whose expires is in the past (NULL expires excluded); 'active' matches keys that never expire or expire in the future. Omitting the param preserves existing behavior for every caller. An unrecognized value returns HTTP 400 rather than silently returning all keys. The filter is pushed to the database via the existing Prisma where builder so callers avoid pulling the full key table into application memory. Resolves LIT-3387 * refactor(proxy): declare VALID_EXPIRES_FILTER_VALUES before its first use * refactor(ui): colocate the usage view, keeping the shared usage components (#32952) Split for the usage (UsagePage) segment. Most of the folder is the usage page's own view, but four pieces are reused elsewhere and stay in @/components/UsagePage: TopKeyView (old-usage), KeyModelUsageView and value_formatters (activity_metrics), and the shared types (activity_metrics, chartUtils). The other 21 files move into usage/_components, preserving the folder structure. The external consumers import only the retained files, so they are untouched. The moved files' imports of the retained files become @/components/UsagePage paths, other escaping relative imports are absolutized, and lint suppressions are re-keyed for moved files only. No behavior change. * chore: update Next.js build artifacts (2026-07-11 23:35 UTC, node v20.20.2) (#32960) * test(e2e): cover model-aware mid-conversation system handling on Bedrock Invoke /v1/messages * docs(github): add QA runbook section to the PR template * docs(github): scope the QA runbook to tests/e2e edits and add example checklists * docs(github): shape QA runbook examples as node id plus behavior bullets * refactor(ui): convert projects page chart to shadcn/recharts (#32722) * fix(xecguard): use StandardLoggingGuardrailInformation in logging hook (#32911) XecGuard's async_logging_hook wrote a bare dict to standard_logging_object["guardrail_information"] while the typed contract is Optional[List[StandardLoggingGuardrailInformation]]. Readers that iterated the field walked dict keys, raised on info.get, or silently dropped the entry from guardrail usage tracking and spend-log writes Construct the typed entry and append it to the existing list or create a new one, matching the shared helper pattern. Record the configured guardrail name instead of a hardcoded "xecguard" and pass the GuardrailEventHooks enum for guardrail_mode * feat(ui): adopt openapi-react-query ($api) and convert useCustomers (#32949) * feat(ui): adopt openapi-react-query and convert useCustomers to $api Add openapi-react-query and expose $api = createQueryClient(fetchClient) alongside fetchClient. Rewrite useCustomers as $api.useQuery("get", "/customer/list", {}, { enabled, select }), which derives the query key from method + path (dropping the hand-written createQueryKeys entry and the manual key) and forwards the request signal for cancellation. The response type still flows from schema.d.ts as CustomerResponse[]. Tests assert the path, the admin/token enabled gate, and the empty-body select fallback. * test(ui): read the last render's options in useCustomers helper The lastCallOptions helper was named for the last call but read mock.calls[0]. Harmless while each test renders once, but it would silently assert against first-render options if a test ever re-renders. Read the final call instead. * refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface (#32968) * refactor(ui): colocate the usage view, keeping the shared usage components Split for the usage (UsagePage) segment. Most of the folder is the usage page's own view, but four pieces are reused elsewhere and stay in @/components/UsagePage: TopKeyView (old-usage), KeyModelUsageView and value_formatters (activity_metrics), and the shared types (activity_metrics, chartUtils). The other 21 files move into usage/_components, preserving the folder structure. The external consumers import only the retained files, so they are untouched. The moved files' imports of the retained files become @/components/UsagePage paths, other escaping relative imports are absolutized, and lint suppressions are re-keyed for moved files only. No behavior change. * refactor(ui): colocate the mcp-servers view, keeping the shared mcp_tools surface * docs(github): add Final Attestation and per-test sanity-check step to QA runbook * refactor(ui): convert endpoint usage charts to shadcn/recharts (#32723) * refactor(ui): convert endpoint usage charts to shadcn/recharts Adds a LineChart wrapper to the shared charts kit, mirroring the BarChart/AreaChart composition with connectNulls and curveType props, and converts EndpointUsageBarChart and EndpointUsageLineChart from tremor to the shared wrappers. Both endpoint chart tests now assert on real recharts SVG output instead of tremor mocks. * refactor(ui): drop unused endpointData prop from EndpointUsageLineChart * fix(ui): point endpoint chart test type imports at the UsagePage types alias after colocation move * fix(auto_router): filter embedding models out of tier selects, require all tiers, add inline validation The Add Auto Router complexity tab let chat models fill the embedding-model slot (and vice versa) since neither dropdown filtered on ModelGroup.mode, and submit only required at least one of the four tiers instead of all four. Adds getMissingTiersError alongside the existing getSemanticConfigError, and highlights unfilled tier/embedding selects inline once a submit attempt fails. * fix(model-cost-map): anchor the bedrock-claude-ids routing rule to the start of the id * fix(proxy-auth): deny provider-wildcard access inferred through an unrecognized model namespace * fix(auto_router): reset inline validation errors when switching router type * fix(auto_router): flag name field and tier fields together on empty submit Clicking Add Auto Router with the name empty returned early with only a toast, so blank tier selects never got their inline error state. The empty-name branch now sets showValidationErrors and triggers antd validation on the name field, so every unfilled mandatory field is flagged at once. Adds a regression test for the tab component. * feat(mcp): mint gateway-bound envelope at the token endpoint for dcr_bridge oauth_delegate * feat(mcp): seal the authorizing key hash in the dcr_bridge envelope The mint bound only user_id/server_id into the envelope, which gave admission no way to reload the caller's key and enforce its current restrictions. Seal the hashed authorizing key instead (a one-way digest, not a usable credential), so admission reloads the live UserAPIKeyAuth by it and the key's team/org/tool permissions and revocation apply per request. Extract the token endpoint's key resolution into a shared _resolve_active_litellm_key so the per-user token store (user_id) and the bridge mint (key hash) derive from one active-key-gated path, and fail the mint closed with invalid_request when no active key accompanies the request. * fix(mcp): return 502 not KeyError when a bridge upstream response lacks access_token The eager access_token = token_response["access_token"] extraction ran before the dcr_bridge branch, so a missing upstream access_token raised an unhandled KeyError and _bridge_grant_from_token_response's nil guard (which maps to a clean 502) was dead code. Move the extraction onto the non-bridge result path so the bridge branch reaches its 502 guard. * fix(mcp): let a keyless-user active key mint a bridge envelope _resolve_active_litellm_key gated on _active_key_user_id, which returns None both for blocked/expired keys AND for valid keys with no user_id, so a team-scoped or service-account key was wrongly rejected with invalid_request at bridge token exchange. Split the active-state gate (_key_is_active: blocked/expiry only) from the user_id extraction; the mint seals the key hash, not the user, and admission already handles a keyless-user key. The per-user token store still gets no user for such a key, as there is none to key a stored credential by. * style(mcp): use X | None annotations on the touched key-resolution helpers The keyless-user fix moved these signatures, so their pre-existing Optional[...] annotations counted against the diff and tripped the UP045 strict-budget gate. Modernize the four touched return annotations to the X | None form the gate wants; runtime behavior is unchanged. * fix(mcp): coerce numeric expires_in and make the active-key check total Two correctness gaps in the bridge mint. _bridge_grant_from_token_response only accepted an int expires_in, dropping a float (3600.0) or numeric-string ('3600') lifetime to None so the envelope fell back to its 1h cap and could outlive a shorter-lived upstream token; coerce it to a positive int (bool excluded). And _key_is_active called datetime.fromisoformat on the str|datetime expires outside the resolver's try, so a malformed stored expiry raised an unhandled 500 instead of the fail-closed invalid_request; it now fails closed (inactive) on an unparseable expiry. Regression tests cover int/float/string/bool coercion, the short-float TTL, and the malformed-expiry fail-closed path. * fix(mcp): harden the bridge token mint (multi-lens review pass) Findings from a full adversarial review of the mint path across security, correctness, error-handling, concurrency, and OAuth-protocol dimensions. - expires_in coercion is now total: int(float(...)) can raise OverflowError on Infinity / a giant numeric string, which escaped the ValueError/TypeError catch and 500'd the token endpoint. Unified to catch OverflowError too. - Resolve the litellm identity BEFORE exchanging the single-use upstream code, so a missing or transiently-unresolvable identity fails closed with invalid_request without burning the code (the mint re-resolves via a cache hit). - The no-identity failure is now an RFC 6749 5.2-shaped invalid_request (JSONResponse, top-level error, no-store) instead of a detail-wrapped HTTPException, matching the BYOK OAuth endpoint. - EnvelopeTooLarge (upstream token too big to seal) surfaces a 502, not a 500. - The upstream refresh_token is no longer sealed into the envelope: the edge never consumes it, so it was dead weight embedding a long-lived upstream credential in the client bearer and enlarging the envelope; refresh is a follow-up (a dedicated refresh-envelope). Security review found no exploitable defect (forgery, cross-server/user replay, leakage, confused-deputy all closed). Regression tests cover the OverflowError, the code-not-burned path, the RFC-shaped error, the 502, and the dropped refresh. * fix(mcp): close the burn-before-check gate for both grants and validate master_key first Follow-up to the pre-exchange identity gate, which I had only added to the authorization_code branch and which left the master_key check inside the mint (after the upstream exchange) - so the very burn-then-fail pattern it was meant to prevent still applied to refresh_token grants and to a misconfigured gateway. - Hoist a single pre-exchange gate above the upstream call that covers BOTH grant types: it fails closed (invalid_request) on an unresolvable litellm identity and 500s on an unset master_key BEFORE the single-use code or refresh token is exchanged/rotated, so a bad key or a misconfigured gateway never burns the upstream credential. - Report expires_in from the envelope JWT's own second-truncated exp (rounding the elapsed portion up) instead of the raw expires_at - now delta, so the client is never told the bearer is valid past the ~1s point admission already expires it. Regression tests assert the upstream exchange is never called on the no-identity refresh grant and the master_key-unset path, and that the reported expires_in does not overstate the JWT exp. * refactor(mcp): make the bridge delegate mint a phased failures-as-values pipeline The dcr_bridge oauth_delegate token mint validated its preconditions in two places: a pre-exchange guard inside exchange_token_with_server (master_key set, resolvable litellm identity) and an authoritative re-check inside the post-exchange _mint_bridge_delegate_token_response. Keeping the two in step by hand is what kept producing the same class of finding: a precondition guarded on one grant branch but not the other, master_key checked after the exchange on one path, identity resolved twice, and each failure raising an ad-hoc HTTPException with its own status and body shape. Model the mint as three phases whose failures are values. _prepare_bridge_mint runs before the exchange, checks every precondition once (master_key, then identity), and returns either a frozen _BridgeMintReady carrying the resolved key hash and the master-key-derived envelope keys, or a _BridgeMintError literal. Because every precondition lives in prepare, and prepare runs before the upstream POST, no failure can burn the single-use code or rotate a refresh token, for either grant type, by construction rather than by a guard we have to remember to keep in sync. _finish_bridge_mint runs after the exchange and has no preconditions left that can fail; its only failure values are properties of the upstream response itself (no usable access_token, or a token too large to seal). One mapper, _bridge_mint_error_response, turns each _BridgeMintError into an RFC 6749 section 5.2-shaped body with a status truthful about where the failure is (400 for the caller, 500 for gateway config, 502 for the upstream), with an exhaustive match plus assert_never so a new failure mode cannot be added without a matching status. Behavior is unchanged for the client. Every failure that previously raised now returns the same status as an OAuth error body, which is the correct token-endpoint contract; the three tests that asserted a raised HTTPException now assert the returned response. _exchange_for_bridge_server additionally asserts the identity resolver is awaited exactly once for a bridge server and never for a non-bridge one. * fix(mcp): let the bridge envelope report expires_in 0 at the jwt exp boundary _finish_bridge_mint floored the reported expires_in at 1. Admission expires the envelope against the JWT's second-truncated exp, so when the mint lands in the same second that exp falls on (a sub-second upstream lifetime, for instance), the true remaining life is 0 and reporting 1 tells the client the bearer lives one second past the point admission already rejects it. Floor at 0 instead so the reported lifetime never overstates the exp; the value still cannot go negative. The regression pins the boundary directly: minting at now=100.25 with a 1s upstream token seals exp=101, and the reported expires_in is max(0, 101 - ceil(100.25)) = 0. Under the old floor of 1 it reads 1, so the test fails on that mutation. Also drops the unused mcp_server parameter from _prepare_bridge_mint; identity and key derivation there never referenced the server. * refactor(mcp): make bridge-mint resolvers return tagged unions so status is truthful by construction Three findings landed together, all one defect: a resolution step crushed several distinct outcomes into a single None or a silent default, so the mint's error mapper could not tell them apart and assigned the wrong status. Identity resolution mapped a database outage to the same None as a missing credential, which the mint reported as 400 invalid_request, blaming the caller for a gateway outage while admission statuses the same outage 503/500. Lifetime coercion mapped an explicit non-positive expires_in to the same None as an absent one, so an upstream token the IdP reports as already dead was sealed into an hour-long envelope. And the refresh_token grant was run through the upstream exchange (which can rotate the client's upstream refresh credential) and its result then discarded, even though a bridge server seals no refresh_token and the client never holds one to present. Rather than add a mapping branch per finding, the fix changes the return types so a wrong status is not representable. Each resolution step now returns a precise tagged value instead of None: identity resolution returns a _ResolvedKey or one of no_active_key / unavailable / unresolvable, classified the same way admission's _reload_admitted_key classifies the same conditions; upstream-lifetime classification returns a positive number of seconds, "unspecified" (absent or unparseable, which the envelope caps), or "expired" (a parseable non-positive value, an already-dead token); and upstream-grant validation returns a typed grant or one of no_access_token / expired_lifetime. Thin exhaustive mappers (match plus assert_never) lift each vocabulary into one bridge-mint taxonomy of eight named failures, and a single _bridge_mint_error_response gives each its truthful RFC 6749 §5.2 status: 400 for the caller's missing credential or an unsupported grant, 503 for a transient auth-DB outage, 500 for a gateway that cannot resolve identity or is not configured, and 502 for an upstream response with no usable token, an already-expired lifetime, or a token too large to seal. Adding a failure mode now requires a new literal and a match arm the type checker forces, so the class of wrong-status bug cannot recur silently. The refresh_token grant is rejected in _prepare_bridge_mint before the exchange with unsupported_grant_type, so it can never rotate or consume the client's upstream refresh credential; renewal is re-running authorization_code, as the sealed refresh_token=None already intends. An absent or unparseable expires_in still mints a capped envelope (the by-design behaviour for an upstream that omits the field); only an explicitly-dead lifetime is rejected. Tests cover the resolver's three failure classes (including a real connection-error outage and a missing prisma_client), the mint statuses for each (503 before the upstream exchange, 500, 502 on an expired upstream lifetime, and a capped mint on an unknown one), and the refresh-grant rejection before any exchange. The three findings are mutation-checked: reverting each fix turns its regression test red. * fix(mcp): treat a positive sub-second upstream lifetime as alive, not expired _classify_upstream_lifetime decided "expired" from int(float(expires_in)), which truncates toward zero, so a positive fractional lifetime in (0, 1) became 0 and was misread as already elapsed. That rejected the mint with 502 in _finish_bridge_mint after the single-use upstream code had already been consumed, even though the upstream reported a positive remaining lifetime. Decide expired on the parsed numeric value rather than its truncated int, so only a genuinely non-positive value is expired. The envelope works in whole seconds and cannot represent a sub-second lifetime, so a positive value that truncates to 0 clamps up to the 1s floor instead of being rejected. Values >= 1 still truncate toward zero so the envelope never claims more life than the upstream stated, and NaN / Infinity / oversized input still read as unparseable ("unspecified"). Regression covers the classifier (0.5 and 0.001 clamp to 1, 1.9 truncates to 1, -0.5 stays expired) and the mint (a 0.5s upstream lifetime mints a 200 envelope rather than a 502); reverting to the truncate-then-check reddens both. * fix(auto_router): inline error for missing LLM classifier model Selecting the LLM classifier without picking a model only surfaced a toast on submit; the classifier model select now gets the same red outline and helper text as the tier and embedding selects once a submit attempt has failed. * build(dev-env): add make bootstrap and unprovisioned-checkout preflight to pre-commit * feat(router): random-pick multi-model complexity tiers (#32967) * feat(router): random-pick multi-model complexity tiers Tier pools already make sense without adaptive; stop pinning lists to index 0 and shuffle within the classified tier instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): format complexity router config Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): use PEP 585 types for tier pools Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(xecguard): sanitize scan result before recording it for logging (#32935) * chore: keep it brief * chore: keep it brief * docs(readme): point developer-mode setup at make bootstrap * chore: keep it concise * feat(router): add Router(plugins=[...]) routing-plugin pipeline (#32972) * feat(router): add Router(plugins=[...]) routing-plugin pipeline Runs a sequence of user-supplied plugins before the routing decision is made. Each plugin reads/mutates a RoutingContext (messages, candidate models, metadata, signals); the narrowed candidate list is enforced when picking a deployment, raising rather than silently falling back if a plugin narrows to zero candidates. Prototype for the routing-plugin pipeline discussed in #32168. * fix(router): use ruff-modern typing, add raw/structured messages to RoutingContext - Use dict/list/X|None instead of Dict/List/Optional in new code, staying within the ruff strict-rule budget ratchet - Extract the guardrail-translation message normalization ComplexityRouter already had into a shared resolve_structured_messages() helper (litellm_core_utils/prompt_templates/factory.py), reused by ComplexityRouter and the new routing-plugin pipeline instead of duplicating it - RoutingContext now exposes both raw_messages (as received) and structured_messages (normalized across chat completions / Anthropic messages / Responses API), mirroring CustomGuardrail.apply_guardrail's pattern, per review feedback on #32972 - Add direct unit tests for _run_routing_plugins and _filter_by_routing_plugin_candidates (router_code_coverage gate requires every router.py function be called by name somewhere in tests/) * fix(test): rename to test_router_routing_plugins.py router_code_coverage.py's AST scanner only inspects test files whose filename contains the substring "router" -- test_routing_plugins.py doesn't match (routing != router), so it silently skipped this file and flagged _run_routing_plugins/_filter_by_routing_plugin_candidates as untested despite the direct unit tests added for them. * fix(router): fail closed when plugins are configured but the resolved routing path can't run them Router.completion() (and other sync entry points) resolves deployments via the synchronous get_available_deployment(), which never runs async_pre_routing_hook and therefore never runs the routing-plugin pipeline. async_get_available_deployment() itself falls back to that same synchronous method for routing strategies without an async-native selector (e.g. legacy "usage-based-routing" v1). Both paths would let a policy plugin (e.g. a deny-all rule) be silently bypassed. Raise instead of silently proceeding when self.routing_plugins is configured and the sync path is reached, since applying the pipeline to every selector path is a larger change out of scope for this PR. Per review: https://github.com/BerriAI/litellm/pull/32972/changes/BASE..bdfb583c2c6f8df10004fb249e11629d41ce71fa#r3565373303 * feat(router): soft-floor adaptive mode for complexity router (#32947) * feat(router): soft-floor adaptive mode for complexity router Let complexity_router_config.adaptive=true Thompson-sample across the union of tier pools with a tier-distance penalty, and wire the existing adaptive post-call bandit so mis-tiered requests can still recover. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): reattach adaptive hooks for hybrid complexity Finalize was wiping every AdaptiveRouterPostCallHook and only re-registering standalone auto_router/adaptive_router deployments, so complexity adaptive=true never received bandit updates. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): drop unnecessary hybrid docstrings Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attribute adaptive feedback Credit user reactions to the model that produced the previous response while keeping current-response signals on the serving model Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): tune hybrid cold defaults Use the cost-weighted policy that beat equal-pool complexity in the full bakeoff, and make the committed harness compare identical tier pools Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): preserve hybrid cold quality floor Sample only unobserved models in the classified tier until feedback exists, then apply adaptive scoring without mis-penalizing models shared across tiers Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): bound feedback context cache Cap retained session feedback so unique session IDs cannot exhaust router memory Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): preserve exhaustion signals Include tool-result exhaustion in adaptive feedback and clear strict lint regressions blocking CI Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): remove stale owner cache Remove obsolete attribution state, tighten the embedded router type, and keep the test diff focused on adaptive behavior Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): centralize hook cleanup Use the callback manager to discover and remove adaptive hooks across every registered callback list Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(responses): continue MCP gateway tool turns from the final response and surface failures When a /responses request uses a hosted MCP tool (server_url: litellm_proxy/<label>) with store=true and the model calls a tool, the gateway auto-executes the tool and streams one logical response stitched from several upstream responses: an interim response whose only output is the function_call, then the post-tool answer B1 (correctness): every streamed event was pinned to the first round's response id, i.e. the interim response that carries the function_call but no tool output. The client then continued the next turn from that dangling response and the provider rejected it with "No tool output found for function call <id>", which on the streaming path surfaced as a silent empty completion. The fix adopts each auto-execute round's own response id (the cached id is reset when a follow-up round starts) so the client continues from the final round, whose stored input chain includes the function_call_output B2 (robustness): initial and follow-up call failures were swallowed; the stream emitted the mcp_list_tools discovery events and then closed with HTTP 200 and no output and no error. The fix stashes the failure, makes the initial call eagerly in aresponses_api_with_mcp so a pre-stream failure re-raises as a real 4xx before any SSE bytes are written, and emits a terminal error event when a follow-up call fails mid-stream Adds regression tests covering continuation exposing the final round's response id rather than the interim tool-call id, a follow-up failure emitting a terminal error event, and an initial-call failure being stashed for eager re-raise * ci(ui): report only error-level knip findings in CI (#32971) * feat(batches): track cost for unmanaged Bedrock batches, generalize the flag (#32315) * feat(batches): track cost for unmanaged Bedrock batches, generalize the flag CheckBatchCost skipped Bedrock batches whose unified_object_id is a raw model-invocation-job ARN, the same root cause previously fixed for unmanaged Vertex batches. Bedrock batches embed the model name in their s3:// input file name instead (litellm-bedrock-files-{model}-{uuid}.jsonl), so the same routing mechanism now derives the model from that layout and matches it to a configured bedrock deployment. track_unmanaged_vertex_batch_cost is renamed to track_unmanaged_batch_cost since two providers now share this mechanism. * fix(batches): parse Bedrock batch output and price with deployment model name Bedrock model-invocation-job results use modelOutput/error rows and short internal model ids that are not in the cost map, so unmanaged batch cost tracking logged tokens but $0 spend. Use deployment model name for pricing and add regression tests. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): walk custom_tool_call_output items in _content_utils (#32969) * fix(guardrails): walk custom_tool_call_output items in _content_utils * Change _OUTPUT_ITEM_TYPES to Frozenset type * fix(guardrails): use builtin frozenset generic for _OUTPUT_ITEM_TYPES annotation Frozenset is not a defined name (typing exports FrozenSet, the builtin is frozenset), so module import raised NameError and broke every proxy test suite. The builtin generic is valid on the supported python floor (3.10) and keeps the UP006 ruff-strict budget at its ceiling, which the typing alias would exceed * fix: show and allow editing team model aliases after team creation (#33047) * refactor(ui): rename OldTeams component file to Teams * fix: show and allow editing team model aliases after team creation * fix(ui): mark team model_aliases as nullable to match the prisma schema * fix(ci): bump pillow to 12.3.0 to resolve osv-scan CVEs (#33093) * fix(proxy): track unauthenticated pass-through requests in spend logs (#32410) Pass-through endpoints configured with auth=false reach the cost-tracking callback with no key/user/team/end-user, so _should_track_cost_callback returned False and the spend-log write was skipped, leaving the request out of request/usage logs. Track pass-through call types even when unauthenticated so the SpendLog row is still written. Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(lasso): send source.type=litellm for Used By attribution (#33090) Co-authored-by: Or Gershoni <org@lasso.security> * feat(mcp): generalize the bridge envelope identity to a key_hash or user_id subject The scripted two-header client mints under a virtual key it presents at the token endpoint (key_hash), but the interactive DCR client authenticates via SSO at the bridged authorize, which yields a user, not a key. Make EnvelopeIdentity a discriminated subject (subject_type key_hash | user_id) with key_hash_identity / user_identity constructors, and dispatch admission on it: a key_hash reloads the key, a user_id reloads the user and admits them as themselves (user-level budget and SCIM enforced via the same centralized gate; no team bound, since a user belongs to many teams or none). The interactive producer that mints a user_id envelope lands in the follow-up commit. * feat(mcp): interactive SSO sign-in for dcr_bridge oauth_delegate DCR clients Completes the oauth_delegate bridge for real DCR clients (Claude Code, Claude Desktop), which send no litellm key and cannot use the scripted two-header path. On the short-circuit bridge arm the gateway now captures the SSO-authenticated litellm user from the browser session at /authorize and seals it into the OAuth state; at /callback it seals that user plus the upstream code into a gateway authorization code the client echoes back; at /token it recovers the user, exchanges the real upstream code, and mints a user-subject envelope. The user identity captured in the browser thus rides to the back-channel token call with nothing stored server-side, and admission opens the envelope under that user. The scripted key_hash path is unchanged (raw upstream code, key from the request); without a session the browser is sent through login first. * fix(mcp): classify the user-subject reload's errors like the key path (503 outage, 401 missing) _reload_admitted_user mirrored only part of _reload_admitted_key's error contract: it caught ProxyException and HTTPException but had no arm for anything else, so a transient DB outage surfaced as an opaque 500 instead of the retryable 503 the key path guarantees, and a missing user surfaced as a 500 too. The missing-user case is the subtle one: get_user_object raises a bare Exception for a deleted user (not a ProxyException like get_key_object does for a missing key), so the ProxyException/HTTPException clause never caught it and the user_object-is-None branch it was supposed to hit is unreachable on the production path. Add the same except-Exception arm the key path uses, with the one deliberate difference the differing get_user_object contract requires: a database-service-unavailable error still raises the retryable 503, while a missing user or any other non-outage resolution failure fails closed as a 401 rather than propagating as a 500. The regression tests now drive the real behavior (get_user_object raising) rather than a None return that never happens in production, and cover both the 503 outage and the 401 missing-user paths. * fix(mcp): admit a user-subject envelope with the user's own MCP object permission _reload_admitted_user returned a bare UserAPIKeyAuth(user_id=...), so the shared get_allowed_mcp_servers found no key/team/object-permission grants and an interactive SSO client could admit successfully yet see zero tools on a normal (allow_all_keys=False) server. The key path returns the full key record whose object permission drives that computation; the user path dropped it. Resolve the user's own MCP object permission and put it on the returned auth, so the same get_allowed_mcp_servers the key path uses grants the user their litellm-granted servers and access groups. This reuses get_object_permission (the id-to-grants resolver keys and teams already use) and does not duplicate any permission logic; get_user_object does not load object_permission, so it is resolved from the user's object_permission_id the same way the key and team paths do. Only the user's own object permission is bound. A UserAPIKeyAuth carries a single team_id while a user may belong to many teams, so team-inherited MCP grants for a user are a follow-up: they need a many-teams union get_allowed_mcp_servers does not do off one auth object, and faking one here would be the kind of half-measure that spawns more bugs. Tests cover the user's object permission riding onto the admitted auth, and the existing admit/SCIM/missing-user/503 cases still hold. * fix(ui): respect litellm_key_header_name in BYOK credential save and workflow runs fetches (#33103) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant (#33040) * feat(ui): rebuild the Virtual Keys table on the shared DataTable (#32991) * feat(ui): rebuild the Virtual Keys table on the shared DataTable Replaces the hand-rolled Tremor table and bespoke toolbar/pagination on the admin Virtual Keys page with the shared DataTable: server-side sort, paginate, and filter, a sticky scrolling body, a search plus column-visibility plus filters toolbar, a right-side filter drawer, and a rows-per-page footer. A page header with the existing key icon carries the Create New Key action. Adds reusable, shadcn-default building blocks for the tables migrating onto the DataTable next: shared IdentityCell, ModelsCell, and SpendBudgetCell in shared/table_cells, plus a shared PageHeader. The models cell reveals overflow in a hover tooltip and the spend/budget cell uses the Meter primitive. All data and domain logic is preserved, including the useKeys query, team and org alias resolution, the user popover, and the KeyInfoView detail swap. The rich async Team/Org/Alias filters move into the drawer, and the toolbar search maps to the key-alias substring search. Status now also reflects key expiry alongside blocked and SCIM-blocked. The VirtualKeysTable tests are updated to the new markup and extended with focused coverage for each new shared cell * fix(ui): address Virtual Keys redesign review feedback Fold the status badge into the clickable Key cell and drop the separate Status column so a key's alias, secret, and status read as one unit. The Key cell is now the single click target that opens the key detail; the whole-row click is removed Migrate the filter drawer off AntD to shadcn. A new Combobox composed from Popover and Input backs the Team, Organization, and Key Alias filters, keeping search and the alias infinite-scroll Show $0.00 for zero spend instead of a hyphen, and extend the shared DataTable with badge, chips, and meter skeleton shapes so the loading state matches the loaded cells (status pill, model chips, spend meter) rather than uniform bars Fix key sorting: the Key column sent its column id "key" as sort_by, which /key/list rejects with 400. It now sorts by the backend field key_alias * fix(ui): use the shadcn base combobox and refine the keys filters and skeletons Replace the hand-rolled filter combobox with the supported shadcn Base UI combobox (ui/combobox, added via the CLI and reused through a small SearchSelect wrapper). Its vended input-group and textarea deps are written for React 19 (plain functions with ref-as-prop); this app is on React 18, where those subcomponents drop the refs Base UI passes for focus and anchoring, so InputGroupInput, InputGroupButton, and ComboboxTrigger are adapted to forwardRef. Those ui/ files now diverge from the registry, and a future shadcn add would overwrite the adaptation until the app moves to React 19. Adds class-variance-authority, which input-group needs Give loading skeletons a per-column renderSkeleton escape hatch on the shared DataTable and mirror the Key cell exactly (alias line, secret, status pill), so skeleton rows match the real rows instead of being shorter and simpler Resolve the automated review: the toolbar search and the drawer Key Alias filter both mapped to the key-alias query, so the search silently overrode the drawer value while its chip stayed visible. Consolidate to a single alias search in the toolbar (placeholder now "Search by key alias…") and drop the redundant drawer field. Re-add coverage for the Created By column's alias-over-email display Refine the Team and Organization filters: they match on name and id, so the labels read "Team" and "Organization" rather than "... ID", each option shows the name with the id on a muted second line instead of "name (id)", and the active-filter chip shows the friendly name * chore(ui): drop duplicate class-variance-authority, use the repo cva package in input-group * fix(mcp): relay upstream OAuth token and DCR rejections instead of a generic 500 An upstream token endpoint rejection (e.g. Google requiring client_secret even for PKCE web clients) escaped exchange_token_with_server as a raw httpx.HTTPStatusError, which the global exception handler turned into an opaque 500 Internal server error in the create-flow UI. The RFC 6749 section 5.2 error body the IdP sent (error, error_description, error_uri) is now relayed with the upstream's own 400/401 status; rejections outside the section 5.2 contract map to 502 so a broken upstream is not misattributed to the caller. The same relay covers the non-bridge DCR registration arm, and a 200 token response without a usable access_token now answers 502 instead of a KeyError 500. The catch wraps the post call itself because litellm's AsyncHTTPHandler raises MaskedHTTPStatusError at call time, which also made the pre-existing bridge-relay status check unreachable in production. The dashboard's token exchange error message now composes error and error_description so the form shows the IdP's reason * refactor(mcp): drop bridge relay status check made unreachable by the unified relay The try/except around the registration post now relays every upstream 4xx/5xx for both arms, so the bridge_relay status_code check could never fire; removing it addresses the Greptile P2 dead-code finding * fix(mcp): classify get_user_object's wrapped DB outage across the exception chain get_user_object catches every DB failure in a broad except and re-raises a bare ValueError (litellm/proxy/auth/auth_checks.py), so a real outage and a missing user look identical and the original error survives only as __context__. The dcr_bridge admission path keyed its 503-vs-401 decision on the exception type, so a transient outage during a user-subject reload surfaced as a 401 rather than a retryable 503, and the regression test injected a raw ConnectionError, a shape get_user_object never produces, so it passed on a fiction Add PrismaDBExceptionHandler.is_database_service_unavailable_error_in_chain, which walks __cause__/__context__ (bounded and cycle-safe) the PEP 3134 way, and route _raise_503_if_db_unavailable through it. Move the user's object_permission resolution inside the single classified try so an outage there is a 503 too, never an opaque 500. Pin get_user_object's wrapping with a contract test that drives the real function, and drive the reload tests with that same faithful shape so a chain-blind regression fails them * fix: redact async complete streaming response for custom callbacks (#33106) * fix response not being redacted for custom callbacks with streaming enabled * reduce code duplication * add unit test * fix: resolve lint violations in adopted redaction fix * fix: scope streaming response redaction to the opted-out custom logger --------- Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de> * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 (#33041) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 * fix(ui): address Virtual Keys redesign review nits (#33112) * fix(ui): address Virtual Keys redesign review nits Restore sorting by budget on the merged Spend / Budget column. The column now uses a new DataTableMultiSortHeader whose chevron opens a menu offering Spend and Budget in both directions plus Reset, so the progress-bar cell stays merged while the sort field becomes an explicit choice. Sorting is server-side, so the chosen field id (spend or max_budget, both accepted by /key/list) flows straight through as sort_by Fill the DataTable to its container width when column resizing is on. The table width was pinned to the sum of column widths, so hiding columns left an empty gutter on the right. It now keeps that width as a minimum and stretches to 100% on underflow while still scrolling on overflow, which also covers the same gap in TeamVirtualKeysTable since both share the component Drop the dark background box behind the page-header icon so the Virtual Keys header reads like the Teams header, and pull the 4-line inline filter lambda in SearchSelect out into a named matchesQuery helper Extends the DataTable and VirtualKeysTable tests to cover the new multi-field sort menu (field id maps to sort_by, active indicator, reset) and the fill-to-container width * fix(ui): emphasize the active field in the Spend / Budget sort header The merged Spend / Budget header always read "Spend / Budget" regardless of which field drove the sort, so after picking Budget descending there was no way to tell what was sorted without reopening the menu. The header now builds its label from the sort fields and emphasizes whichever one is active (bold, full-strength text) while muting the other, so the sorted column reads at a glance alongside the direction chevron. Drops the now-redundant title prop since the label is derived from the fields * fix(ui): remove w-full so the keys page content stops overflowing by 32px The virtual keys content wrapper used "w-full mx-4", which sets the width to 100% of the parent and then adds 16px of horizontal margin on each side, so its margin-box came to 100% + 32px and overflowed the scrollable main region by exactly 32px. That surfaced as a horizontal scrollbar along the bottom of the whole content area, under the pagination. A block div is already full-width, so dropping w-full lets mx-4 inset it correctly with no overflow * fix(ui): darken the clickable Key cell on hover so it reads as clickable The Key cell was the click target that opens the key detail, but hovering only faded the chevron in with no change to the cell itself, so there was no cue that the area was clickable. Give the cell a subtle muted background and a pointer cursor on hover. The button spans the full cell (a negative inline margin plus a matching width offset so the hover fill reaches both cell edges while the title stays aligned with the other columns) * fix(openai/responses): clamp max_output_tokens below API minimum (#33098) * fix(openai/responses): clamp max_output_tokens below API minimum Claude Code sends a max_tokens=1 warmup probe when running /model, which the Anthropic Messages -> Responses adapter forwards as max_output_tokens=1. OpenAI's Responses API rejects values below 16, so the probe failed with a 400. Clamp anything below the minimum up to 16 in map_openai_params so all Responses API entrypoints (direct, chat->responses, anthropic->responses) are covered. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * refactor(openai/responses): extract _enforce_min_max_output_tokens helper Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(prometheus): read v3 rate limiter remaining values for per-key model gauges (#33119) * fix(ui): drop w-full from page-content wrappers to remove 32px horizontal overflow (#33118) Several dashboard pages wrap their content in a div styled w-full mx-4, so the element's width is 100% of the scrollable main while mx-4 adds 16px of margin on each side. That makes the margin-box 100% + 32px wide, which overflows main by exactly 32px. Because main uses overflow-y-auto its overflow-x computes to auto, so the overflow surfaces as a horizontal scrollbar along the bottom of the whole content area under the pagination The wrapped block is already full width without w-full, so removing that one token keeps the layout and drops the overflow to 0. This is the same fix already applied to the Virtual Keys page in #33112, extended to the remaining pages that share the wrapper: Models + Endpoints, Tag Management, Organizations, Vector Stores, AI Hub, and Logging & Alerts * refactor(ui): migrate straightforward value debounces to react-pacer (#33042) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 * refactor(ui): migrate straightforward value debounces to react-pacer * feat(mcp): client-held refresh envelope for the dcr_bridge oauth_delegate flow A dcr_bridge oauth_delegate access envelope is capped at one hour, and until now the mode had no refresh at all: when the envelope expired the client had to re-run the interactive authorization_code flow. This adds a second client-held credential, the refresh envelope, so the client renews on a back channel and only re-authenticates when the refresh envelope expires or the upstream refresh token dies. The refresh envelope is a distinct llm_refresh_ credential that seals only the upstream refresh token (never the access token) bound to the same litellm identity and MCP server as the access envelope, under the same master-key-derived keys, with nothing stored server-side. Both envelopes now carry a signed kind claim ("access" or "refresh") that open() requires to match, so a refresh envelope can never open as an access credential even if its wire prefix is swapped (the prefix is not signed; the claim is). A refresh envelope presented at the MCP tool-call edge is not an access envelope, so admission fails it closed the same way it already fails any non-access bearer. At the token endpoint the authorization_code mint now returns a refresh envelope alongside the access envelope whenever the upstream returned a refresh token, and the refresh_token grant is supported for bridge servers: the client presents its refresh envelope, the endpoint opens it, re-validates the sealed litellm key so a revoked key cannot keep refreshing, unwraps the real upstream refresh token, exchanges it with the upstream IdP, and returns a fresh access envelope. Because the endpoint re-seals a refresh envelope only when the upstream returns a new refresh token, the design mirrors the upstream's own rotation policy rather than reinventing it: with a rotating upstream the client rotates and reuse is detected upstream; with a non-rotating upstream the original refresh envelope stands until its bounded 14-day TTL. Both preconditions and the unwrap run before the exchange, so a rejected refresh never consumes or rotates an upstream token. The pure envelope and credential layers stay side-effect free: mint/open share one signing, size, and kind gate across both envelope kinds, and every failure is a value. Tests cover the refresh round-trip, the kind-claim and server-id bindings, the revoked-key gate, upstream rotation carried through, the unwrap sending the real upstream token upstream, and edge rejection of a refresh envelope; the three security bindings are mutation-checked. Limitation documented in the PR: gateway-enforced refresh rotation with reuse detection would require server-side state, which this zero-custody mode omits by design, so the refresh envelope inherits the upstream's rotation posture plus gateway identity binding and a bounded TTL. * fix(mcp): reject a refresh envelope explicitly at the tool-call edge The live proof showed a refresh envelope presented at the MCP tool-call edge was rejected, but through the generic oauth2 arm ("expected a virtual key starting with sk-") rather than the bridge arm, because the admission routing gate is_bridge_envelope_shaped matched only the access prefix. The rejection was already fail-closed and never forwarded anything upstream, but the path was imprecise and the unit test modelled a route the real router did not take. Match either envelope kind in is_bridge_envelope_shaped so the bridge arm engages for a refresh envelope too, and have resolve_bridge_envelope return BridgeEnvelopeInvalid for it: a refresh envelope is a valid gateway credential but only ever presented back to the token endpoint, never usable to authenticate a tool call. Admission now fails it closed with the bridge arm's own 401 ("Invalid or expired credential"), live-verified, with the upstream never touched. is_bridge_envelope_shaped has a single caller (the admission routing gate), so the change is contained. * fix(mcp): SecretStr the unwrapped refresh token, drop the dead request arg, fail closed on a missing user Three review findings on the refresh path, addressed at the root: _BridgeRefreshReady.upstream_refresh_token was a plain str, the one credential in the envelope/bridge layer that escaped the SecretStr discipline every other one follows (RefreshCredential.refresh_token, UpstreamTokenGrant.access_token, EnvelopeKeys.signing_key). A repr or a traceback capturing a local _BridgeRefreshReady would have logged the raw upstream refresh token. It is now a SecretStr, carried as the SecretStr open_bridge_refresh_envelope already returns and unwrapped only at the point the exchange builds the upstream request body. _prepare_bridge_refresh took a request it never read; on the refresh path identity comes entirely from the sealed envelope, not the HTTP request, so the parameter was dead and misleadingly implied it read from the request the way the authorization_code prepare does. Removed, and the caller updated. _reload_active_user_by_id misclassified a missing user as unresolvable (500). This is the same root cause as the admission user-reload fix: get_user_object raises a bare Exception for a deleted user rather than a ProxyException, so its except-Exception arm must fail closed to no_active_key (which the refresh path maps to invalid_grant) for anything that is not a database-service-unavailable outage, rather than treating a missing user as an opaque gateway fault. Regression tests cover the missing-user and DB-outage classifications directly. * fix(mcp): make the dcr_bridge refresh path fail correctly on outages, dead tokens, and revoked owners Four fixes to the refresh_token grant for dcr_bridge oauth_delegate, surfaced by an adversarial pass over the exchange path Route the user-subject re-validation's outage check through the chain-aware classifier, so a transient DB outage (which get_user_object wraps in a bare ValueError) reports as unavailable (a retryable 503) rather than collapsing to no_active_key and an invalid_grant, matching how admission now handles the same wrapper When the upstream reports its own refresh token as already elapsed (refresh_expires_in non-positive), do not seal it into a full-TTL refresh envelope; return no refresh so the exchange degrades to an access-only response, mirroring how the access grant refuses an already-elapsed access token instead of capping it When the upstream rejects the sealed refresh token with 400 invalid_grant (revoked or expired at the IdP), return an RFC 6749 invalid_grant response so the OAuth client re-runs authorization_code, rather than surfacing the opaque upstream error it cannot act on Gate key-subject renewal on the owner's SCIM state, mirroring admission's _reject_if_admitted_owner_scim_deactivated, so an offboarded user cannot keep refreshing a still-active key; the check fails open on a missing owner or a DB blip so a key that outlives its owner record does not get wrongly revoked Each fix has a mutation-checked regression test * test(proxy): add regression tests for management_endpoints edge cases (#32976) Mutation testing surfaced branches in cost_tracking_settings and common_utils that the suite executed but never asserted on. Pin those behaviors with targeted tests: the returned (model, provider) from _resolve_model_for_cost_lookup for deployments carrying a custom_llm_provider and for deployments missing the litellm_params / model_info keys, plus the exact error-response bodies, the caller-identity lookup arguments, and the member and guard branches in common_utils. * fix(auto-router): correct Responses API tool_choice shape and propagate alias litellm_params (#32974) * fix(anthropic-messages): send bare-string tool_choice to Responses API, propagate router-alias litellm_params The Anthropic /v1/messages -> Responses API adapter always wrapped tool_choice in an object ({"type": "auto"}, {"type": "required"}), but the Responses API's tool_choice schema for these cases is a bare string ("auto"/"required"/"none"). Sending the object shape to an OpenAI-compatible backend (e.g. vLLM) fails Pydantic validation with a 400. The "none" case also fell through to "auto" instead of mapping to "none". Separately, litellm_params configured directly on a router-alias deployment (auto_router/complexity_router, adaptive_router, quality_router, or semantic auto_router) - e.g. cache_control_injection_points, drop_params - were silently dropped for every request through that alias. async_pre_routing_hook swaps `model` from the alias name to the selected tier/route's model before the deployment lookup runs, so the outbound call only ever merged in the tier deployment's own litellm_params, never the alias's. Register non-routing-config litellm_params from the alias deployment and apply them to the request whenever a pre-routing hook substitutes the model. * fix: satisfy ruff-strict-budget UP006 and router coverage checker Use builtin dict[...] generics instead of typing.Dict for the two new annotations introduced in the previous commit, since they pushed UP006 over the codebase ceiling in ruff-strict-budget.json. Add a direct unit test for _register_pre_routing_alias_overrides so the text-based router_code_coverage.py checker sees it exercised by name. * fix(router): replace alias-param denylist with a tight allowlist _PRE_ROUTING_ALIAS_RESERVED_PARAMS excluded router-init-only keys from the alias's litellm_params before forwarding the rest as request kwargs, but GenericLiteLLMParams also holds deployment-management fields (tpm, rpm, weight, tags, max_budget, budget_duration, use_in_pass_through, litellm_credential_name, ...) on the same object. Any of those left off the denylist would get silently forwarded as if they were request kwargs. Replace the denylist with a tight allowlist of exactly the two request-shaping params this feature exists for - drop_params and cache_control_injection_points - so unrelated management fields never reach the outbound call regardless of what else GenericLiteLLMParams grows to hold. * fix(router): re-register adaptive-alias overrides on set_model_list reload set_model_list() unconditionally clears pre_routing_alias_overrides on every call (e.g. /config/reload), but _finalize_adaptive_router_if_configured() skips rebuilding an AdaptiveRouter whose model_name already exists in self.adaptive_routers - so _register_pre_routing_alias_overrides() never ran again for an auto_router/adaptive_router alias after a reload, silently dropping its drop_params/cache_control_injection_points. Build the Deployment unconditionally and re-register its overrides even on the skip-existing-router path; only the (expensive) AdaptiveRouter construction itself stays skipped. * style: ruff format after merging litellm_internal_staging * fix(router): drop the alias-param allowlist, exclude only model Per review discussion: instead of a router.py-local allowlist of exactly which litellm_params an alias (auto_router/complexity_router, adaptive_router, quality_router, semantic auto_router) can forward to the request it routes, _register_pre_routing_alias_overrides now forwards everything except `model` (the alias marker itself, e.g. auto_router/complexity_router, never a real provider model). Router-init-only fields (complexity_router_config, complexity_router_default_model, auto_router_config, auto_router_config_path, auto_router_default_model, auto_router_embedding_model, adaptive_router_config, adaptive_router_default_model, quality_router_config, quality_router_default_model) now flow into request_kwargs unfiltered too. That's safe because litellm.completion()/acompletion() already strips anything in litellm.types.utils.all_litellm_params before building the provider request - added these 10 keys there, alongside the deployment-management fields (tpm, rpm, weight, ...) already listed. Verified live: without that addition, complexity_router_config lands in extra_body and ships raw to the provider; with it, it's stripped. This moves the "which fields aren't real LLM params" list from a router.py-local allowlist to the single existing global list every completion() call already depends on, instead of maintaining two. * refactor(router): look up alias litellm_params on demand instead of caching them _register_pre_routing_alias_overrides cached each alias's litellm_params into self.pre_routing_alias_overrides at deployment-init time, which required keeping that cache in sync with set_model_list() reloads - the exact bug the previous adaptive-router-reload fix was patching around (AdaptiveRouter survives a reload, but the cache didn't always get refreshed to match). Delete the cache and the registration method entirely. async_pre_routing_hook now looks up the alias's own litellm_params directly from self.model_list via self.model_name_to_deployment_indices at request time, the same model_list that's already correctly rebuilt on every set_model_list() call. No second piece of state to invalidate, so the reload staleness bug class isn't possible anymore, and it's less code than before. * fix(mcp): keep out-of-contract upstream error bodies out of client responses The token and DCR relays serve unauthenticated OAuth clients, so only the RFC 6749/7591 error fields may cross the trust boundary. A rejection body outside those contracts (HTML error page, proxy banner, stack trace) is now logged server-side, bounded, and the client response names only the upstream status. Addresses the Veria information-exposure finding * fix(mcp): re-request the sealed scope on a bridge refresh when the client omits it The refresh envelope seals the upstream scope as the scope to re-request (RefreshCredential), but _prepare_bridge_refresh dropped it, unwrapping only the refresh token, and the exchange added scope to the upstream request only from the client's HTTP form. A DCR/MCP client typically omits scope on refresh, so the sealed scope was never sent and a stricter upstream could narrow or drop the renewed token's scope Thread the sealed scope through _BridgeRefreshReady.upstream_scope and fall back to it when the client sends none; a client-supplied scope still wins, which RFC 6749 section 6 bounds to the original grant. The regression test drives a refresh where the client omits scope and asserts the upstream POST carries the sealed scope, mutation-checked against both the drop and the fallback * fix(ui): render the sidebar scrollbar with shadcn ScrollArea (#33124) * fix(ui): render the sidebar scrollbar with shadcn ScrollArea The sidebar navigation scrolled through a native overflow-y-auto container, so the browser drew its default scrollbar. It now scrolls through the shadcn ScrollArea primitive so the thumb matches the rest of the dashboard Switching to ScrollArea surfaced a latent styling gap. The Base UI scroll-area, tabs, and separator primitives rely on data-horizontal and data-vertical Tailwind variants that resolve to [data-orientation="horizontal"] and [data-orientation="vertical"], and those variants ship in shadcn's shared stylesheet. The project never imported it, so the classes matched nothing and the scrollbar collapsed to zero width. This adds shadcn as a devDependency and imports shadcn/tailwind.css, which also repairs the vertical tabs and separator styling. See shadcn-ui/ui#9196 for the upstream tracking issue * refactor(ui): inline the Base UI data-* variants, drop the shadcn dep The earlier fix imported shadcn/tailwind.css through the shadcn devDependency, which pulled 219 packages and tied the CSS build to shadcn's package exports (an open Turbopack-breaking bug, shadcn-ui/ui#10931). shadcn's model is that we own the components, so the custom variants those components depend on belong in our own stylesheet rather than a runtime dependency. This inlines the nine data-* custom variants and the no-scrollbar utility that the Base UI primitives reference into globals.css, and removes the shadcn package. * refactor(ui): migrate callback debounce sites to react-pacer with regression tests (#33043) * refactor(ui): standardize debounce waits behind shared DEBOUNCE_WAIT_MS constant * build(ui): bump @tanstack/react-pacer from 0.2.0 to 0.22.1 * refactor(ui): migrate straightforward value debounces to react-pacer * refactor(ui): migrate callback debounce sites to react-pacer with regression tests * chore(ui): restore trailing newline in eslint-suppressions.json * test(ui): mock all pacer debounce hooks in VirtualKeysTable test * fix(ui): update merged debounce tests for OldTeams to Teams rename * fix(mcp): carry the requested scope forward when the upstream omits it on a bridge refresh The prior fix sent the sealed scope on a refresh, but the re-minted refresh envelope re-seals scope from the upstream response, and RFC 6749 section 5.1 lets an upstream omit scope when it is unchanged. So after one refresh whose response omitted scope, the new envelope sealed scope=None and every subsequent refresh dropped it, letting a stricter upstream narrow the renewed token When the upstream omits scope on a bridge refresh, seal the scope we requested (which RFC 6749 section 5.1 defines as the granted scope when omitted) into the renewed access and refresh envelopes, so the scope survives the whole refresh chain. The regression test refreshes against an upstream that omits scope, asserts the new refresh envelope still carries it, and refreshes again off that envelope to prove the chain does not lose it, mutation-checked * refactor(mcp): classify upstream OAuth faults once and derive status, code, and prose from the value Replaces the accreted relay helpers with a faults package (types, classify, render_oauth): every upstream token/DCR rejection is classified into exactly one fault value and the response status, wire error code, and prose are all derived from that value, so a caller-fault code can never ship on a server-fault status (the bugbot finding on invalid_grant over a 500). Classification takes the credential source into account: invalid_client and friends against the server's stored credentials are the operator's fault and render as 502 server_error with gateway-authored prose while the IdP's prose stays in server logs; the same codes against caller-supplied credentials relay on the status the code implies. Classifiers are total, so an unreadable rejection body (lying content-encoding, unconsumed stream) yields the same 502 fault instead of resurrecting the opaque 500 (the second bugbot finding); DCR rejections normalize to 400 per RFC 7591 regardless of the upstream's status * style(mcp): unquote annotations and use PEP 604 unions in the faults package * fix(mcp): detect upstream invalid_grant by the RFC 6749 error field, not a body substring The bridge refresh path decided whether an upstream token-endpoint rejection was invalid_grant by substring-matching the raw response body, so a rejection whose actual error is something else but whose error_description merely contains the string invalid_grant would false-match, map to invalid_grant, and trigger a needless authorization_code re-run Parse the RFC 6749 section 5.2 error object and compare the error field. A non-JSON body, or an error that is not invalid_grant, now propagates as the upstream error rather than being reinterpreted. The regression test drives an invalid_client rejection whose description contains the string invalid_grant and asserts it is not mapped, mutation-checked against the substring match * fix(mcp): keep upstream self-blame codes and gateway capability gaps off the caller Extends the fault matrix per review: server_error and temporarily_unavailable are codes by which the upstream blames itself, so they classify as a new UpstreamReportedFault arm rendering 502/503 with a matching wire code instead of a 400 that blames the caller; invalid_target is a gateway capability gap (RFC 8707 resource indicators, LIT-4339) and is gateway-blamed regardless of whose credentials were presented; the DCR classifier shares the same blame assignment. The gateway-fault arm is renamed GatewayRejected since it now covers capability gaps as well as stored-credential rejections * chore: add CODEOWNERS for ui and proxy UI build artifacts (#33131) * feat(ui): rebuild the Teams table on the shared DataTable (#33128) * feat(ui): rebuild the Teams table on the shared DataTable The Your Teams tab moves off the Ant Design table onto the shared DataTable that the Virtual Keys page uses, following the new dashboard design. It gains server-side sort, pagination and filtering, a toolbar with a filter drawer and a columns menu, and a per-row actions menu Sorting is wired only to the columns /v2/team/list can actually order by (team_alias, created_at); Spend / Budget and Updated stay unsorted because the endpoint silently ignores those fields. The design's "Created by" column is dropped since the team object has no such field, and the drawer's "Has keys" filter is dropped for the same reason. The Resources cell shows members, models and keys as colored pills, and the actions menu keeps the existing Edit, Copy team ID and Delete behaviors, with Edit and Delete gated to Admin The teams grid gets its own unit tests in TeamsPage/TeamsTable.test.tsx. Teams.tsx keeps the create-team modal, delete modal, detail view and tabs, now refreshing the list through React Query invalidation instead of a manual refetch * fix(ui): match Teams loading skeletons to the rendered row height The default twoLine and chips skeleton shapes rendered the Team and Resources cells shorter than the loaded row (a real row measures 55px, the old skeleton ~49px), so the loading state looked visibly squat. Give the Team column a custom renderSkeleton that mirrors the two-line IdentityCell (measured 54px) and the Resources column one that mirrors the pills, and mark the hidden Rate Limits column as two-line so it matches when shown * fix(ui): keep team admins' Members tab by deriving is_team_admin from the selected team The redesign computed is_team_admin from useTeam(selectedTeamId), but that hook returns teamInfoCall's nested { team_info: { members_with_roles } } shape, so the top-level members_with_roles read was always undefined and is_team_admin was always false. For a non-proxy-admin team admin that hid the Members, Member Permissions and Settings tabs in the team detail view, which broke the team-admin add/remove member e2e tests. Pass the Team object up from the table instead (/v2/team/list returns it with a top-level members_with_roles), matching the pre-redesign behavior; proxy admins were unaffected because is_proxy_admin already granted access Also point the Delete-a-team e2e at the new kebab: open the row actions menu, then click Delete team, rather than clicking the old inline delete icon * fix(keys): persist key_type so the UI shows correct key scope instead of "All Proxy Models" (#33115) * fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models' key_type is not persisted on a key (the proxy maps it to allowed_routes and drops it), so the keys tables only inspected the models list and rendered 'All Proxy Models' for any key with an empty models array, including SCIM, Management and Read-only keys that cannot call a single model. Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a scope tooltip for those recognized scopes; unrestricted, AI-API and custom keys keep the existing model-list rendering. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): move key_scope helper to components root Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(keys): persist key_type on virtual keys so the UI reads scope directly Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy, and proxy-extras schemas plus an additive migration) and stop dropping the value in handle_key_type, so management/read_only/llm_api/default keys store their type alongside the derived allowed_routes. Surface it on the key read and create response models. The dashboard now prefers the persisted key_type for the no-inference buckets and keeps the allowed_routes derivation as the fallback for keys created before the column existed (key_type null), so no backfill is required. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(keys): add key_type column to LiteLLM_DeletedVerificationToken The deleted-token archive model inherits key_type from the verification token, so regenerate/delete flows write key_type into LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration) so the archive insert does not fail with FieldNotFoundError. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(router): opt-in session affinity for complexity router (#33126) * feat(router): opt-in session affinity for complexity router Complexity router reclassified every turn, which could flip the routed model group mid-session and break provider-side prompt caching. Add a session_affinity config flag: when a session_id is resolvable, pin the model chosen on the first turn and reuse it for the rest of the session, skipping reclassification. Pinned turns still stamp the adaptive bandit's chosen-model metadata so reward feedback keeps working when adaptive=True. * fix(router): refresh session-affinity TTL on hit, scope pin by API key Two issues from review: the TTL was only set on the first classification, so an active session outliving session_affinity_ttl_seconds silently lost its pin instead of refreshing as documented. And the cache key was scoped only by session_id, which is client-supplied and unauthenticated, so two different callers reusing the same session_id could poison each other's routing pin. Refresh the TTL on every cache hit, and namespace the cache key by the proxy-derived API key hash when available. * feat(prometheus): expose video duration and image count consumption metrics (#33138) * test(e2e): otel trace completeness on /chat/completions (#33132) * test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage: a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export through a preset-owned provider - the code path where trace splits happen), a typed Jaeger query read-back client, and the first test: one successful non-streaming /chat/completions call exports ONE complete trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). * test(e2e): harden the otel trace read-back per review Jaeger reads now query server-side by the litellm.call_id span tag instead of paging recent traces and filtering client-side; the compose stack's background jobs alone can push a request trace past the page. A failed query hard-fails instead of reading as an empty result, the settle predicate now also waits for the prefix-matched db span the assertion demands, parent-chain walking follows CHILD_OF references only, the zero-trace and split-trace failures get distinct messages, jaeger gets a healthcheck so the depends_on condition is accurate, and the chat docstring names the route the code actually asserts * test(e2e): author the chat trace test docstring * Update logging section in CLAUDE.md Removed mention of OTEL trace-tree completeness from logging integration section. * fix(sso): paginate through all pages when fetching service principal group assignments (#33149) get_group_ids_from_service_principal only read the first page of the Graph API appRoleAssignedTo response, so tenants with more than 100 groups assigned to the enterprise application silently lost group memberships during SSO login. Loop over @odata.nextLink with the same MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already uses, and warn when the cap is hit. Ported from #32792 by @saisurya237 so CI can run. Fixes #32790 Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com> * test(e2e): otel trace completeness on /v1/messages (#33133) * test(e2e): OTEL trace completeness on /v1/messages Extends the LIT-3787 trace-completeness suite to the Anthropic-native route: one successful non-streaming /v1/messages call must land at the destination as ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT span, no dangling parents). Adds the raw /v1/messages sender to the logging suite client. * test(e2e): reuse the shared AnthropicMessagesBody per review Drops the duplicate /v1/messages request model in favor of the one models.py already provides (budget_client uses the same one), passes max_tokens at the call site to match the sibling chat test, notes in the docstring why the gen-AI span is named chat on this surface, and adopts the hardened read-back signature * test(e2e): author the messages trace test docstring * test(e2e): declare the messages surface on the covers marker * test(e2e): otel trace completeness on /v1/responses (#33134) * test(e2e): OTEL trace completeness on /v1/responses Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API route: one successful non-streaming /v1/responses call must land at the destination as ONE connected trace. Adds the raw /v1/responses sender, a CHEAP_OPENAI_MODEL config constant, and registers responses in the otel registry cell's exercised_on. * test(e2e): author the responses trace test docstring * test(e2e): declare the responses and chat surfaces on the covers markers * feat(ui): add adaptive routing settings to Auto-Router v2 (#33146) * refactor(mcp): extract the dcr_bridge token flow into bridge_token_flow.py discoverable_endpoints.py had grown to 2695 lines mixing FastAPI route handlers with the dcr_bridge token-flow logic, against the no-monster-files convention. This moves the bridge token flow (the litellm-key/user resolution, the SCIM revalidation gate, and the mint/refresh envelope logic with their types and error mappers) into a dedicated bridge_token_flow.py, leaving the route handlers and the shared exchange_token_with_server orchestrator in discoverable_endpoints.py importing from it Pure relocation, zero behavior change. The moved code is byte-verbatim except one type annotation quoted as a forward reference (_BridgeAuthorizationCode is used only for typing and imported under TYPE_CHECKING to avoid a cycle), and the new module imports nothing from discoverable_endpoints at runtime. 275 tests pass unchanged; the test patch targets for moved internals were repointed to the new module and verified to still apply * bump: litellm-enterprise 0.1.49 -> 0.1.50, litellm-proxy-extras 0.4.76 -> 0.4.77, litellm 1.93.0 -> 1.94.0 (#33229) * chore(deps): pin httplib2 and setuptools transitive floors (#33233) Raise the constraint floors for two transitive dependencies so resolution moves them to their latest maintenance releases: httplib2 0.31.2 -> 0.32.0 and setuptools 82.0.1 -> 83.0.0. Both are pulled in only by optional integrations (Google API client, grpc tooling, lunary observability, the nvidia-riva extra), all lower-bound only, so the floors stay inside every requirer's allowed range and a default install is unaffected * feat(ui): left-anchor the Create Key and Create Team CTAs (#33248) Move the Create New Key and Create Team buttons out of the page header's right-side action slot. On Teams the button now sits in the tab bar's left slot, separated from the three tabs by a vertical rule, so the CTA and tabs read as one left-anchored cluster. On Keys, which has no tabs, the button anchors left on its own row beneath the title. * fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models (#33244) * fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models * test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping * fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models Narrow the fix to the temperature reconciliation; the reasoning_effort budget cap is reverted because the live translation grid relies on budget_tokens >= max_tokens to reject unsupported effort tiers (xhigh/max) on budget-mode models, so capping turned those 400s into 200s. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook (#33136) * fix(guardrails): run apply_guardrail-style model-level pre_call guardrails at deployment hook * fix(guardrails): keep request-body dispatch predicate unchanged * fix(guardrails): fail closed when proxy extras are missing at deployment hook * fix(proxy)!: enforce user budget on team keys (read-time + reservation) with UI opt-out (#32005) * fix: enforce user budget on team keys User budget was skipped when the key belonged to a team, letting users exceed their personal budget by going through a team key. Remove the team_object guard in _user_max_budget_check so user budgets are always enforced. Add skip_user_budget_on_team_key general_settings flag to opt back into the old behavior. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: update test to expect user budget enforcement on team keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: assert budget_exceeded ProxyException in personal budget test Tighten the broad pytest.raises(Exception) so the test only passes when the auth flow rejects with a budget_exceeded ProxyException, and switch the new ConfigGeneralSettings field to Optional[bool] to match the surrounding annotation style * fix: revert to bool | None to stay under UP045 strict budget --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> * fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265) The rate-limited batch spend test snapshotted unattributed rows via the unpaginated /spend/logs whole-table read, which grows with the environment (58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an explicit date window instead, and SpendLogsParams now rejects a filterless read so the whole-table call cannot come back * refactor: make the code easier to read * feat(pricing): add gemini-omni-flash-preview with video output token pricing * fix(gemini): map video response modality instead of MODALITY_UNSPECIFIED * fix(anthropic): use native output capability (#33235) * fix(anthropic): route native structured output Use model capability metadata so new native structured-output models do not require transformation allowlist changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): pass provider to capability Co-authored-by: Cursor <cursoragent@cursor.com> * test(anthropic): cover dotted model IDs Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): handle remote capability lag Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): retry setup-uv installs to survive transient manifest fetch failures * docs(e2e): add cache_hit to the naming grammar assertion vocabulary * refactor(e2e): share anthropic cache-control shapes in endpoints_client * fix(proxy): never log raw virtual keys in key insertion debug output (#33268) * fix(proxy): never log raw virtual keys in key insertion debug output * fix(proxy): tolerate None token in insert_data debug log redaction * fix(auth): scope the JWT enterprise gate to actual JWTs (#33296) With enable_jwt_auth enabled but no enterprise license (premium_user False), the JWT premium check fired on every request before the token was inspected, so the master key, sk- virtual keys, and the encrypted CLI/UI SSO session token that `lite login` issues all 401'd with "JWT Auth is an enterprise only feature" and were never decoded. That broke `lite login`, `lite claude`, and the proxy master key on any deployment that turned JWT auth on without a license. Move the premium check inside the is_jwt branch so it gates only real JWTs. Non-JWT credentials fall through to their own auth paths regardless of license; actual JWTs still require premium, so the enterprise gate is unchanged for the feature it protects. * test(e2e): scope virtual keys to the deployment under test * fix(s3): sanitize slashes in response-id-derived object key file name (#33271) * refactor(ui): migrate guardrails table onto shared DataTable (#33303) * feat(ui): migrate guardrails table onto shared DataTable Move the guardrails list onto the shared DataTable + cell library as the proof-of-concept for the simple-tables design migration, following the Teams reference pattern. Split the table into a thin container (guardrail_table.tsx) and column defs (guardrailTableColumns.tsx): client-side sort defaulting to created_at desc, a search + refresh toolbar, IdCell / DateCell / StatusBadge cells, real provider logos, a rich empty state, and skeleton loading rows. Row actions move into a per-row overflow menu; deletion stays disabled for config-file guardrails, now surfaced as a disabled menu item instead of a greyed trash icon. Detail view and the delete modal remain owned by GuardrailsPanel. Restyle the "Add New Guardrail" control to the shared Button + dropdown menu. Update the regression tests for the menu-based actions and drop the now-stale eslint suppression entry that the rewrite eliminated. * fix(ui): match guardrails table to the design Address design-review feedback on the guardrails migration: - Drop the search + refresh toolbar. The original table had neither and the SimpleTable design has no toolbar; the container now just renders the sorted table and its empty state. - Give the Guardrail ID cell the design's hover affordance by rendering it with the shared IdentityCell (monospace, chevron on hover) instead of the blue IdCell pill. - Stop pinning the actions column. Pinning added a sticky divider that the design and the Teams table don't have; it is now a plain right-aligned menu column, matching Teams. * fix(ui): match loading skeleton row height to loaded rows The compact skeleton row did not carry the h-8 height that real compact rows get, so loading rows rendered shorter than loaded ones and the table height jumped when data arrived. Mirror the same size-based height on the skeleton row in the shared DataTable so every compact table loads at a stable height * test(ui): drop stale onGuardrailUpdated from guardrails table baseProps The prop was removed from GuardrailTableProps when the toolbar went away; the test baseProps still listed it. Harmless at the call site since it is spread rather than an object literal, but dead and worth removing * fix(ui): remove dead edit_guardrail_form after guardrails migration The guardrails table migration dropped the last import of EditGuardrailForm, which knip flags as an unused file. The form was already unreachable before the migration: the table wired a delete button only, and nothing ever called handleEditClick to open the modal, so the import was the sole thing keeping the file referenced. Delete it and prune its now-stale eslint suppression entry. Guardrail editing is unchanged and lives in the detail view (GuardrailInfoView) * feat(guardrails): streaming text transformation in generic_guardrail_api (#33110) * feat(guardrails): support streaming text transformation in generic_guardrail_api * chore(guardrails): address PR review feedback * fix(guardrails): fail closed on tool-call and prefix-rewrite leaks in streaming transform * fix(guardrails): address Bugbot review on streaming transform correctness * fix(guardrails): coerce holdback in handler for in-process guardrails * fix(guardrails): harden streaming transform (holdback coercion, tool-call passthrough, n>1 finish_reason) * test(guardrails): targeted _mode_matches coverage for all guardrail_mode shapes * fix(guardrails): inspect streamed tool calls and harden incremental_diff edge cases * test: move ComplianceChecker mode tests to the compliance PR * fix(guardrails): strip content from tool-call passthrough so streamed text can't bypass the transform * fix(guardrails): four correctness fixes for incremental_diff streaming path Four bug fixes on top of the OSS PR's incremental_diff streaming text transformation, all inside the incremental_diff code paths only. No existing block_only, non-streaming, or pre_call behavior is touched. Fix #1 — Mixed content+tool_call finish_reason ordering _tool_call_passthrough_chunk now takes an optional finish_reason_per_choice map. For a choice carrying both delta.content and delta.tool_calls, finish_reason is stripped from the passthrough and recorded on the map so the final synthetic text chunk delivers it. Without this, SSE-compliant clients stopping at finish_reason drop the guardrailed text — defeating the redaction the whole feature exists for. (Greptile P1 twice, Veria.) Fix #2 — Choice index sort in _process_streaming_transform indices/texts_to_check were derived from dict insertion order. For n>1 streams where choice 1 emits before choice 0, guardrail-returned texts aligned to the input order mapped back to the wrong choice indices on write-back — wrong text goes to wrong choice. Sort raw_by_index.keys() up front so realignment is deterministic. (Bugbot Medium.) Fix #3 — Cross-chunk pre-tool-call text flush With default streaming_sampling_rate=5, text chunks followed by a pure tool-call chunk carrying finish_reason='tool_calls' would emit the passthrough with finish_reason before any transformed text delta had fired. Same failure mode as fix #1 but cross-chunk. Now we flush any accumulated text via _round(is_final=False) BEFORE yielding the tool-call passthrough. (Greptile P1.) Fix #4 — Terminator chunk for deferred finish_reason on empty mutated_text _build_transform_chunk returned None early when mutated_text_per_choice was empty. If a mixed content+tool_call chunk had deferred its finish_reason (via fix #1) and the guardrail then suppressed the text (empty return), the deferred finish_reason was never delivered. Now on is_final=True with empty mutated_text_per_choice, we emit a terminator carrying finish_reason per choice from finish_reason_per_choice. (Bugbot High.) Also normalized Optional[X] → X | None across the OSS PR's added surface via ruff UP045 autofix to keep the strict-rule gate within budget. Pure mechanical typing style change, no semantic effect. Regression tests for all four fixes: - test_mixed_chunk_finish_reason_arrives_after_transformed_text (#1) - test_text_flush_precedes_tool_call_passthrough (#3) - test_final_finish_reason_flushed_when_guardrail_suppresses_text (#4) - test_transform_sends_texts_sorted_by_choice_index (#2) All fixes reachable only when streaming_transform_mode == 'incremental_diff' is configured (via _run_incremental_transform_stream) or when a StreamTransformSink is present (via _process_streaming_transform). Verified scope-clean: no changes to block_only, non-streaming, pre_call, moderation, or sibling guardrails. --------- Co-authored-by: Marton Schneider <marton@schneider.co.nl> * test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) * test(claude_code): move the Claude Code compatibility matrix under tests/e2e Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: restore the upload-coverage job dropped by mistake with the compat gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments * test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> Co-authored-by: tin-berri <tin@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Shivam Rawat <shivam@berri.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Thibault Serot <thibault@linktr.ee> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Or Gershoni <org@lasso.security> Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com> Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Marton Schneider <marton@schneider.co.nl> Co-authored-by: mateo <mateo@berri.ai>	2026-07-14 19:30:42 -07:00
Mateo Wang	8c776605d8	Merge pull request #33274 from BerriAI/litellm_gemini_omni_flash_preview_pricing ... feat(pricing): add gemini-omni-flash-preview with video output token pricing	2026-07-14 16:58:04 -07:00
Marty Sullivan	75faed1778	fix(bedrock_mantle): route xai.grok-4.3 via /openai/v1 frontier path (#33027) ... grok-4.3 is a third-party frontier model on Bedrock Mantle, served on the /openai/v1 base (like gpt-5.x and gemma-4), not the standard /v1 path used by open-weights models such as gpt-oss. #31916 added the model without use_openai_responses_path, so mantle_base_segment() routed it to /v1, where Bedrock rejects the call with "Berm is not enabled for this account" (access_denied) — the model is only reachable on the frontier /openai/v1 path. Add use_openai_responses_path=true to the bedrock_mantle/xai.grok-4.3 price-map entry (both model_prices_and_context_window.json and the backup) so mantle_base_segment() returns "openai/v1", and update the registry test to assert use_openai_path is True.	2026-07-14 16:38:13 -07:00
Krrish Dholakia	477ef3a7e2	fix(anthropic): use native output capability (#33235) ... * fix(anthropic): route native structured output Use model capability metadata so new native structured-output models do not require transformation allowlist changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): pass provider to capability Co-authored-by: Cursor <cursoragent@cursor.com> * test(anthropic): cover dotted model IDs Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): handle remote capability lag Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-07-14 14:23:49 -07:00
mateo-berri	598fa9d64d	feat(pricing): add gemini-omni-flash-preview with video output token pricing	2026-07-14 14:04:51 -07:00
Mateo Wang	8447cd3ad3	Merge pull request #32836 from BerriAI/litellm_gemini_image_supports_reasoning_31766	2026-07-11 22:50:33 -07:00
mateo-berri	2ed4ceb12e	fix(model-cost-map): anchor the bedrock-claude-ids routing rule to the start of the id	2026-07-11 19:00:03 -07:00
mateo-berri	6fa088224b	fix(fallback-generalizations): cover bare Claude majors in baseline and routing, require claude- prefix in adaptive gate	2026-07-11 10:42:47 -07:00
mateo-berri	1ccc3382d9	feat(fallback-generalizations): widen adaptive-thinking gate to any claude family at major 5+	2026-07-11 00:27:45 -07:00
mateo-berri	77885779ca	refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds	2026-07-11 00:27:45 -07:00
Mateo Wang	c15891fc98	fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882) ... Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache.	2026-07-10 22:51:41 -07:00
Mateo Wang	5e23a5ab05	fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831) ... * fix(bedrock): gate in-place system role messages on model support for Claude Invoke * feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule	2026-07-10 20:21:48 -07:00
Mateo Wang	4737e75c86	fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map (#32840) ... * fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map * test: use apac regional profile for cost-map fallback test since jp now has an entry	2026-07-10 20:05:37 -07:00
Deepanshu	fd862bb2b8	fix(model_cost): add supports_reasoning: false to gemini/gemini-3-pro-image	2026-07-10 22:04:02 +00:00
Deepanshu	aa717bc4d0	fix(model_cost): apply supports_reasoning: false to root pricing JSON ... The backup file is used by tests; the root model_prices_and_context_window.json is what gets published to the pricing URL and loaded by the proxy at runtime. Without this, the proxy would continue resolving supports_reasoning via the provider-level fallback and returning true for Gemini image generation models. Also covers vertex_ai/gemini-3-pro-image and vertex_ai/gemini-3.1-flash-image (non-preview variants) and gemini/gemini-3.1-flash-image which exist only in the root JSON.	2026-07-10 22:04:02 +00:00
devin-ai-integration[bot]	f90b3efb2e	feat(models): add Azure GPT-5.6 (sol/terra/luna) pricing and metadata (#32678)	2026-07-09 20:46:21 -07:00
devin-ai-integration[bot]	d82645d163	feat: add Meta Model API provider and muse-spark-1.1 (day-0) (#32701)	2026-07-09 20:45:27 -07:00
devin-ai-integration[bot]	a874de6ac6	feat(models): add GPT-5.6 (sol/terra/luna) pricing and metadata (#32659) ... * feat(models): add GPT-5.6 (sol/terra/luna) pricing and metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: allow gpt-5.6 service-tier cache-write keys in model prices schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: floating point entry errors --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-07-09 11:51:12 -07:00
devin-ai-integration[bot]	e1b9ec1cd6	feat(pricing): add xai/grok-4.5 model pricing and metadata (#32549) ... Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>	2026-07-08 17:37:38 -07:00
Mateo Wang	bd6cabee83	fix(model_prices): add gpt-realtime-2.1 models with regional processing uplift (#32387) ... * fix(model_prices): add gpt-realtime-2.1 models with regional processing uplift * fix(model_prices): add cache_read_input_audio_token_cost to gpt-realtime-2.1	2026-07-07 18:56:00 -07:00
Mateo Wang	43b0a25f07	feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support (#32274) ... * fix(llm_http_handler): send dict transcription request data as a JSON body httpx form-encodes dicts passed via data= and silently ignores json=, so the generic audio transcription path never actually sent a JSON body. No provider hit this before; JSON-body speech APIs need it. * feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support Adds a VertexAIAudioTranscriptionConfig wired through ProviderConfigManager so vertex_ai/chirp_3 works on /v1/audio/transcriptions (sync and async) via the Speech-to-Text v2 recognize API. Auth reuses the standard Vertex credential resolution (vertex_project/vertex_location/vertex_credentials or ADC); the location defaults to the us multi-region since chirp_3 is only served from the us and eu multi-regions, and non-global locations use the regional <location>-speech.googleapis.com host. Maps language to languageCodes (auto language detection by default), joins all result alternatives into the transcript, and tracks cost from totalBilledDuration with a vertex_ai/chirp_3 price entry at Google's published $0.016/min. * fix(vertex_ai): map bare ISO-639-1 language codes to BCP-47 for Speech-to-Text OpenAI clients send language codes like "en", which Google rejects with 400 ("not supported by the model chirp_3 in the location us"); Speech-to-Text wants region-qualified BCP-47 like "en-US". Adds a shared normalize_transcription_language_to_bcp47 helper in audio_utils (NVIDIA Riva's transcription config already hand-rolled the same table privately) that maps common bare codes and passes region-qualified ones through, and applies it in the Vertex transcription request. Also narrows the response JSON parse guard to ValueError. * fix(vertex_ai): drop zero output_cost_per_second so chirp_3 cost tracking works cost_per_second prefers output_cost_per_second whenever it is not None, so the 0.0 in the chirp_3 entry priced every transcription at $0.00 instead of using input_cost_per_second. Remove it from both cost maps and pin the behavior with a regression test computing 18s of chirp_3 audio to ~$0.0048. * fix(vertex_ai): validate client-controllable location to prevent SSRF in Speech-to-Text get_complete_url interpolated vertex_location straight into the request host, and vertex_location is client-controllable on the proxy (it flows from the request body and is not on the request-body blocklist). An authenticated caller could send vertex_location="attacker.example/" to point the host at their own server, so the proxy would POST the audio plus its admin-minted Google bearer token and x-goog-user-project header to the attacker, exfiltrating a cloud-platform-scoped OAuth token minted from the admin's credentials. Factor the location validation the rest of vertex_ai already applied in get_vertex_base_url (^[a-z][a-z0-9-]*$ plus the global allowance) into a shared validate_vertex_location helper in common_utils and call it from both the chat host builder and the new speech host builder. Invalid locations now raise a 400 VertexAIError instead of building a host. Also reject vertex_project values that carry URL-structural characters, since it lands in the URL path. Regression tests assert on the parsed netloc so the security property is pinned: valid locations always resolve to a *speech.googleapis.com host and injection inputs are rejected. * fix(vertex_ai): reject unsupported transcription response_format values instead of silently ignoring	2026-07-06 18:25:22 -07:00
devin-ai-integration[bot]	5cb0721f64	Merge pull request #32279 from BerriAI/litellm_azure_long_context_datazone_pricing ... feat(pricing): add azure data-zone and long-context pricing for gpt-5.4/5.5	2026-07-06 19:32:17 -04:00
Mateo Wang	8bb4e62412	feat(tencent): add Tencent TokenHub as a provider (#31903) ... * feat(tencent): add Tencent TokenHub as a provider Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a new provider: TencentChatConfig routes /v1/chat/completions and gates the thinking/reasoning_effort params behind supports_reasoning, and TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API. Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider endpoint support metadata. * test(tencent): add unit tests for Tencent TokenHub provider Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig (messages API) across transformation, param mapping, URL building, and header validation, plus get_optional_params routing. Tests mock supports_reasoning to stay independent of remote model cost data. * fix(tencent): correct max_output_tokens and reuse parent messages env validation Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling * fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix * fix(tencent): register tencent in models_by_provider The provider was added to the LlmProviders enum and cost map but not to the models_by_provider lookup, so test_models_by_provider (which asserts every litellm_provider present in the cost map is registered) failed once the tencent models were loaded. Add the tencent_models set, populate it from the cost map, and expose it under the tencent key, mirroring deepseek. * fix(tencent): import generic_cost_per_token from its canonical module Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils instead of the top-level litellm.cost_calculator dispatcher, which imports the tencent cost module at load time. Removing the back-reference avoids the circular import and matches how deepseek and the other providers source the helper. --------- Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-07-02 18:31:59 -07:00
Shivam Rawat	1543725916	fix(bedrock): honor ttl for tool_config cache injection points (#31929) ... * fix(bedrock): honor ttl for tool_config cache injection points Pass cache_control_injection_points control.ttl through to Bedrock toolConfig cachePoint blocks, matching message/system cache behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a manual update for every new Claude release (it already silently missed Sonnet 5 and Fable 5). Replace it with a lookup against cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json, which AWS docs confirm tracks the same 1h-TTL-capable model set. Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried that pricing field (their own regional variants didn't have it), which would have made the JSON-driven check wrongly grant them 1h TTL support. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id (...-20250514-v1:0) that never shipped. This passed under the old regex-based is_claude_4_5_on_bedrock, which matched on substring alone, but fails now that it looks up cache_creation_input_token_cost_above_1hr in litellm.model_cost, since the fake id has no pricing entry. Also force the bundled local cost map in both tests so ttl eligibility reads this branch's pricing data instead of the network-fetched main copy, which lacks the fix until merge. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): restore cache and tool config compatibility * fix(bedrock): preserve Sonnet 5 parallel tool config * fix(bedrock): decouple parallel tool support from cache ttl * refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and bedrock_converse_supports_strict_tool_schemas (dead code) with a supports_parallel_tool_use_config key in model_prices_and_context_window.json, matching how is_claude_4_5_on_bedrock already reads cache_creation_input_token_cost_above_1hr from the pricing JSON. New models pick up parallel tool use support automatically when their pricing entry ships with the key set, with no code change required Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in model_prices_and_context_window.json, so bedrock_converse_supports_parallel_tool_use_config returned False and the test died with KeyError on additionalModelRequestFields. Use jp.anthropic.claude-opus-4-7, a real entry that carries supports_parallel_tool_use_config without 1h-TTL cache pricing, which is exactly the decoupling this test exists to cover * test(utils): allow supports_parallel_tool_use_config in pricing schema The misc unit test job validates model_prices_and_context_window.json against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects unknown keys. Add the supports_parallel_tool_use_config key this PR introduced so test_aaamodel_prices_and_context_window_json_is_valid passes again * fix(bedrock): preserve ttl for regional claude models * fix(bedrock): fall back to base model entry when regional pricing lacks capability fields Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit cache_creation_input_token_cost_above_1hr shadowed the base entry that has it, so is_claude_4_5_on_bedrock returned False and requested cache ttl values were dropped for those deployments. Both capability lookups now consult the full model id and the region-stripped base entry, matching the coverage of the old name-pattern list. Also restores ToolBlock keyword construction for the tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every supported Python version --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-07-02 16:30:06 -07:00
Sameer Kankute	64dc5080b9	fix(bedrock): drop strict/additionalProperties from toolSpec for Claude Sonnet 4 (#31943) ... * fix(bedrock): drop strict/additionalProperties from toolSpec for Claude Sonnet 4 Claude Sonnet 4 on Bedrock Converse rejects toolSpec.strict and additionalProperties the same way Opus 4.7/4.8 do. Add bedrock_converse_supports_strict_tools: false to all Sonnet 4 regional variants so those fields are suppressed before the request is sent. Co-authored-by: Cursor <cursoragent@cursor.com> * test(bedrock): assert additionalProperties dropped for strict-unsupported models Rename the regression test to reflect Opus 4.7/4.8 and Sonnet 4 coverage, and assert both strict and additionalProperties are stripped from toolSpec. Co-authored-by: Cursor <cursoragent@cursor.com> * test(fireworks): skip embeddings live test when provider account is suspended --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-07-01 23:56:25 -07:00
Mateo Wang	85f924148a	fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8 (#31582) (#31923) ... * fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8 Bedrock Converse routes Claude Opus 4.7/4.8 through an Anthropic-compatible validator that maps toolSpec to the native tool shape and rejects the extra `strict` key with `tools.N.custom.strict: Extra inputs are not permitted`, even though Anthropic's native API accepts `strict` as a top-level tool field for the same models. Sonnet 4.5/4.6 and Opus <=4.6 accept `toolSpec.strict` unchanged. The existing gate `get_bedrock_base_model(model).startswith("anthropic")` (introduced in #29814 to forward `strict` for Claude on Bedrock Converse) is too broad and regressed Opus 4.7/4.8 callers — see #31582. Replace the inline check with a small `bedrock_converse_supports_strict_tools` helper that excludes the Opus 4.7/4.8 family from strict forwarding. All other Anthropic models on Bedrock keep the existing behavior. Closes #31582. * fix(bedrock/converse): move strict-tools regression to a clean test file The original regression test was added to test_litellm_core_utils_prompt_templates_factory.py, which has pre-existing ruff-format violations throughout (multi-line asserts that fit on one line). The lint workflow runs `ruff format --check` on changed files only, so touching that file surfaces those pre-existing violations and fails CI for unrelated reasons. Move the #31582 regression coverage into a new dedicated test file so the format check stays green. Also collapses the helper's `not any(...)` onto a single line to satisfy ruff format. Covers: #31582 * refactor(bedrock/converse): drive strict-tools gate from model cost map Replace the hardcoded Opus 4.7/4.8 pattern list with a bedrock_converse_supports_strict_tools flag on the affected entries in model_prices_and_context_window.json, resolved via get_model_info with a local cost map fallback, so future models with the same restriction only need a JSON update * chore: revert unrelated credential_migration.py reformat --------- Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>	2026-07-01 19:08:17 -07:00
Mateo Wang	6e023f7cf2	fix(model_prices): apply claude-sonnet-5 introductory pricing through 2026-08-31 (#31917) ... * fix(model_prices): apply claude-sonnet-5 introductory pricing through 2026-08-31 Anthropic launched Sonnet 5 with introductory pricing of $2/$10 per million input/output tokens through August 31, 2026 (sticker price $3/$15 applies from September 1, 2026). Bedrock, Vertex AI, and Azure Foundry mirror the introductory rate. LiteLLM was charging the sticker price on all ten claude-sonnet-5 entries, over-billing by 50% during the introductory period. Update input, output, cache write (5m and 1h), and cache read costs on the base entries to the introductory rate, and keep the 10% cross-region premium on the us/eu/au/jp Bedrock inference profiles on top of it. Also add an anthropic-sonnet-5 entry to the dev proxy config. * test: document exact sticker prices to restore on 2026-09-01	2026-07-01 17:45:57 -07:00
devin-ai-integration[bot]	7e993446d8	feat(bedrock_mantle): add xai.grok-4.3 to model cost map for SigV4 auth (#31916) ... Register bedrock_mantle/xai.grok-4.3 with /v1/responses in supported_endpoints so the data-driven gate routes it through BedrockMantleResponsesAPIConfig (which inherits SigV4 signing via BedrockMantleAuthMixin). Without this entry the model falls through to None and forces bearer-token-only auth. Pricing sourced from AWS Bedrock pricing page. Closes #31196 Co-authored-by: unknown <> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-01 15:44:30 -07:00
mateo-berri	6d43c21ec6	fix(anthropic): drop redundant supports_output_config from Vertex/Azure Sonnet 5 ... The Vertex AI and Azure AI Sonnet 5 entries carried supports_output_config: true, which the gen-5 siblings (vertex_ai/claude-opus-4-8, azure_ai/claude-fable-5, etc.) do not. The flag only feeds AnthropicConfig._model_supports_effort_param, which already returns true for these entries via supports_xhigh/max_reasoning_effort, so output_config.effort still forwards on both routes. Removing it is behavior neutral and matches the existing per-platform convention for gen-5 Claude.	2026-06-30 19:19:48 +00:00
Cursor Agent	a126cdf5b7	feat(anthropic): add Claude Sonnet 5 ... Register claude-sonnet-5 across the Anthropic, Bedrock (base + global/us/eu/au/jp cross-region inference profiles), Vertex AI, and Azure AI cost-map entries in both the root and bundled-backup model maps, plus BEDROCK_CONVERSE_MODELS and the setup-wizard provider list. Sonnet 5 ships with the gen-5 adaptive-thinking profile (adaptive thinking always on, no extended thinking, effort defaults to high), so the entries mirror the Fable 5 / Opus 4.8 sampling-param and prefill restrictions rather than the older Sonnet 4.6 behavior: supports_sampling_params and supports_assistant_prefill are false while supports_adaptive_thinking, supports_xhigh_reasoning_effort, and supports_max_reasoning_effort are true. Pricing follows standard Sonnet rates ($3 / $15 per MTok) with the 10% regional premium on the us/eu/au/jp profiles. Add a reasoning-effort grid entry for the Anthropic direct route and a regression test pinning pricing, capabilities, regional premiums, backup parity, and bare-name provider resolution. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-06-30 18:47:08 +00:00
Mateo Wang	b76a858826	feat: declarative fallback generalizations for unknown models (#29718) ... * feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match	2026-06-27 21:01:19 -07:00
Mateo Wang	ef3dcf91a2	chore: remove unused keys from model cost map (#31528)	2026-06-27 16:29:52 -07:00
Mateo Wang	64d8d7f8cb	fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke (#31364) ... * fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke * style(bedrock): use builtin generics in new Invoke helpers to clear UP006 gate * fix(bedrock): honor explicit thinking budget_tokens=0 in clear_thinking conversion The clear_thinking_20251015 -> adaptive conversion resolved the thinking budget with `thinking.get("budget_tokens") or BEDROCK_MIN_THINKING_BUDGET_TOKENS`, which treats a caller-supplied `budget_tokens=0` as missing and silently substitutes the Bedrock minimum. Resolve the budget with an explicit `is not None` check so an explicit 0 is honored. * fix(bedrock): gate Fable 5 into clear_thinking adaptive injection on Invoke _ensure_thinking_for_clear_thinking_context_management returns early when _supports_extended_thinking_on_bedrock(model) is False, so the adaptive-thinking injection never runs for models absent from that gate. Opus 4.8 slips through on the incidental "opus-4" substring, but Fable 5 had no matching pattern, so a clear_thinking_20251015 request on Fable 5 reached Bedrock with an unsupported context-management edit and no thinking field; the exact 400 this path exists to prevent. Add the fable-5 patterns to the gate so Fable 5 (mapped ids and unmapped aliases) gets thinking.type=adaptive + output_config.effort like the other adaptive models. Extend the adaptive-injection regression test to cover Fable 5 (a mapped id and an unmapped alias) so it fails without the gate entry, and add focused coverage for the budget->effort tiers, the disabled/enabled/adaptive thinking branches, output_config.effort preservation, and list/dict system-role normalization. Also normalize the Invoke transformation module and its test to line-length 88 so ruff format --check (CI format-check) passes. * refactor(anthropic): make supports_adaptive_thinking flag authoritative for thinking detection Replace the per-version name helpers (_is_claude_4_6/4_7/4_8_model, _is_claude_fable_5_model) with cost-map-flag-first detection. _is_adaptive_thinking_model now reads supports_adaptive_thinking from the model cost map and falls back to a single generalized family-version regex (_claude_version_at_least(model, 4, 6)) only when a model is unmapped, instead of hard-coding each new Claude release. Wire supports_adaptive_thinking through ProviderSpecificModelInfo and ModelInfo so the cost map flag actually surfaces at lookup time. Reroute the Bedrock Invoke extended-thinking gate and the two anthropic/chat/transformation.py call sites through _is_adaptive_thinking_model. Known gap left to the fallback_generalizations work (#29718): unmapped Fable 5 aliases have no parseable minor version, so they defer to the cost map and are not detected until a mapped entry or a generalization rule exists. Covered by an explicit regression test. * refactor(anthropic): drop name-based version fallback; resolve adaptive thinking from cost map only The prior commit kept a regex (_claude_version_at_least) as a fallback when an id resolved to no cost-map entry. Remove it: _is_adaptive_thinking_model now reads supports_adaptive_thinking and nothing else, so "which Claude versions think adaptively" lives entirely in the model cost map, and a new adaptive release is a JSON edit rather than a Python edit. To keep the flag authoritative across the id forms the Bedrock Invoke and anthropic paths actually see, backfill supports_adaptive_thinking=true on every adaptive Claude entry that was missing it (Opus 4.6/4.7 and Sonnet 4.6 across region/provider aliases) in both the root and bundled cost maps, and generalize _model_map_lookup_candidates to normalize an id to its base cost-map key: strip a Bedrock version suffix (-v1:0 fully, or just the :0 inference-profile minor so the -v1-keyed 4.6 entries resolve), strip a dated-release suffix (-20260219), and rewrite a dotted family version (4.6 -> 4-6). This is id normalization feeding the lookup, not capability-by-name. Tests load the PR-local cost map (the flags are not on main until merge) and cover each normalization path plus the unmapped-alias deferral to fallback_generalizations (#29718). * refactor(reasoning_effort): single-source effort<->thinking-budget mappings Route every reasoning_effort <-> thinking-budget conversion through the DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants so the numbers stay in sync across providers. The five constants are now 2000/5000/10000/20000/40000 Add reasoning_effort_from_thinking_budget() in litellm_core_utils/reasoning_effort_utils.py and route the three OpenAI-style forward maps (anthropic adapters, responses adapters, hosted_vllm) through it. The bedrock invoke and experimental messages adaptive maps now reference the constants directly; the only behavior change is the xhigh threshold moving from 24000 to 20000. Reverse maps and the cross-provider test grid read the same constants * test(reasoning_effort): lift budget-mode max_tokens above the new high budget The single-sourced DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET thresholds moved high from 4096 to 10000. The live reasoning_effort grid sends budget-mode requests with max_tokens=8192, so reasoning_effort=high now produces budget_tokens=10000 > max_tokens and every provider returns 'max_tokens must be greater than thinking.budget_tokens'. Derive a shared BUDGET_MODE_MAX_TOKENS (2x the high budget) for the spec and the request builder so the ceiling always clears the largest 200-expected tier. Also resolve the inherited base test_reasoning_effort assertion off the same high-budget constant instead of the stale 4096 literal so it tracks the source of truth. * fix(reasoning_effort): keep effort<->budget thresholds at pre-PR values The single-sourcing refactor moved the shared effort<->budget thresholds up (low 1024->2000, medium 2048->5000, high 4096->10000, xhigh 8192->20000, max 16384->40000). That silently changes the effort->budget direction: a caller who sets reasoning_effort together with a max_tokens that used to sit above the old per-tier budget but below the new one now trips the provider's "max_tokens must be greater than thinking.budget_tokens" 400. It spans every backend that derives a budget from an effort (Anthropic, Gemini/Vertex, hosted vLLM), not just Bedrock. Restore the constants to their pre-PR values while keeping every backend reading from the shared DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, so the mapping stays single-sourced without the behavior change. Tests that pinned the raised thresholds now derive their boundaries from the same constants. * test(reasoning_effort): derive high effort->budget assertions from the shared constant The cross-provider translation tests pinned reasoning_effort="high" to a literal budget_tokens=10000, the raised value. Point them at DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET so they track the single source instead of a magic number. * fix(anthropic): resolve adaptive flag for combined dated+versioned Bedrock ids The model-map candidate normalization applied each suffix strip independently to the original id, so the real Bedrock shape "<base>-<YYYYMMDD>-v1:0" never reduced to its base cost-map key: stripping the version left the date, and the dated-suffix regex is anchored to the end so it could not fire while the version was still present. An adaptive Claude model invoked by its full dated+versioned id (e.g. us.anthropic.claude-sonnet-4-6-20251101-v1:0) therefore resolved to supports_adaptive_thinking=null and was treated as non-adaptive, reaching Bedrock with the rejected thinking.type=enabled shape, the exact 400 this path prevents. Add a composed normalization that rewrites the dotted family version, then peels the -vN:rev version suffix, then the -YYYYMMDD dated suffix, so the combined form resolves to its base key. Regression tests pin the combined suffix on sonnet-4-6 and opus-4-8 across provider/region prefixes. * fix(reasoning_effort): align budget<->effort tests with reverted constants and format common_utils The constant revert restored the effort<->budget thresholds to their pre-PR values (1024/2048/4096/8192/16384) and single-sourced the reverse budget->effort ladder through reasoning_effort_from_thinking_budget, but several tests still pinned the briefly-raised values and the old hardcoded reverse buckets, so the "All Other Providers" shard failed Derive the anthropic chat effort->budget assertions from the shared DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, and update the experimental pass-through and responses adapter expectations to the single-sourced reverse ladder (budget 1024 -> low, 5000 -> high) Also run ruff format --line-length 88 over anthropic/common_utils.py so the CI format-check, which checks the whole changed file, passes	2026-06-27 11:35:36 -07:00
Mateo Wang	5a1c7839be	feat(mistral): add mistral/mistral-ocr-2512 (OCR 3) to cost map (#31463) ... Adds the OCR 3 model (mistral-ocr-2512) released 2025-12-18 to both the root and bundled backup cost maps at $2 / 1000 pages and $3 / 1000 annotated pages, mirroring the existing Mistral OCR entries. Regresses the pricing in both maps and verifies completion_cost scales per page.	2026-06-26 10:29:07 -07:00
Sameer Kankute	4476923ac4	test: add realtime proxy e2e suite across providers (#30960) ... * tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * test: multi-window budgets coverage * fix: p0 issues, added types and shared functions for each test suite * chore: add config.yml * test: passthrough endpoints stream/non-stream e2e * style: carry clearer status_code comparison into renamed e2e dir * fix: rename cost breakdown function * fix: pydantic validation for budget info, dont allow explicit type cast * refactor: migrate to gateway client * test: add custom pricing tests * chore: change master key * test(e2e): address greptile review feedback Remove the duplicate cache/cache_params block in the gateway config so the two can't silently diverge under future edits. Reorder the soft-budget test to assert the call isn't a budget block before require_successful_call, since that helper hard-fails any non-2xx and left the budget-block check unreachable; the misleading "skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it so a failed delete doesn't leak a budget on the shared proxy. Scope the spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup import so a broader "pytest tests/" run isn't left with a mutated path. * test(e2e): drop misleading skip comment on require_successful_call require_successful_call fails hard, it does not skip; the trailing comment was factually wrong. The function name already states intent, so the comment is removed in both per-model and tag budget helpers. * test(e2e): assert budget-isolation invariant before success check On the should-still-succeed path of the per-model and tag isolation tests, check is_budget_block before require_successful_call. If the isolation bug fires the unaffected model/tag is blocked, so asserting the specific 'blocked by X' invariant first yields the diagnostic message instead of a generic upstream-failure. Matches the ordering in test_soft_budget_e2e.py. * fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows * fix(e2e): run case init() inside try so partial-init failures tear down run_case called case.init() outside the try/finally that runs teardown(), so a case that registers cleanups progressively (create team, then user, then key) and then fails partway through init() would leak the already-created entities on the long-lived shared proxy. Move init() inside the try so teardown always runs. Add a regression test that registers a cleanup then raises mid-init and asserts the resource is still released. * test(e2e): mark known pricing-leak isolation test xfail(strict) test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy gap (a deployment's custom per-token pricing leaks into the shared cost map for sibling deployments of the same underlying model) and was left unconditionally failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True) so the suite stays green while the leak persists and turns into a failure the moment isolation is fixed, prompting the marker's removal. * refactor(e2e): make suite pass its shipped strict basedpyright config The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright --project tests reported four errors in it: three reportAny on the parametrize ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed autouse fixture _require_live_proxy. Replace the untyped lambda with a typed _case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and rename the fixture to require_live_proxy so basedpyright no longer treats it as an unused private function (it is referenced only by pytest's autouse machinery). basedpyright --project tests now reports zero errors. * fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory * test(e2e): run harness unit tests without a live proxy The autouse session fixture skipped the whole tests/e2e session when no proxy answered, which also skipped test_lifecycle.py, a pure unit test of run_case that never touches the proxy. A regression test that silently skips gives no signal, so the skip now lives in pytest_runtest_setup gated on the same e2e marker the spend-log truncate guard already uses: live tests skip when no proxy is up while harness unit coverage always runs. The liveness probe is cached with lru_cache so it still runs once per session * test(e2e): clean up gateway config comment debris Fix the typo on the header comment and drop the orphaned namespace/ttl comment remnants left indented under cache_params; the active values are already set above. Flagged by greptile review. * fix: add new tests, split gateway * test(e2e): type the redis spend-counter probe for strict basedpyright The new cold-counter reseed test drove its redis client untyped, so the strict tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the file landed: scan_iter/get came back unknown and the pool.map lambda had an untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING import (the runtime import stays lazy so the suite still skips, not errors, when redis is absent), which resolves scan_iter to Iterator[str] and get to str | None, and replace the lambda with a typed inner function mirroring _burst. basedpyright --project tests is back to zero errors. * test(e2e): xfail the known team multi-window failure and isolate member teardown Greptile flagged two issues in the mirrored split-gateway commit. The team multi-window budget test documents a real /team/new write bug (budget_limits go straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and /team/update paths) and was left as an unconditional hard failure, which would turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing isolation test so the suite stays green while the bug persists and flips to a failure the moment the write is fixed and the marker should go. The class-scoped member fixture in test_team_member_budget_e2e.py tore down its key, user, and team sequentially with no exception isolation, so a failed delete_key would strand the user and team on the long-lived shared proxy. Route cleanup through a ResourceManager: register each delete progressively and run them LIFO best-effort in a finally, so a partial-setup failure still releases what came before and one failed delete never blocks the rest. * test: add realtime proxy e2e suite across providers Add tests/realtime_e2e covering the proxy realtime websocket endpoint end to end against live providers (openai, azure, gemini, vertex_ai, bedrock, xai). Two layers: a raw-websocket suite asserting the normalized OpenAI GA event sequence, delta/transcript consistency, usage, and a full tool-call round-trip; and a pipecat smoke driving the proxy through the GA OpenAIRealtimeLLMService. Tests carry a new realtime_e2e marker and skip cleanly when the proxy or provider creds are absent, so they stay out of the default unit run. * test: move realtime e2e suite into tests/e2e harness Replace the standalone tests/realtime_e2e with a tests/e2e/realtime suite that follows the existing e2e conventions: a session-scoped client fixture, a frozen-dataclass RealtimeClient wrapping the shared Gateway, pydantic models for every sent and received event, and the e2e marker with the parent harness's liveness skip. The suite opens the proxy realtime websocket (websockets.sync to stay synchronous like the rest of the harness) and asserts the normalized OpenAI GA event sequence for a text conversation plus a full tool-call round-trip, parametrized across providers. A provider whose realtime alias is not configured on the proxy skips via /model/info. Adds a gemini realtime model to the gateway config and fixes the openai realtime model id. * test: add pipecat realism layer to realtime e2e suite Add test_realtime_pipecat_e2e driving the same providers through pipecat's GA OpenAIRealtimeLLMService with base_url pointed at the proxy, as a coarse realism check on top of the raw-websocket suite. Each test stays synchronous and runs the async pipecat pipeline via asyncio.run, and the module skips unless pipecat-ai is installed. Lift the shared provider matrix, ws-url helper, and skip helper into realtime_client so both suites use them. * fix(e2e): parse GA realtime transcript events in e2e client The realtime e2e client speaks the GA protocol, but transcript() only aggregated beta delta event names. Handle GA deltas, fall back to response.done output, and accept nested usage details on response.done. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): address realtime code-review findings - Use the real openai/gpt-4o-realtime-preview model ID in the gateway config (gpt-realtime-2 does not exist and would fail every live test) - Pass a bare base_url to pipecat's OpenAIRealtimeLLMService so pipecat can append ?model= itself; the previous realtime_ws_url already contained ?model= causing a malformed duplicated query parameter - Wrap connection.recv() in a try/except TimeoutError in collect_until so a deadline expiry inside recv preserves the collected-events diagnostic instead of raising a bare, message-free exception Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): filter configured_models to mode:realtime entries only ModelInfoEntry.model_info used CustomPricing (extra="ignore") so the mode field from /model/info was silently dropped, making it impossible to distinguish realtime from non-realtime deployments. Add an optional mode field to CustomPricing and filter configured_models() to entries whose model_info.mode == "realtime" so skip_if_unconfigured never accidentally skips a realtime test due to a naming-pattern collision with a non-realtime deployment. Co-authored-by: Cursor <cursoragent@cursor.com> * Update litellm-config.yml * fix(e2e): use TypeVar instead of PEP 695 generic in realtime parse_last PEP 695 type-parameter syntax (def f[T: Bound](...)) is only parseable on Python 3.12+, but the project declares requires-python >=3.10. Importing the realtime e2e client on 3.10/3.11 raised a SyntaxError before any test could run. Switch parse_last to the backport-safe TypeVar idiom so the suite imports across the full supported range. * fix(e2e/realtime): use GA openai/gpt-realtime model id The realtime gateway config used openai/gpt-realtime-2, which is not a real OpenAI model id and would 404 once live OpenAI realtime credentials are wired in. The GA speech-to-speech model is openai/gpt-realtime (snapshot gpt-realtime-2025-08-28); switch the openai-realtime alias to it. * fix(realtime): harden Gemini/Vertex Live for audio-native e2e Coerce TEXT responseModalities to AUDIO on native-audio and flash-live models, suppress the orphan turnComplete response.done that arrives immediately after tool results, omit function_response.id on Vertex, stop appending client query params to Gemini/Vertex WSS URLs, and add regression tests for these paths. Co-authored-by: Cursor <cursoragent@cursor.com> * Add xai full compatibility * Add working vertex ai realtime tests * Add audio + server vad e2e tests * Add config for e2e testing models * Add fix xai server vad * fix: use correct OpenAI realtime model ID in e2e gateway config openai/gpt-realtime is not a valid model; replace with the correct openai/gpt-4o-realtime-preview model ID to prevent model-not-found errors when running the openai-realtime e2e tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert: restore openai/gpt-realtime model ID gpt-realtime is a valid model; reverting the unnecessary change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve UP006 violations, mock test failures, and stale spec field - Guard gemini setup-without-tools deferral with litellm.gemini_live_defer_setup flag so the default (False) path sends setup immediately, fixing two failing mock tests: test_client_ack_caches_setup_to_prevent_duplicate_session_update_setup and test_deferred_setup_sends_session_update_before_buffered_audio - Replace deprecated typing generics (Dict, List, Tuple, Optional) with builtin equivalents in xai/realtime/transformation.py, gemini/realtime/transformation.py, and realtime_streaming.py to satisfy the UP006 ruff-strict ceiling - Remove 'role' from OpenAPI compliance test expected fields; Google removed it from the Interaction schema in their live spec Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use Optional[dict] in xai normalizer to preserve Black line-split dict[str, Any] | None is shorter than Optional[Dict[str, Any]] by enough that Black collapses the _normalize_usage signature to a single line (86 chars), conflicting with the existing multiline format. Using Optional[dict[str, Any]] keeps the line at 90 chars (> 88 limit) so Black preserves the multiline shape, while still satisfying UP006 by replacing Dict with dict. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove proxy-level setup-tools deferral, delegate to transformer The _gemini_setup_deferred / _gemini_pre_setup_buffer block in _send_to_backend was double-deferring: GeminiRealtimeConfig already handles the session.update-to-setup mapping internally and always returns a ready-to-send setup on the first session.update call (session_configuration_request=None). The proxy layer was incorrectly holding back that setup waiting for tools that the transformer had already incorporated. Removing the block fixes two failing tests: test_client_ack_caches_setup_to_prevent_duplicate_session_update_setup test_deferred_setup_sends_session_update_before_buffered_audio Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: abstract Gemini protocol keys out of core and use cost map for live model detection Move Gemini-specific message key knowledge (setup, realtimeInput, clientContent, toolResponse) out of the core RealTimeStreaming module into provider-level methods. BaseRealtimeConfig gains is_setup_message and is_content_message (both default False); GeminiRealtimeConfig overrides them with the actual Gemini key checks. Add gemini_native_audio and gemini_audio_only_live capability flags to the 10 affected model entries in the cost map. _is_audio_only_live_model and _is_native_audio_model now read from the cost map first and fall back to the existing string markers for models not in the map. * fix: apply black formatting and register gemini capability fields in schema * refactor: drop string-marker fallback; resolve audio-only live models via cost map only * fix: use registered cost-map model name in vertex realtime tests * fix: patch cost map in tests so they don't depend on remote main branch state * fix: align gateway config vertex-realtime model ID with cost-map registered name * fix: patch gemini-2.5-flash-native-audio in cost map fixture for CI * fix(e2e): use correct OpenAI realtime model id in gateway config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(e2e): add budget rescheduler short intervals to gateway config Without proxy_budget_rescheduler_min/max_time set, the rescheduler defaults to ~600s, causing all budget-reset e2e tests to timeout before the reset fires. Set to 5–10s so tests complete within 90s. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(e2e): strip non-realtime files from PR scope Restore budget, spend-tracking, and custom-pricing test files to their litellm_internal_staging state. Keep the mode field addition to CustomPricing in models.py (needed by realtime configured_models filter). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): restore async_realtime regression test and add missing fixture - Restore the end-to-end async_realtime regression test for Vertex query-param forwarding; the previous unit-only version did not exercise the code path where the original bug lived - Add patch_gemini_audio_cost_map_entries fixture to test_gemini_audio_only_live_models_drop_text_from_text_audio_combo so it does not depend on the cost map having gemini_audio_only_live set in CI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): resolve ANN401 violations in realtime streaming code Define RealtimeEventNormalizer Protocol and replace bare Any annotations with typed alternatives (object for event/value params, the Protocol for the normalizer) to stay within the strict-rule budget. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: black format realtime_streaming.py * fix(tests): add gemini_native_audio and gemini_audio_only_live to model prices schema * fix(lint): fix I001 import sort order in realtime_streaming.py * fix(lint): restore import litellm to correct position before from-litellm imports * undo budget removal * test(e2e): pin explicit credentials for gemini and vertex realtime models * test(e2e): share keepalive-safe LiteLLMRealtimeLLMService across pipecat suites The pipecat smoke test drove the proxy through the stock OpenAIRealtimeLLMService, which sends websocket keepalive pings at its default interval. The proxy does not answer them, so the connection is closed with a 1011 before the run completes. Move the proxy-aware LiteLLMRealtimeLLMService (keepalive disabled) into a shared pipecat_service module and use it from both the smoke and audio suites. * test(e2e): document that LiteLLMRealtimeLLMService._connect keeps the ?model= param The proxy routes realtime websockets on the ?model= query param, and pipecat's OpenAIRealtimeLLMService.__init__ bakes it into self.base_url before _connect runs. Passing self.base_url through preserves it; spell that out so the override is not misread as dropping the param. * fix(realtime): set _content_sent_after_setup only after the backend send succeeds A failed content send used to flip _content_sent_after_setup to True before the send was confirmed, mirroring the correct-on-failure ordering the adjacent session-config cache already follows. If the send raised, the flag stayed True and a later session.update that produced a setup frame was silently dropped even though the backend never received any content. Set the flag after the send succeeds and add a regression test that fails if the ordering is reverted. * fix: normalize realtime passthrough events * refactor(realtime): declare patch_outgoing_session on normalizer Protocol; fix wav chunk return type The RealtimeEventNormalizer Protocol only declared should_drop and normalize, so the outgoing session.update patch went through a getattr(..., None) lookup even though should_drop/normalize are called directly. The sole implementer (XAIRealtimeNormalizer) already provides patch_outgoing_session, so declare it on the Protocol and call it directly for consistent, fully-typed dispatch. Also correct _load_wav_chunks' return annotation from list[bytes] to tuple[list[bytes], int]; it returns (chunks, sample_rate) and the caller unpacks both. --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-26 09:36:49 -07:00
Sameer Kankute	133da06aa3	chore: litellm oss staging (#31185) ... * fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped The Total Tokens Over Time and Total Requests Over Time AreaCharts on the Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow once totals pass the hundred-million mark — leading digits of labels like "100.00M" / "4500.00M" got clipped against the chart edge. The requests chart was worse: it formatted with toLocaleString(), so billion-scale request counts produced "1,000,000,000" (13 chars) and overflowed immediately. Fix in two places so neither alone has to carry the whole margin: - activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and switch the requests chart to the shared valueFormatter so it uses the same compact k/M/B suffixes as the tokens chart. - value_formatters.tsx: add a >= 1e9 branch to valueFormatter / valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping every formatted label at most 7 chars. Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com> * Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * docs(readme): add Deploy on AWS/GCP with Terraform section Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): add 1-click deploy buttons for AWS + GCP GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): move AWS + GCP deploy buttons next to Render button * docs(readme): unify deploy button sizes and badge styles * docs(readme): bump deploy button height to 48 to match Render/Railway * docs(readme): bump AWS/GCP badge height to compensate for SVG padding * docs(readme): bump AWS/GCP badge height to 72 * docs(readme): bump AWS/GCP badge height to 84 * fix(readme): make deploy buttons same height (48px) https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc * docs(readme): flag GCP project ID substitution in image_registry * docs(readme): equalize deploy button heights and fix Cloud Shell button font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface. * docs(readme): collapse Railway deploy anchor to a single line The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline. * Add Claude Fable 5 cost map entries as a data-only hotfix Backports only the model map changes from #30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano Three bugs in model_prices_and_context_window.json: 1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K max output, but the values were set as max_input=128000, max_tokens=272000. This caused token limit errors when sending prompts over 128K tokens to GPT-5 Pro. 2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was 272000, but GPT-5.4 Mini shares the same 1,050,000 token context window as GPT-5.4. This was inconsistent with the azure/ variants which already correctly had 1,050,000. 3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini, max_input_tokens was 272000 instead of 1,050,000. Source: OpenAI model documentation and contextwindows.dev which aggregates official context window sizes. Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini should be 400K; their 272K values are correct per OpenAI docs) * fix: also correct max_output_tokens for gpt-5-pro (272000→128000) Per reviewer feedback, max_output_tokens was left at 272000 while max_tokens was corrected to 128000, causing an internal inconsistency. Both should be 128000 per OpenAI docs. * fix(cost): price gpt-image generated output tokens as image tokens (#31147) The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return usage with no output token breakdown — litellm's `ImageUsage` has no `output_tokens_details` field — so generated-image OUTPUT tokens were priced at the text rate (`output_cost_per_token`) instead of the image rate (`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x undercount on the dominant cost component (image output is ~74% of spend). This also affects azure gpt-image, which shares this calculator. The OpenAI gpt-image cost calculator re-implemented usage handling instead of reusing `calculate_image_response_cost_from_usage`, the shared helper that azure_ai/gemini/vertex_ai already use. That helper classifies generated output tokens as image tokens when the provider does not itemize output, and splits text/image when it does. Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage` (pre-transformed chat Usage objects are still costed directly). Adds a regression test for the no-breakdown ImageUsage case (gpt-image-2). * fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098) A bare application-inference-profile ARN passed as bedrock/arn:... fell through to the invoke route, which cannot derive a provider from the opaque profile id and raised 'Unknown provider=None'. The converse route needs no provider, so detect these ARNs in get_bedrock_route and route them to converse, matching the behavior of the already-documented bedrock/converse/arn:... workaround. Explicit invoke/ prefixes still win, and they remain a dead end for these ARNs by design (no provider derivable). System-defined inference-profile ARNs that embed a known model, and other opaque ARN types (provisioned-model, imported-model, custom-model-deployment) that are frequently invoke-only, are deliberately left on their current routes; tests guard both boundaries. * fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060) _add_tool_choice_required_message appended the "select a tool" prompt to the caller's messages list in place, so transform_request corrupted the caller's conversation history and appended a duplicate prompt on every retry. Build and return a new list instead so the call stays idempotent. Adds a regression test asserting the input messages list is unchanged across repeated transform_request calls. Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> * fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996) gpt-4o-transcribe and compatible ASR backends return a diarized_json response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8. TranscriptionUsageDurationObject typed seconds as int, so parsing the response raised a pydantic ValidationError (int_from_float). That error surfaces as an APIConnectionError which the router treats as retryable, so it keeps re-calling the upstream (200 every time) until the upstream rate-limits and returns 429 to the caller. OpenAI specs this field as a float (see openai SDK UsageDuration.seconds), so widen seconds to float. With the parse succeeding there is no exception left to retry, which removes the loop. Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> * fix(deepseek): drop non-function tools before chat completions call (#30910) * fix(deepseek): drop non-function tools before chat completions call DeepSeek's /chat/completions only accepts tools of type "function". Requests bridged from /v1/responses can carry responses-API-native tool types, for example a Codex CLI tool typed "namespace", which DeepSeek rejects with "unknown variant 'namespace', expected 'function'" so the whole request fails (issue #30722). Filter unsupported tool types in the DeepSeek request transform so the function tools still go through; when nothing callable remains, also drop the now-dangling tool_choice and parallel_tool_calls Fixes #30722 * test(deepseek): cover async tool filtering and document tool_choice assumption Add an async_transform_request regression test so the sync and async tool filtering paths cannot silently diverge, and document in _drop_unsupported_tools that only non-function tools are dropped, so a function-named tool_choice always references a surviving tool * feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840) * feat(ui): surface team budget on key overview when key has no own budget (#30801) * feat(ui): surface team budget on key overview when key has no own budget * fix(ui): replace IIFE with derived variable and use find() for team budget display * fix(anthropic): emit replayable streaming thinking blocks (#31022) * feat(proxy): read cold-storage prompts back in the logs detail view (#30364) * feat(proxy): read cold-storage prompts back in the logs detail view When a deployment offloads prompts and responses to cold storage instead of Postgres, the spend-log row holds only "{}" placeholders plus a metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed nothing. The detail endpoint only read the placeholder columns and never fetched the object back. Resolve the payload per row based on actual content, not a config flag: if Postgres has content, return it; otherwise read the exact stored object key and fetch from the configured cold storage backend through ColdStorageHandler. Reading the persisted key is a single GET. The key embeds a microsecond timestamp that cannot be reconstructed from the millisecond-precision startTime column, and listing the day's prefix to match on request_id would be too expensive for this per-open path. Also teach the detail drawer's pretty-view parser to accept a bare messages array. The cold storage payload carries the prompt as a top-level messages list with no proxy_server_request, so without this the output rendered while the input stayed blank. ColdStorageHandler gains an optional injected logger so the resolver can be unit tested without monkeypatching. Postgres-stored prompts are unaffected: the fast path returns the existing columns and the request-body object still renders the same way. * Update litellm/proxy/spend_tracking/spend_management_endpoints.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure Add unit tests for ColdStorageHandler (injected logger, graceful None when no logger is configured, and resolution of a configured logger from the callback registry) and a regression test asserting a cold storage backend exception degrades to the Postgres values instead of surfacing a 500. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068) * fix(mavvrik): advance metricsMarker after upload + fix scheduler startup Two bugs fixed: 1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a successful GCS upload, so metricsMarker stayed at 0 and every daily run re-exported the same dates in an infinite catch-up loop. Fix: add _update_metrics_marker(date_epoch) called at the end of deliver() after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS file is already committed). A 410 raises consistent with the rest of the destination. 2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call has triggered lazy instantiation of MavvrikFocusLogger, so it found no logger instance and silently skipped registering the daily export job. Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call _init_custom_logger_compatible_class to force instantiation before the APScheduler job is registered. * fix(mavvrik): catch up from earliest window when metricsMarker=0 When the connector is freshly registered, metricsMarker=0 parses to None. The catch-up block was guarded by `if last_ingested and ...` which skipped it entirely for None, so only yesterday was exported instead of the full _MAX_CATCHUP_DAYS window. Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup). The existing > 7 day warning only fires for non-None markers that are old. * fix(mavvrik): use now as end_time for yesterday's export window LiteLLM_DailyUserSpend rows for a given date get their updated_at bumped by the spend flush job throughout the next morning. The core database query filters on updated_at, so capping end_time at midnight (yesterday + 1 day) missed any spend rows flushed after midnight. Fix: pass now (cron fire time) as end_time for the daily "yesterday" window so all fully-settled rows are captured regardless of when the flush job ran. Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per row in the exported FOCUS CSV. * fix(mavvrik): also use now as end_time for catch-up windows * fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class Calling it with only logging_integration raised TypeError at proxy startup because internal_usage_cache and llm_router have no defaults. Also fix test name to reflect the actual status code (5xx not 4xx) used in the mock. * ci: retrigger CI run * feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757) * Add optional `instruction` passthrough to the rerank API vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction` field (folded into the model's chat_template_kwargs and consumed by the chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently dropped it: RerankRequest / OptionalRerankParams had no such field, so the outgoing body was rebuilt without it. Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(), get_optional_rerank_params, and the hosted_vllm transformation into the request body, only when non-None. When callers omit it, model_dump(exclude_none) drops the field and the outgoing request is byte-for-byte unchanged — fully backward-compatible. (DeepInfra already forwards `instruction` via non_default_params; this formalizes the field in the shared types.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: thread `instruction` as a typed param + cover rerank_utils Per PR review (greptile P2 + codecov): - Make `instruction` a typed, named argument on the rerank provider interface instead of recovering it from the opaque `non_default_params` blob. Adds `instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params` and every provider override, and forwards it explicitly from `get_optional_rerank_params`. hosted_vllm now reads the named param directly. It is still also surfaced in `non_default_params` so providers that read it there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction` as a named param rather than leaving it in **kwargs. - Add get_optional_rerank_params unit tests (present + absent) to cover the previously-uncovered threading line flagged by codecov. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan rerank `instruction` through request guardrails The rerank guardrail translation (CohereRerankHandler.process_input_messages) only scanned `query`, so the newly added `instruction` field reached the backend model unscanned. Since instruction-aware rerankers (hosted vLLM / Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller could place content there to bypass configured rerank request guardrails. Generalize the handler to scan every user-controlled text field (`query` and `instruction`) in one apply_guardrail call and write each sanitized value back by index. Query-only requests are unchanged (single-element list at index 0); non-string fields are left untouched. Adds tests covering instruction scanning, PII masking write-back, and the non-string case. Addresses the Veria AI security review on PR #30757. * test: narrow Optional results before len() to satisfy basedpyright budget The lint gate (basedpyright delta-vs-base budget) flagged one new reportArgumentType: len(result.results) where results is List[RerankResponseResult] | None. Assert results is not None first to narrow the type before len()/indexing. * fix: read rerank `instruction` from kwargs to satisfy basedpyright budget The basedpyright delta-vs-base gate flagged one new reportArgumentType: the Router forwards rerank calls via an untyped `**kwargs` unpack (`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a typed named param on the public `rerank`/`arerank` entrypoints made pyright check that key against `str | None`, adding an error at router.py with no real safety gain. Read `instruction` from kwargs in `rerank` instead. It remains fully typed where it matters - threaded as a typed argument through `get_optional_rerank_params` and each provider's `map_cohere_rerank_params` (the original Greptile P2 ask). Whole-repo reportArgumentType is back to the base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(github_copilot): synthesize empty choices at the provider seam (#30929) Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with choices=[], either carrying Anthropic-native content blocks or, for the max_tokens=1 probe Claude Code sends, no content at all. github_copilot is dispatched through the OpenAI SDK handler, which calls convert_to_model_response_object directly and never invokes GithubCopilotConfig.transform_response, so the empty-choices guard there surfaced as a 500 Instead of synthesizing choices inside the shared convert_to_model_response_object (which would silently turn empty choices into a fabricated success for every provider), add a no-op transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig overrides it to synthesize choices from Anthropic-native content, reusing its existing parsing, and the OpenAI SDK handler routes its parsed response through the hook before generic conversion. The core utility keeps treating empty choices as an error for all other providers Fixes: https://github.com/BerriAI/litellm/issues/30927 Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> * fix(router): stop fallback lookups from mutating the router fallbacks config (#30624) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens * test: scope local cost map env var with monkeypatch to avoid test pollution * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764) * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold _mask_value did partial reveal by showing the first visible_prefix and last visible_suffix characters, but for a value whose length was at or below visible_prefix + visible_suffix (8 by default) it returned the value verbatim. A value of exactly 8 chars fell through the length guard and computed masked_length == 0, reconstructing the original string with no mask characters; anything shorter hit the early return. Either way short credentials were emitted in plaintext. mask_dict routes real secrets through this path, so an 8-char-or-shorter redis password, api key, or token could be written to logs and the UI unmasked. The sibling helper mask_sensitive_keys already guards this case; _mask_value now does the same by fully masking any value at or below the threshold. * fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers Fully masking short values is the right default for secret masking, but CooldownCache reuses the masker purely to truncate exception messages to the first 50 characters, and it relies on short messages being returned readable. Masking those blanked out short exception text and broke its tests. Add a mask_short_values flag (default True, secure) and have CooldownCache pass False so it keeps the truncation behavior, while every secret-masking caller still gets short values fully masked. * fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview MCPDebug uses the masker to preview auth tokens in debug headers and documents that values of 10 chars or fewer are shown unchanged so token types stay distinguishable. Pass mask_short_values=False so that diagnostic behavior is preserved while secret maskers keep masking short values. * fix(mcp_debug): mask short auth values in debug headers instead of echoing them Earlier this masker opted out of short-value masking to keep a token preview, but that echoes short authorization and token values verbatim in debug response headers, which is the same leak this change is meant to close. Auth material should never be emitted in full, so mask short values here too; the first/last character preview still applies to longer tokens. Only CooldownCache keeps the opt-out, since it truncates exception text rather than masking secrets. * test(mcp_debug): assert masked short value preserves length * refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917) Fireworks AI deprecated audio inference on 2026-06-10 (https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation). Live API testing confirms the endpoint is already non-functional: a valid Fireworks API key receives HTTP 401 "Unauthorized" from api.fireworks.ai/inference/v1/audio/transcriptions for every request, regardless of payload. The audio-prod.api.fireworks.ai host referenced in the test suite returns 401 for every path; the entire host is decommissioned. Remove the dead FireworksAIAudioTranscriptionConfig class and every reference to it across the codebase: - Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line config class that inherited from OpenAIWhisperAudioTranscriptionConfig) - Remove the Fireworks branch from ProviderConfigManager.get_provider_audio_transcription_config() in litellm/utils.py; update the stale comment in get_optional_params_transcription that referenced fireworks ai - Remove the FireworksAIAudioTranscriptionConfig entries from LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in litellm/_lazy_imports_registry.py - Remove the TYPE_CHECKING re-export in litellm/__init__.py - Remove the transcription branch in the fireworks_ai case of get_supported_openai_params() in litellm/litellm_core_utils/get_supported_openai_params.py - Remove the whisper-v3 and whisper-v3-turbo entries from model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json (both had mode: audio_transcription and zero-cost pricing) - Remove the TestFireworksAIAudioTranscription test class and its imports from tests/llm_translation/test_fireworks_ai_translation.py No other provider is affected. The openai_compatible_providers list, FireworksAIMixin, and the OpenAI Whisper transcription handler all stay because they are shared with other Fireworks endpoints and other providers. The provider_endpoints_support.json registry already had audio_transcriptions set to false for fireworks_ai. * feat: add darkbloom provider (#30876) * feat: add darkbloom provider * fix: document darkbloom provider endpoints * fix: address darkbloom review feedback * fix: update darkbloom tool metadata * fix: fail fast for non-Postgres database URLs (#30883) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging * Validate DIRECT_URL alongside DATABASE_URL startup guards * fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946) * fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608) * test(bedrock): mid-stream server errors trigger streaming fallback (#24608) * style(bedrock): black-format stream-error helper (#24608) * fix(mcp): re-land native tool preservation with typed annotations (#30645) * fix(mcp): preserve native tools in semantic filter hook with typed annotations * fix(mcp): tighten _is_mcp_tool Chat Completions shape check * fix(sambanova): return embeddings supported params instead of dropping them (#30937) * fix(router): send fallback metadata when streaming (#30914) When a streaming request triggers a fallback, there was previously no way to know it happened. This commit addresses this in a few ways: 1. The response now correctly populates the fallback headers (`x-litellm-attempted-fallbacks`) so callers know a fallback happened. 2. The correct model ID is passed in the streaming chunks. 3. A streaming chunk with the fallback error can be optionally sent back to the client (opt-in) by passing `include_fallback_errors: true` in the request. The format of the fallback errors while streaming is intentionally OpenAI compatible to not break existing libraries that parse these events. It was tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not delieved unexpectedly to callers by default. * fix(mistral): drop output-only reasoning fields from input messages (#30884) LiteLLM attaches reasoning_content and thinking_blocks to assistant responses. Replaying those assistant turns verbatim forwarded the fields back to Mistral, whose input schema forbids unknown keys, so the whole request failed with a 422 extra_forbidden and reasoning models became unusable across multiple turns. Strip both fields from assistant messages before the request is built, in a spot that runs ahead of the image/file branch so it applies on every path. Fixes #30835 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652) * fix(perplexity): bill search queries at the per-request price, not 1/1000 The fallback cost calculator divided search_context_cost_per_query by 1000, but that field stores the per-request price in USD: sonar is {low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published $5/$8/$12 per 1,000 requests expressed per request. The gemini cost calculator reads the same field per request with no division (its docstring calls it "the per-request cost"). The division understated search cost by 1000x on every Perplexity call that falls back to manual calculation (i.e. when the API does not return a pre-computed usage.cost). Use the value directly. Update the tests that had encoded the /1000 factor in their expectations, and drop an unused import flagged by ruff in the touched test file. * test(perplexity): update integration test search-cost expectations to per-request The integration tests still encoded the old /1000 search-cost factor, so they failed once the fallback calculator was corrected to bill search_context_cost_per_query per request. Update the four expected-cost computations (and the high-volume dollar-value comments) to match. * test(perplexity): drop unused mock imports flagged by ruff * fix: include model_access_groups when expanding all-team-models in get_team_models (#30622) * fix(fireworks_ai): return None for transcription in get_supported_openai_params Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is decommissioned. Without an explicit transcription branch, requests with request_type='transcription' fell through to the else and returned FireworksAIConfig chat-completion params. Return None instead to signal the provider does not support transcription. * fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting Without an operator gate, any authenticated caller could set include_fallback_errors=True, trigger a fallback, and read raw upstream exception messages from the x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event. Strip include_fallback_errors from request data in common_processing_pre_call_logic when expose_fallback_errors_to_caller is not set, so the router never builds the error list. Also gate _should_include_fallback_errors on the same setting as a secondary check for the streaming SSE injection path. * test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test The operator gate added in e7ff3e1 means include_fallback_errors is only honoured when general_settings.expose_fallback_errors_to_caller is True. Set that flag via monkeypatch in the test that exercises the emit path. * test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos test_convert_url called convert_url_to_base64 against a live picsum.photos URL and asserted nothing, so it added no real signal and broke CI whenever the host was unreachable (it was returning 522 and blocking this branch). Replace the live call with a mocked HTTP client and assert the produced base64 data URL, so the conversion path is exercised deterministically with no network dependency. This suite runs under VCR, which is why a transport level mock (respx) does not reliably intercept; mocking the client object itself is robust regardless. * fix(interactions): drop role from Interaction response to match Google spec Google removed the output-only role field from the Interaction schema (it now lives only on Turn), so the live OpenAPI compliance canary started failing with 'role' not in spec. Reconcile our generated types by removing role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse, stop stamping role=model in the responses-to-interactions transformation, and update the compliance and integration tests accordingly. Turn.role is kept since the spec still defines it. * fix: align all-team-models sentinel access * fix(router): forward include_fallback_errors through multi-hop fallbacks run_async_fallback received include_fallback_errors as an explicit named parameter, so it was bound out of **kwargs and never reached the nested async_function_with_fallbacks call. Multi-hop fallback chains (a fallback group that itself fails over) therefore stopped collecting fallback errors beyond the first hop when a caller opted in. Re-inject the flag into kwargs before the nested call so inner hops keep accumulating errors, which add_fallback_headers_to_response already merges across levels. * fix(router): stop fallback lookups from mutating the router fallbacks config get_fallback_model_group resolved a bare-string fallback by popping it out of the fallbacks list it was handed. That list is frequently the live router.fallbacks config, so a single lookup permanently removed the entry and the configured fallback stopped applying to later requests until restart. The pop also ran inside enumerate(), shifting indices and skipping an adjacent string fallback. Read the item instead of popping it, and add a regression test that fails on the old mutating behavior --------- Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * fix(sambanova): update pricing, deprecate retired models, and add missing models (#30016) * feat(bedrock): add amazon.titan-embed-g1-text-02 embedding model support - Add model to provider routing allowlist in embedding.py - Add request transformation using AmazonTitanG1Config - Add response transformation using AmazonTitanG1Config - Add pricing metadata to model_prices_and_context_window.json - Add unit tests for embedding and model info Fixes missing cost tracking reported in #29786 Related to VANDRANKI/litellm PR #29790 * style: fix syntax error, trailing whitespace and missing newline * style: apply black formatting to embedding.py * style: apply black formatting to test_bedrock_embedding.py * fix(sambanova): update pricing, fix context windows, add deprecation dates, and add missing models * fix(sambanova): sync model_prices_and_context_window_backup.json with primary * fix(sambanova): fix indentation on Meta-Llama-3.2-1B-Instruct deprecation_date * fix(bedrock): add amazon.titan-embed-g1-text-02 to unmapped model error message * style: apply black formatting to embedding.py * fix(sambanova): correct indentation on DeepSeek-V3.2 entry * fix(sambanova): replace gemma-3-12b-it with gemma-4-31B-it (verified pricing) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info (#30880) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info get_model_info rebuilt ModelInfo by copying a fixed allow-list of input/output_cost_per_token_above_<N>_tokens keys (128k/200k/272k/512k), so any other threshold a user registered was dropped before reaching _get_token_base_cost, which already reads an arbitrary threshold out of the key name. Custom tiers such as above_500k_tokens were silently ignored and billing fell back to the base per-token rate. Carry over any _above_<N>_tokens cost key present on the source cost-map entry that the fixed fields miss Fixes #30344 * test(cost): keep suite hermetic by popping the temp tiered-pricing model Wrap the regression body in try/finally so litellm.model_cost no longer leaks the litellm-test-non-standard-tier entry into later tests that iterate or reset the global cost map. Addresses Greptile review thread. * fix: resolve UP045 lint violations (Optional[X] -> X | None) Convert Optional[X] type annotations to X | None syntax across rerank transformations, spend tracking, and other modules to satisfy ruff strict gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: run black formatting on UP045-fixed files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused Optional imports after UP045 migration Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: black format cold_storage_handler.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): correct OSS staging branch name in guard-main-branch errors Co-authored-by: Cursor <cursoragent@cursor.com> * fix: strip trailing zeros from M/B spend formatter * fix: address focus and streaming edge cases * feat: add LAR-1 semantic routing strategy Optional router strategy that picks a deployment tier from request_kwargs.metadata.lar1 (confidence, evidence, time). Deployments are tagged with model_info.type (cloud-smart, cloud-fast, local, deep). Thresholds are configurable via routing_strategy_args. Includes 30 unit tests and an Ollama example config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mavvrik): advance metricsMarker on empty-content deliver When deliver() receives empty content (no spend data for a date), it now registers with Mavvrik and PATCHes the metricsMarker before returning instead of short-circuiting. Dates with zero spend no longer stall marker advancement, preventing unnecessary catch-up API calls on subsequent runs. * style: black format mavvrik_destination * fix: handle empty mavvrik exports and lar1 reset * test: add regression test for _reset_custom_routing_strategy * fix(test): mock async destination.deliver in mavvrik export window test * style: ruff format spend_management_endpoints after merge * fix(router): apply LAR-1 strategy atomically so invalid thresholds don't leave partial state apply_lar1_routing_strategy set router.routing_strategy to "lar1" before constructing LAR1RoutingStrategy, whose __init__ validates thresholds via _normalize_thresholds and raises on a misconfigured (out-of-order or out-of-range) set. On a live update_settings call with bad thresholds the router was left advertising routing_strategy="lar1" with no custom selector bound, while the previous strategy's selectors stayed registered. Build (and validate) the strategy before mutating any router state, so a threshold error leaves the router exactly as it was. Add a regression test that asserts a failed switch keeps the prior strategy intact. --------- Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com> Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: xbrxr03 <abrarhabib03@gmail.com> Co-authored-by: hayden <sktpghks138@gmail.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Wassim Badraoui <98709649+Wassbdr@users.noreply.github.com> Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> Co-authored-by: Neimar Avila <neimar.avila@gmail.com> Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> Co-authored-by: Jerry-Scintilla <jerrycaocao@126.com> Co-authored-by: AlexBGoode <me.at.forum@gmail.com> Co-authored-by: Carsten Boloz <cdboloz1@gmail.com> Co-authored-by: jesco <team@srswti.com> Co-authored-by: Praveen Ghuge <pghuge@digitalex.io> Co-authored-by: Jim Smith <j.h.smith@ieee.org> Co-authored-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Vedant Agarwal <43557509+Vedant-Agarwal@users.noreply.github.com> Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: bhumikadangayach <139267865+bhumikadangayach@users.noreply.github.com> Co-authored-by: Ewertonslv <ewertoncom297@gmail.com> Co-authored-by: carlsonchik <carlsonchik@users.noreply.github.com>	2026-06-26 09:17:44 -07:00
Mateo Wang	6cc9ea2538	fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases (#31373) ... * fix(cost-map): retarget mistral-medium-latest to Medium 3.5 and add date-pinned aliases Mistral repointed the rolling mistral-medium-latest alias from Medium 3.1 to Medium 3.5, but the static cost map still carried Medium 3.1 specs, showing wrong pricing/context in the model hub and undercharging spend by about 3.75x (LIT-3883). Update mistral/mistral-medium-latest to Medium 3.5 ($1.50/$7.50 per 1M, 256K context, reasoning + vision), add the bare date-pinned aliases mistral/mistral-medium-2604 (Medium 3.5) and mistral/mistral-medium-2508 (Medium 3.1) that match Mistral's real API model ids, and add supports_reasoning to mistral/mistral-medium-3-5. Apply every change to both model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json so the two stay in sync, and extend the regression tests to lock the resolved get_model_info values and the main/backup parity for all touched models. * test(cost-map): force local cost map in mistral-medium-latest resolution test get_model_info reads litellm.model_cost, which is fetched from the remote main branch at import time when LITELLM_LOCAL_MODEL_COST_MAP is unset. Until this PR lands on main, that remote map still carries the pre-merge Medium 3.1 pricing, so the assertion was only passing when the remote fetch happened to fail and fell back to the bundled backup. Force the local cost map (the same fixture pattern the other get_model_info tests use) so the alias resolution is verified deterministically against the in-repo file.	2026-06-25 18:27:18 -07:00
Mateo Wang	e0e920d80e	feat(mistral): support Mistral OCR 4 (mistral-ocr-4-0) (#31353) ... * feat(mistral): support Mistral OCR 4 (mistral-ocr-4-0) Add the mistral/mistral-ocr-4-0 model to the cost map and reprice mistral/mistral-ocr-latest, which now resolves to OCR 4 server-side, at $4 / 1000 pages. Add the include_blocks param so callers can request OCR 4's paragraph-level bounding boxes and typed content blocks. OCR 4's new per-page response fields (blocks, confidence_scores, tables, hyperlinks, header, footer) already pass through transform_ocr_response via the extra="allow" config on OCRPage; add a regression test pinning that behavior alongside cost and param coverage. * fix(mistral): revert unverified OCR 4 annotation_cost_per_page bump Mistral's published OCR 4 pricing lists $4/1000 pages for the API and no separate annotation rate; the $5/1000 figure is the distinct Document AI (Studio) tier. The earlier 0.003 -> 0.005 bump on annotation_cost_per_page had no cited source, and ocr_cost() never reads that field (it bills off ocr_cost_per_page), so the value is documentation-only. Revert annotation_cost_per_page to the existing 0.003 convention for both mistral-ocr-latest and mistral-ocr-4-0, keeping only the verified, tested ocr_cost_per_page: 0.004 change. * fix(mistral): set OCR 4 annotation_cost_per_page to verified $5/1000 rate Verified against Mistral's authoritative sources: the pricing page, the OCR 4 announcement, and the ocr-4-0 model card all list OCR 4 at $4/1000 pages for basic OCR and $5/1000 for annotated pages (Document AI). The $5/1000 figure is the annotated-pages rate, which is exactly what annotation_cost_per_page encodes, mirroring the original OCR entry's 0.001 basic / 0.003 annotated split. Restore annotation_cost_per_page to 0.005 for mistral-ocr-latest and mistral-ocr-4-0; the earlier revert to 0.003 was based on an incomplete reading that treated Document AI as a separate product. ocr_cost_per_page stays 0.004, which is the value billed by ocr_cost(). * fix(mistral-rust): include_blocks in Rust OCR supported params --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-25 16:42:37 -07:00
Mateo Wang	b7f28bd89f	feat(aiml): add openai/gpt-image-2 image model (#31323) ... * feat(aiml): add openai/gpt-image-2 image model Adds aiml/openai/gpt-image-2 to the cost map and teaches AimlImageGenerationConfig to route OpenAI-style image models through the upstream OpenAI request schema instead of the AI/ML flux schema. Without this, size, n, and response_format would be remapped to image_size/num_images/output_format, which the gpt-image-2 endpoint on api.aimlapi.com does not accept. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(aiml): note gpt-image-2 flat-rate pricing basis; apply ruff format Documents in the cost-map notes that output_cost_per_image is AI/ML's published medium-quality rate, billed as a flat per-image price like the other aiml image entries. Reformats the touched files under the repo's ruff formatter (migrated from black in #31317). * fix(aiml): drop /v1/images/edits from gpt-image-2 supported_endpoints LiteLLM only implements an image generation transformer for AIML, so listing /v1/images/edits overclaimed support. Align with every other aiml image entry, which lists only /v1/images/generations. * style(aiml): format transformation.py at line-length 88 The repo formats litellm/ with ruff at line-length 88 (Makefile/CI call sites), while ruff.toml's global 120 only governs E501/import sorting. Reformat the transformer to 88 so make format-check / CI lint pass, and restore the test files to their original layout since tests/ is not part of the auto-formatted tree. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-06-25 16:41:43 -07:00
milan-berri	7ffce15766	Add GA pricing for gemini-3-pro-image and gemini-3.1-flash-image. (#30022) ... Fixes #29794. Adds bare, gemini/, and vertex_ai/ entries copied from preview models so proxy cost tracking works for GA model names. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-26 00:40:54 +02:00