8121 commits

Author	SHA1	Message	Date
Yassin Kortam	f49707bc66	fix(otel): cap metric attribute cardinality with include/exclude lists (#30257) ... * fix(otel): cap metric attribute cardinality with include/exclude lists OTEL metrics stamped every per-request hidden_params and metadata.* field onto each gen_ai.client.* sample, so near-unique values created one metric time series per request and backends like Splunk Observability Cloud throttled and dropped the data. Add an attributes block under callback_settings.otel with mutually-exclusive include_list (allowlist) and exclude_list (denylist), validated against the known attribute names at startup and applied once to the metric attributes in _record_metrics. Spans are untouched, and with no config every attribute is still emitted so existing setups are unaffected. Resolves LIT-3600 * fix(otel): resolve metric attribute filter from callback_settings The proxy usually constructs the OpenTelemetry logger without forwarding the attributes kwarg, while the filter lives under litellm.callback_settings["otel"]["attributes"]. __init__ only read the kwarg, so the recording instance kept config.attributes=None and shipped metrics at full cardinality even when the filter was configured; a live proxy run exposed this. Fall back to the global at init for the base otel logger, and add a regression test that drives the real success hook through the callback_settings path (the unit tests passed before because they injected the config directly). * fix(otel): reject gen_ai.token.type from metric attribute filter lists gen_ai.token.type was a member of VALID_METRIC_ATTRIBUTE_NAMES, so an operator could list it in include_list or exclude_list and pass startup validation. The attribute is injected into the input/output token series after _filter_metric_attributes runs, so the filter never sees it and the request silently has no effect. Reject it loudly from either list instead, matching the contract that a non-actionable attribute name fails fast rather than falling through to a no-op. It stays a structural discriminator on the token-usage histogram. * fix(otel): resolve metric attribute filter lazily at record time The proxy constructs the OpenTelemetry logger before it populates litellm.callback_settings["otel"]["attributes"], so resolving the filter at __init__ left config.attributes None and shipped metrics at full cardinality. A live proxy run confirmed the leak. Resolve the filter on the first metric record instead, when callback_settings is populated, while still validating an explicit config eagerly so a bad SDK config fails at startup. The regression test now constructs the logger before populating callback_settings to mirror that ordering, so it fails if the filter is resolved too early. * fix(otel): don't cache invalid filter on lazy callback_settings path On the lazy callback_settings resolution path, _ensure_metric_attribute_filter wrote self.config.attributes before validating it. When validation then failed, _metric_attr_filter_resolved stayed False while config.attributes held the bad filter, so the next record skipped the callback_settings re-read and re-raised the stale error indefinitely; fixing the misconfiguration required a restart. Drop the premature write and resolve from the local value. A subsequent record now re-reads callback_settings, so a corrected config takes effect without a restart. The write was dead on the success path anyway, since the resolved frozensets are what the filter reads.	2026-06-12 17:29:46 -07:00
Sameer Kankute	079c136742	chore(oss): litellm oss staging 120626 (#30292) ... * feat(bedrock): add bedrock mantle gemma 4 models (#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from #25776 Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on #30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (#30200) (#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. #27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes #30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes #27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes #27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes #27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes #27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes #29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR #29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes #28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request #29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to #30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-12 09:49:25 -07:00
Sameer Kankute	7d1f68e72a	fix(proxy): populate access_via_team_ids on /v1/model/info (#30274) ... * fix(proxy): populate access_via_team_ids on /v1/model/info Team metadata enrichment previously only ran on /v2/model/info with include_team_models=true, leaving /v1/model/info without access_via_team_ids for project model-picker flows. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(dashboard): sync OpenAPI schema for /v1/model/info query params Add include_team_models and teamId to the generated schema for /model/info and /v1/model/info after the proxy endpoint gained team-access filtering. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): always return direct_access on /v1/model/info Set direct_access to true or false on every enriched model so clients can filter without treating a missing field as ambiguous. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(proxy): fail fast when teamId is set without a connected DB on /v1/model/info Raise the db_not_connected error before building, enriching, and translating the model list instead of after, so a teamId query against a proxy with no database no longer wastes the full enrichment pipeline. * fix(proxy): fail fast when include_team_models is set without a database include_team_models=True relies on _populate_team_access_on_models to set direct_access/access_via_team_ids, which only runs when a database is connected. Without one, _filter_models_to_user_accessible discarded every model and the endpoint returned an empty list with HTTP 200. Mirror the teamId guard so the request fails fast with a clear db_not_connected error before any model-list work. * fix(proxy): populate direct_access on single-model /model/info lookup The /v1/model/info list path populates model_info.direct_access (and access_via_team_ids) when a database is connected, but the litellm_model_id single-model lookup returned early without it. This made the two endpoints disagree, breaking the parity assertion in test_get_specific_model. Run the same population on the single-model path so both responses match. * fix(proxy): apply no-DB fast-fail before litellm_model_id branch The teamId/include_team_models no-DB guard sat after the litellm_model_id early return, so ?litellm_model_id=X&teamId=Y with no DB returned 200 with unpopulated access fields instead of the 500 raised on every other path. Move the guard ahead of the branch so the fast-fail is uniform. * fix(proxy): apply teamId/include_team_models filters on single-model lookup The litellm_model_id early-return branch in model_info_v1 populated the team access fields but returned before the teamId and include_team_models filters ran, so a single-model lookup surfaced the deployment regardless of team access when the DB was connected. Run both filters on the single-model list before returning so the documented query params behave the same with and without litellm_model_id. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 09:49:09 -07:00
Sameer Kankute	729b005e4e	fix(google_genai): preserve complete SSE events in Vertex/Gemini image streaming (#30270) ... * fix(google_genai): preserve complete SSE events in image streaming Use iter_lines/aiter_lines instead of byte chunking so large inlineData base64 payloads from Vertex/Gemini streamGenerateContent are not split across events, which caused truncated JSON and SDK parse failures. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(google_genai): buffer SSE lines until event delimiter Assemble multi-field SSE events on blank-line boundaries instead of terminating each field line individually. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): update google_ai_studio mocks from aiter_bytes to aiter_lines Streaming iterator was changed to use iter_lines/aiter_lines instead of iter_bytes/aiter_bytes. Update the two mocked streaming responses in test_google_ai_studio.py to match. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-12 07:49:30 -07:00
Sameer Kankute	e8e5b47fc2	feat(passthrough): add configurable pass-through request timeouts (#30266) ... * feat(passthrough): add configurable pass-through request timeouts Allow operators to set general_settings.pass_through_request_timeout and per-endpoint timeout values, and apply them to native HTTP passthrough routes and SDK passthrough paths such as Bedrock /converse. * fix(passthrough): address CI lint and regenerate dashboard API types * refactor(passthrough): extract timeout utils to proxy-free module, fix router_timeout drop - Move resolve_llm_passthrough_timeout + resolve_pass_through_request_timeout to litellm/passthrough/timeout_utils.py (no fastapi/proxy imports at module scope) - router.py and passthrough/main.py now import from timeout_utils directly, avoiding the fastapi transitive import in pure SDK contexts - pass_through_endpoints.py re-imports from timeout_utils for backward compat - resolve_llm_passthrough_timeout now accepts router_timeout so Router(timeout=X) is respected for passthrough calls instead of being silently dropped - Use _get_httpx_client (cached) instead of bare HTTPHandler(...) in sync path to avoid creating an unclosed client per call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(router): use _explicit_timeout for passthrough to not shadow general_settings self.timeout defaults to litellm.request_timeout (6000s) when the user doesn't pass timeout= to Router(). Using it as router_timeout caused general_settings.pass_through_request_timeout to be silently ignored. Only pass router_timeout when the user explicitly set Router(timeout=X). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(timeout_utils): avoid fastapi transitive import by using sys.modules resolve_pass_through_request_timeout previously did a lazy `from litellm.proxy.proxy_server import general_settings` which loads the proxy module (and transitively fastapi) even in pure SDK contexts. Replace with a sys.modules lookup: if the proxy module is already loaded (i.e. we're inside the proxy), read general_settings from it; otherwise skip and fall back to the 600s default. No import is triggered. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): remove unused imports from pass_through_endpoints.py DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS and resolve_llm_passthrough_timeout are not used in this file; only resolve_pass_through_request_timeout is. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pass_through_endpoints): re-export DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS Tests import this constant directly from pass_through_endpoints.py; re-add it to the import from timeout_utils for backward compatibility. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pass_through_endpoints): re-export resolve_llm_passthrough_timeout for backward compat Tests import both DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS and resolve_llm_passthrough_timeout from pass_through_endpoints.py; use noqa comments to suppress the unused-import lint warning on re-exports. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-12 07:40:02 -07:00
Sameer Kankute	ccc20b121f	fix(responses): presidio PII masking for Azure WebSocket and streaming (#30003) ... * fix(responses): Presidio PII masking for Azure WebSocket and streaming Wire Presidio into native Responses WebSocket forwarding and fix streaming output unmasking so masked tokens are restored for HTTP and WS clients. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix unused imports in responses handlers * fix(responses): address Greptile review - Azure WebSocket model URL and PII logging - Add model_in_websocket_url() to BaseResponsesAPIConfig (default True) so providers can opt out of ?model= being appended to WebSocket URLs. - Override model_in_websocket_url() to return False for Azure, since Azure sends the model in the response.create body, not the URL query string. - Use this flag in llm_http_handler to conditionally append ?model=. - Pass masked message to _store_input() instead of the original PII-containing message so logging destinations do not receive unmasked PII. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(responses): mask nested response.create input format for Presidio PII Handle the nested {"type":"response.create","response":{"input":[...]}} format in _mask_response_create. Previously only the flat top-level input was masked; the nested shape bypassed Presidio and forwarded raw PII upstream. Now both shapes are normalized and masked before forwarding. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: apply black formatting to llm_http_handler and streaming_iterator Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: suppress PLR0915 on async_responses_websocket Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(responses): add apply_to_output masking on Responses API WebSocket path Previously the WebSocket guardrail filter excluded callbacks with apply_to_output=True, leaving model-generated PII unmasked before returning to the client. - Collect apply_to_output callbacks separately in llm_http_handler and pass them to ResponsesWebSocketStreaming as output_guardrail_callbacks. - Add _mask_response_completed method that calls check_pii(output_parse_pii=False) on text blocks in response.completed events, masking model output PII. - backend_to_client now chains unmask (pii_tokens) → mask (apply_to_output) before forwarding each event to the client. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(responses): unmask PII tokens in streaming delta events and warn on guardrail init failure - Rename _unmask_response_completed -> _unmask_response_event and extend it to also unmask response.output_text.delta (and other delta types) so real-time streaming clients receive original values, not PII tokens. - Split the broad except-and-swallow into ImportError (expected in SDK-only environments) vs Exception (unexpected — now logs a warning so operators know masking is disabled). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(responses): enforce authorized model on WebSocket frames and remove proxy import Security: add _enforce_authorized_model to ResponsesWebSocketStreaming that overwrites both flat and nested model fields in every response.create frame with the connection-authorized model, preventing deployment-substitution attacks where an authenticated user sends a different model name in the frame body after connecting with an allowed model. Layering: remove the _OPTIONAL_PresidioPIIMasking isinstance check and proxy import from the SDK handler. Use duck-typed checks (callable check_pii + get_presidio_settings_from_request_data) so any guardrail implementing the interface works, not just Presidio. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(responses): add _unmask_pii_text to duck-typed contract and mask delta frames - Add callable(_unmask_pii_text) check to the guardrail_callbacks filter so a custom guardrail missing that method cannot cause an AttributeError and silently kill the WebSocket session. - Extend _mask_response_completed to also mask response.output_text.delta (and other delta types) for apply_to_output callbacks, so real-time streaming clients do not receive unredacted model-generated PII in deltas. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix Responses WebSocket guardrail edge cases * fix(responses): log masked output and suppress deltas when apply_to_output active - Move _store_event to after _mask_response_completed so logs receive the redacted form, not raw model output containing PII. - Suppress delta event forwarding when output_guardrail_callbacks are present: per-fragment Presidio cannot catch PII that spans multiple chunks (e.g. "alice@" + "example.com"). Clients receive only the fully-masked response.completed, which Presidio scans on complete text. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(responses): mask and suppress response.output_item.done for apply_to_output response.output_item.done carries completed item text in item.content[*].text before response.completed arrives, allowing unmasked PII to reach the client. - _unmask_response_event: unmask input-PII tokens in item.content[*].text - _mask_response_completed: run check_pii on item.content[*].text for apply_to_output callbacks (same as response.completed handling) - backend_to_client suppression: also skip response.output_item.done when output_guardrail_callbacks are active; client receives only the fully-masked response.completed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(types): cast response_obj to ResponsesAPIResponse to satisfy mypy Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(types): cast response_obj to ResponsesAPIResponse to satisfy mypy" This reverts commit d5969557628f9aff58948b9d37cc64d577f95a15. * Revert "fix(responses): mask and suppress response.output_item.done for apply_to_output" This reverts commit 219fd54ea3446f4399fde40c07ba0617e2834573. * fix(types): accept dict responses in guardrail output write-back Streaming response.completed events pass a dict response object, so widen _apply_guardrail_responses_to_output to match its existing runtime handling. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(responses): skip Presidio masking on suppressed WebSocket delta events Delta events are dropped wholesale when apply_to_output masking is active, so masking them first issued a wasted check_pii call per fragment. Move the suppression check ahead of the unmask/mask passes; the event type is invariant across both, so client-visible behavior is unchanged. * test(responses): cover Responses WebSocket PII masking hooks Add regression tests for the native Responses WebSocket guardrail path: input masking and model enforcement in _mask_response_create, token unmasking in _unmask_response_event, apply_to_output masking and delta suppression in _mask_response_completed/backend_to_client, and the get_websocket_url / model_in_websocket_url defaults for the base and Azure configs. Raises diff coverage above the codecov patch target. * fix(responses): suppress text-bearing done events under output PII masking When apply_to_output masking is active on a native Responses WebSocket, response.output_text.done, response.content_part.done, and response.output_item.done carry the full model output before the masked response.completed arrives, so an authenticated client could read unmasked PII from those events. Suppress them alongside delta events; the client receives only the fully-masked response.completed. * refactor(responses): drop dead delta branch in WebSocket output masking Delta events are suppressed in backend_to_client before _mask_response_completed runs when output masking is active, so the method's delta-handling branch was unreachable. Restrict it to response.completed and cover the Responses API unmask path with a Pydantic ResponseCompletedEvent regression test. * fix(presidio): flush buffered chat chunks on mixed unmask stream _stream_pii_unmasking buffered ModelResponseStream chunks but returned early once a /v1/responses event was seen, silently dropping the buffered chat chunks. Flush them in order before switching to passthrough, mirroring _stream_apply_output_masking, and cover it with a regression test. * fix(responses): mask instructions and tool-call arguments in WebSocket PII path Presidio masking on the native Responses WebSocket path left two gaps. On the request side _mask_response_create only walked the input containers, so PII placed in the instructions field of a response.create frame was forwarded upstream and logged unmasked even with output_parse_pii enabled. Now both the flat and nested instructions strings are masked alongside input. On the response side _mask_response_completed only masked content text blocks, so model-produced PII inside function-call arguments could reach the client when apply_to_output was enabled, both via the standalone response.function_call_arguments.done event and via the function_call output items in response.completed. The done event is now suppressed under output masking and completed function-call arguments are run through check_pii before forwarding or logging. * fix(responses): suppress reasoning_summary_text.done under output PII masking * fix(responses): mask function_call_output.output in WebSocket PII path response.create input items of type function_call_output carry user-controlled text in output, not content, so the Presidio masking pass forwarded that text upstream unmasked. Mask the output field (string or list of text blocks) alongside content. * fix(responses): mask reasoning summary PII in WebSocket output path --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:27:03 -07:00
Sameer Kankute	02bce7b393	fix(mcp): honor server_id for REST tool calls with shared upstream URLs (#30184) ... * fix(mcp): honor server_id for REST tool calls with shared upstream URLs When multiple MCP server entries point at the same backend URL and tool name, REST /mcp-rest/tools/call now routes and applies auth from the requested server_id instead of the global unprefixed tool-name mapping. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): classify prefixed REST tool names against full registry Use all registered MCP server prefixes for prefix detection so unauthorized prefixed names still trigger tool_server_mismatch, and reject ambiguous hyphenated REST tool names with server_id. Co-authored-by: Cursor <cursoragent@cursor.com> * test(mcp): cover server_id fallback for unresolved prefixed REST tool names execute_mcp_tool left the prefix-retry and requested-server fallback branches uncovered, dropping diff coverage below the project target. Add a regression test for a REST call that passes server_id with a prefixed tool name that resolves to no managed tool; it must still dispatch to the server identified by server_id rather than the server named by the prefix. * test(mcp): scope global tool-name mapping mutation with patch.dict * test(mcp): cover server_id guard on prefix-retry tool resolution The prefix-retry branch in execute_mcp_tool re-prefixes the tool name with the requested server's known prefixes when the bare lookup misses. The candidate-found path that assigns mcp_server from that lookup stayed uncovered, so codecov patch coverage remained below the diff target. Add a regression test where the re-prefixed lookup resolves a server whose server_id differs from the requested server_id; the tool_server_mismatch 403 guard must still fire instead of being silently bypassed. * test(mcp): assert requested server credentials injected on cross-server REST routing * perf(mcp): scan registry prefixes only when server_id is supplied * fix(mcp): allow hyphenated upstream tool names when REST server_id is authoritative * perf(mcp): skip registry prefix scan for separator-free REST tool names * test(http_handler): drop httpbin dependence from per-request timeout test The per-request timeout test posted to https://httpbin.org/delay/10 and asserted a Timeout was raised. httpbin's free /delay endpoint intermittently returns 503 even when the /get reachability guard succeeds, so local_testing_part1 flaked on that 503 instead of the expected timeout (failed identically across an initial run and a rerun-from-failed). Serve the slow response from a local ThreadingHTTPServer so the timeout fires deterministically with no third-party network dependence. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:25:53 -07:00
Sameer Kankute	41801d65bd	feat(proxy): add require_managed_files setting for file uploads (#30186) ... * feat(proxy): add require_managed_files setting for file uploads Add an opt-in litellm_settings flag that rejects POST /v1/files without target_model_names, and parse target_model_names[] from OpenAI SDK list extra_body. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): return clean require_managed_files error message Use a plain HTTPException detail string so create_file does not stringify a dict, and import UploadFile at module scope. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): preserve repeated target_model_names[] form fields Read target_model_names from the raw multipart form instead of the dict(form_data)-collapsed request body so repeated target_model_names[] fields (how the OpenAI SDK serialises a list extra_body) keep every value rather than truncating to the last one. Drops the now-unreachable list branch in the value parser. * fix(proxy): reject model param to close require_managed_files bypass --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:25:28 -07:00
Sameer Kankute	680f3ff810	fix: bedrock mantle fixes (#30083) ... * fix: respect aws region * Fix chat completion to responses bridge * Handle response streaming events * Fix bedrock mantle region priority and CI test failures. aws_region_name now overrides BEDROCK_MANTLE_REGION env, and provider tests pass region via litellm_params. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock_mantle): use warmup id prefix constant and log untyped tool drops * refactor(bedrock_mantle): keep aws_region_name extraction inside chat config * fix(bedrock_mantle): validate aws_region_name before host interpolation The client-supplied aws_region_name flows unvalidated into the Bedrock Mantle host (https://bedrock-mantle.{region}.api.aws), so a value containing a slash could redirect the request, along with the configured bearer API key, to an arbitrary host. Validate the region against the AWS region format in both the chat and responses transformations before it is interpolated. * fix(bedrock_mantle): close AWS_REGION_NAME chat gap and surface dropped tools Chat region resolution now consults AWS_REGION_NAME, matching the responses path precedence. Unsupported Responses tools dropped by map_openai_params are logged at warning level so the loss is visible in production. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:25:16 -07:00
Sameer Kankute	87cf67ec30	feat(gemini): forward web search tools in image generation (#30119) ... * feat(gemini): forward web search tools in image generation Map tools and web_search_options to googleSearch on Gemini image generateContent requests for Google AI Studio and Vertex AI. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(gemini): dedupe image search tools and return mapped params Skip web_search_options when tools already include search, dedupe search tool entries, and assign the return value from map_gemini_image_tools_params. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(gemini): preserve toolConfig side-effects in image tool mapping * fix(gemini): forward toolConfig in image generation request body * fix(gemini): track web search grounding cost on image generation Forwarding Google Search grounding to Gemini and Vertex image generation previously incurred billable grounding charges that never reached LiteLLM spend tracking, because the image cost path returns through the Gemini/Vertex image calculators before built-in tool spend is added. Carry the grounding request count from the response onto the image usage object and bill it with the same per-request web search accounting used for chat completions. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:24:36 -07:00
Sameer Kankute	7519e37d26	feat(proxy): enforce key/team guardrails on bedrock passthrough routes (#30194) ... * feat(proxy): enforce key/team guardrails on bedrock passthrough routes /bedrock/... passthrough routes silently skipped all guardrail hooks because CallTypes.allm_passthrough_route had no entry in guardrail_translation_mappings. Add a dispatcher (LlmPassthroughRouteHandler) registered for that call type that routes to BedrockPassthroughGuardrailHandler for Bedrock Converse endpoints; wire post_call_success_hook into both the JSON and AWS event-stream response paths in common_request_processing, including full de-anonymization for streaming responses with proportional text distribution across original event-stream delta frames. * refactor(proxy): address greptile feedback on bedrock passthrough guardrails Move botocore event-stream logic from proxy/ to BedrockPassthroughGuardrailHandler.de_anonymize_converse_stream; _handle_event_stream_allm_passthrough_route becomes a thin provider dispatcher. Pass custom_headers through _handle_non_streaming_allm_passthrough_route so early-return guardrail responses include x-litellm-call-id and related headers. * style: run black on handler and common_request_processing * refactor(proxy): dedupe non-streaming passthrough guardrail handling Replace the inline JSON/eventstream block in the streaming-request branch with a call to _handle_non_streaming_allm_passthrough_route so both paths share one implementation and cannot diverge. * fix(bedrock): preserve trailing bytes when re-encoding converse stream The event-stream re-encoder only emitted parsed frames, so any trailing bytes left after the parse loop (truncated/corrupt final frame, or fewer than 16 bytes after the last complete frame) were silently dropped from the de-anonymized output. Capture and re-append them so the transformer never truncates the stream. * fix(proxy): guard non-dict post-call hook return on bedrock passthrough JSON path * fix(proxy): guard malformed JSON body on bedrock passthrough guardrail path * fix(proxy): close guardrail bypass via tool result text and default-mode post-call guardrails on bedrock passthrough Pre-call extraction only read top-level Converse text blocks, so blocked content placed under toolResult.content[].text was forwarded to Bedrock without the key/team guardrail seeing it. Extraction now walks nested tool result text and write-back mutates the owning block in place. Post-call buffering for passthrough used _has_post_call_guardrails, which excludes event_hook=None guardrails. Those guardrails run at post_call, so their output processing was skipped and the raw upstream body was returned. Add a passthrough-specific predicate that counts them. * refactor(proxy): route bedrock event-stream de-anonymization through llm passthrough dispatcher Remove the hardcoded bedrock provider guard from common_request_processing by delegating event-stream de-anonymization to LlmPassthroughRouteHandler, which resolves the provider from the existing handler registry. Keeps proxy/ provider-agnostic and reuses the same dispatch path as the input and output guardrail handlers. Also log instead of silently dropping the result when post_call_success_hook returns a non-dict on the JSON and event-stream passthrough paths. * fix(proxy): close guardrail bypass on bedrock invoke passthrough routes Pre-call extraction and post-call output processing only handled Converse shapes, so /bedrock/model/{modelId}/invoke and invoke-with-response-stream returned unguarded. An authenticated caller could move blocked content into an InvokeModel payload and skip the key/team guardrail entirely. Non-Converse Bedrock routes now fall back to the generic passthrough handler, which scans the full request and response payloads so blocking guardrails still run, matching how other passthrough providers are guarded. * fix(proxy): keep non-bedrock passthrough streams streaming under post-call guardrails * fix(bedrock): scan non-text converse blocks for passthrough guardrails Key/team guardrails on bedrock converse passthrough only saw top-level text blocks, so a caller could hide prompt content in toolUse.input or toolResult.content[].json and have it forwarded to Bedrock without the configured guardrail inspecting it, bypassing blocking guardrails by default. Walk those arbitrary-JSON subtrees and write masked values back in place. Extend the non-streaming converse response path to the equivalent model-output fields (toolUse.input, reasoningContent text and citationsContent text) while leaving structural values such as reasoning signatures and citation sources untouched. * fix(bedrock): make passthrough guardrail string collection iterative and type-safe Rewrite _collect_strings with an explicit stack so it no longer recurses, satisfying the recursive-function CI guard, and widen the holder container type so mypy accepts indexing JSON nodes by str or int keys. * fix(proxy): scope passthrough post-call guardrail buffering to the request Buffering the Bedrock event stream into a single non-streaming response was gated on whether any post_call guardrail existed globally, so every converse-stream request lost streaming once any post_call guardrail was registered, even for keys that did not reference it. Mirror the gate used by post_call_success_hook (should_run_guardrail against the request's merged guardrails) so only requests whose key/team actually trigger a post_call guardrail are buffered. * fix(bedrock): guardrail non-text converse stream deltas on passthrough de_anonymize_event_stream only routed delta.text through the post-call guardrail, so model output streamed in reasoningContent.text, toolUse.input or citationsContent.content[].text was forwarded raw and skipped masking/blocking. Collect every user-visible text field per contentBlockDelta, concatenate per logical stream so split mask tokens still reassemble, run them through the hook, then redistribute the guardrailed text back into the matching delta fields. This brings streaming coverage in line with the non-streaming Converse output handler. * refactor(proxy): keep bedrock event-stream content-type detection in llms Move the vnd.amazon.eventstream content-type check out of the proxy passthrough path into BedrockPassthroughGuardrailHandler via the LlmPassthroughRouteHandler dispatcher, so proxy code stays provider-agnostic. Also patch the actually-called _has_post_call_guardrails_for_passthrough in the malformed-body regression test instead of the unused _has_post_call_guardrails. * fix(proxy): forward upstream headers on bedrock guardrail passthrough responses Mirror the non-guardrail passthrough path by merging the upstream response headers (via get_response_headers) into the guardrailed non-streaming and event-stream responses, so headers like x-amzn-requestid survive when a post-call guardrail rewrites the body. Drop the stray fastapi HTTPException import from the SDK-tree handler test in favor of a local sentinel exception. * fix(bedrock): scan tool definitions and additional request fields for passthrough guardrails Converse passthrough guardrails only scanned system and message content, so a key holder could route blocked or PII text through toolConfig tool names, descriptions and input schemas or through additionalModelRequestFields, all of which are still forwarded to Bedrock. Collect strings from those fields too so key/team guardrails inspect and rewrite them, matching how the chat-completions path forwards tool definitions to guardrails. * fix(bedrock): log instead of silently dropping passthrough guardrail edge cases * test(local): skip httpbin timeout probe when the service returns 5xx local_testing_part1 was failing on test_post_delay_exceeds_per_request_timeout_raises because httpbin.org/delay/10 intermittently answers 503 instead of delaying, so HTTPHandler.post raised MaskedHTTPStatusError rather than the expected Timeout. The test already means to skip when httpbin is unavailable, but its guard only probed GET /get and ignored a server error on the delay endpoint. Treat a 5xx from httpbin as 'service unavailable' and skip, which is outside this repo's control, while still asserting Timeout when httpbin genuinely delays. * fix(proxy): set content-type on buffered bedrock passthrough event-stream responses * fix(bedrock): skip passthrough output write-back when guardrail returns no texts * fix(proxy): apply response-headers hook on guardrailed bedrock passthrough responses * refactor(bedrock): import event-stream crc32 from binascii not botocore internals * fix(proxy): scope bedrock passthrough stream buffering to de-anonymizable endpoints Only buffer a passthrough event stream into a non-streaming response when the resolved provider and endpoint actually have an event-stream guardrail handler that can rewrite frames (Bedrock converse-stream). Other Bedrock event-stream endpoints such as invoke-with-response-stream keep streaming, since the Converse handler leaves their frames untouched and buffering would silently break the streaming contract for no content change. * test(ui-e2e): re-issue deep-link navigation when auth bootstrap drops the page param navigateToPage deep-links to /ui?page=<page> then proceeds once the network settles, but a fresh load can race the auth bootstrap: the app momentarily treats the session as anonymous, bounces through /ui/login, and returns to the default Virtual Keys page with the ?page= query param dropped. The helper never checked where it actually landed, so any single bounce left callers asserting against the wrong page and timing out (mcpServers, modelHub, addModel). Confirm the requested page is what rendered and re-issue the navigation when it was clobbered; auth is warm by the second load so the param sticks. Migrated path routes are left alone since they intentionally leave the legacy root. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-12 07:23:29 -07:00
Sameer Kankute	cfcdf8714a	feat: litellm oss 110626 (#30202) ... * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) (#29775) * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) Adds first-class support for the gpt-realtime-whisper streaming speech-to-text model, which uses the Realtime transcription session API rather than the file-based /audio/transcriptions path. Model registration: registers gpt-realtime-whisper and azure/gpt-realtime-whisper with audio-duration pricing (input_cost_per_second = 0.017/60, matching the published $0.017/minute input audio rate). REST endpoint: implements POST /v1/realtime/transcription_sessions (plus /realtime and /openai/v1 aliases) to mint an ephemeral transcription session for the WebRTC flow. Adds request/response types, OpenAI and Azure URL builders, a shared base handler (refactored from the client_secrets handler), the acreate_realtime_transcription_session SDK function, and route registration. The proxy encrypts the ephemeral key returned under client_secret.value and records the session type in the token so the follow-up /realtime/calls replays type=transcription rather than type=realtime. WebSocket: forwards intent=transcription through to the Azure handler (OpenAI already received it) with URL-encoding, so gpt-realtime-whisper opens a transcription session. Transcription-only sessions no longer trigger an erroneous response.create. Cost tracking: transcription sessions emit no response.done events; their usage arrives on conversation.item.input_audio_transcription.completed as {type: duration, seconds}. That usage is captured out-of-band (usage only, no transcript duplication) and billed by input_cost_per_second, with a token-billed fallback for token-priced transcription models. Adds tests for pricing math, URL builders, request/response types, the proxy route and SDK function, WebSocket intent forwarding, transcription-session streaming behavior, and the /realtime/calls session-type replay. * Address PR review: URL-encode all Azure WS query params; forward query_params through provider_config branch * Address PR review: session_type validation, model auth fix, cost perf, billing fallback, detail/docs cleanup * Improve test coverage: detection from backend, error paths, unknown usage type, resolved_model None * Backport realtime transcription websocket fixes * Enforce authorized realtime transcription model * Enforce realtime transcription model access * Enforce realtime resolved model scopes * Enforce WebRTC transcription model scope * Lazy evaluate debug log in pass-through endpoint (#30177) * Pass through debug lazy logging * fix(proxy): convert remaining eager pass-through debug logs to lazy formatting * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint (#30157) * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint The Parallel Search API moved from /v1beta/search (processor: base/pro, parallel-beta header) to /v1/search (mode: turbo/basic/advanced, no beta header). Request fields moved too: max_results, source_policy, and excerpt settings are now nested under advanced_settings, and source_policy uses include_domains/exclude_domains. The v1 response returns publish_date per result, which now maps to SearchResult.date instead of being hardcoded to None. The legacy processor param is mapped to the equivalent mode so existing callers keep working. * fix(parallel_ai): default mode to basic and simplify param handling The v1 API defaults to advanced mode when mode is omitted, while v1beta defaulted to the base processor. Without an explicit default, callers who pass no mode would be silently upgraded to a tier costing 2.25x more while litellm's cost map reports the basic-tier price. Sending mode=basic preserves the v1beta default and keeps cost tracking accurate. Also replaces the handled_params set with pop-as-consumed param handling so mapped params no longer need to be tracked in two places, and extends the tests to pin the default mode, processor=base mapping, mode-over-processor precedence, and top-level v1 param passthrough. * fix(parallel_ai): avoid double /v1 when api_base is already versioned A PARALLEL_AI_API_BASE like https://api.parallel.ai/v1 previously produced .../v1/v1/search. Strip a trailing /v1 before appending the search path and cover the api_base variants with a parametrized test. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(focus): add Mavvrik destination for FOCUS export (#29935) * fix: preserve responses streaming flag (#30189) * fix: preserve responses streaming flag * test: cover async responses streaming flag * fix(spend/daily-activity): stable offset pagination via id tiebreaker (#30164) (#30167) date alone is not a unique sort key for LiteLLM_DailyUserSpend or LiteLLM_DailyTeamSpend (many rows per date: api_key x model x model_group x provider x endpoint). Offset pagination over a non-unique sort landed on arbitrary boundaries, so a client paging through all results and summing per-page metrics (the Usage dashboard) got non-deterministic totals - sometimes inflated, sometimes deflated, different at different page_size values. Adding the row's UUID id (present on both tables) as a secondary sort gives every page a stable cursor. order=[{date desc}, {id asc}]. Fixes #30164 * fix(oci): inject a default maxTokens so omitted max_tokens doesn't truncate responses (#30018) * fix(oci): inject default maxTokens so omitted max_tokens doesn't truncate OCI GenAI applies a tiny server-side maxTokens default (~20 tokens) when the request omits it, so any call that doesn't send max_tokens comes back cut off mid-string with finishReason "length". MLflow judges never send max_tokens, so their JSON responses arrived as unterminated strings and json.loads failed in MLflow's gateway adapter. When no maxTokens/maxCompletionTokens target is set, inject DEFAULT_OCI_CHAT_MAX_TOKENS (env-overridable, defaults 4096), mirroring the Anthropic config's default-max-tokens behaviour. An explicit max_tokens still wins, and reasoning models still route to maxCompletionTokens. Used a fixed default rather than the catalog max_output_tokens because the catalog value is unreliable for some models (grok-4 reports max_output_tokens equal to its context window, not a real output cap, which would risk 400s). Adds TestOCIDefaultMaxTokens covering Cohere and generic injection, the explicit-override case, and the reasoning maxCompletionTokens branch. * test(oci): e2e regression that omitted max_tokens isn't truncated Real-proxy integration test asserting a chat completion that omits max_tokens completes with finish_reason "stop" instead of being cut off at OCI's ~20-token server default. Fails before the maxTokens-default injection (finish_reason "length", ~19 tokens), passes after. * test(oci): update cohere default-params test for injected maxTokens test_cohere_default_parameters asserted no maxTokens was injected, encoding the old behaviour where OCI's ~20-token server default truncated responses. Now that transform_request injects DEFAULT_OCI_CHAT_MAX_TOKENS, assert maxTokens equals that default while the other params (topK/topP/frequencyPenalty) stay pass-through with no hardcoded default. * fix(oci): make DEFAULT_OCI_CHAT_MAX_TOKENS a plain constant Drop the os.getenv override. The env knob was not requested and introducing a new env var forced a cross-repo dependency on litellm-docs (test_env_keys.py validates every referenced env var against the docs table there). A plain 4096 constant keeps the PR self-contained; callers who want a different limit pass max_tokens explicitly per request. * fix(oci): route all OpenAI commercial models to maxCompletionTokens OCI serves OpenAI models (gpt-4.1, gpt-5.1 through 5.5, o-series) that the litellm catalog doesn't track, so the supports_reasoning lookup returned False for them and the provider sent maxTokens, which the reasoning families reject with HTTP 400. With the injected default maxTokens this broke every request to those models, not just ones with an explicit max_tokens. Route the whole openai.* vendor prefix to maxCompletionTokens since OpenAI accepts max_completion_tokens on every chat model; the openai.gpt-oss-* open weights are served by OCI's own stack and keep maxTokens. Verified live against gpt-5.2, gpt-5, gpt-4o, gpt-4.1, gpt-oss-120b, llama-3.3, command-a and grok-3-mini * test(oci): hoist transformation imports and drop unused ones Makes the generic-chat test file ruff-clean: the per-test local imports of OCIChatConfig/OCIVendors shadowed the module-level import (F811) and left it unused (F401), and json plus three OCI type imports were never referenced * fix(oci): translate response_format json_schema to OCI's accepted shape (#29691) * fix(oci): translate response_format json_schema to OCI's accepted shape OCI GenAI rejected every json_schema response_format with HTTP 400 "Please pass in correct format of request", which broke structured-output callers such as MLflow LLM judges (they always send a json_schema). The provider forwarded OpenAI's raw json_schema body unchanged. For GENERIC models OCI's ResponseJsonSchema accepts only name/description/schema/isStrict, so OpenAI's `strict` key (and any other extra) 400s the request; the key must be renamed to isStrict and the body whitelisted. For Cohere models there is no JSON_SCHEMA type at all; the schema has to ride on JSON_OBJECT as {"type": "JSON_OBJECT", "schema": ...}. Cohere type values must also be the canonical uppercase TEXT/JSON_OBJECT. _normalize_response_format now branches by vendor and emits the exact shape each one accepts (verified live against OCI GenAI for Cohere, Meta, Gemini and Grok). Drops the unused, incorrect Cohere response-format pydantic models. Two existing tests asserted the broken behavior (lowercase type, raw jsonSchema on Cohere); they are rewritten to assert the corrected shape, and generic/Cohere json_schema regression tests are added. * fix(oci): raise early on json_schema response_format with no body A GENERIC model request with {"type": "json_schema"} and no json_schema object fell through to the JSON_OBJECT branch and emitted a bodyless {"type": "JSON_SCHEMA"}, which OCI rejects with an opaque HTTP 400. Raise a descriptive 400 at translation time instead. Cohere is unaffected since it always maps to JSON_OBJECT. * test(oci): gateway integration test for response_format json_schema Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): accept default n=1 on Cohere instead of hard-failing (#29705) * fix(oci): accept default n=1 on Cohere instead of hard-failing Cohere on OCI has no numGenerations field, so n was mapped to False and map_openai_params raised "param `n` is not supported on OCI" whenever a client sent n. But n=1 (and None) is the OpenAI default single-generation request, which every OCI model produces anyway, so standard clients that always send n=1 (such as the MLflow gateway) were rejected with a 500. Drop n=1/None silently for Cohere; only n>1 is genuinely unsupported and still raises (or drops under drop_params). Generic models are unaffected and keep numGenerations, including n>1. * docs(oci): explain why n is not advertised for Cohere despite tolerating n=1 * test(oci): gateway integration test for Cohere default n=1 Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): drop max_retries instead of hard-failing on OCI (#29727) max_retries is a litellm-level control param (litellm applies retries itself), not a generation param OCI accepts. The provider mapped it to False and raised "param `max_retries` is not supported on OCI" whenever it was present. The litellm proxy injects max_retries on every request, so any OCI call through the proxy 500'd unless drop_params was set. Drop max_retries silently in map_openai_params. Adds a unit test (Cohere and generic) and a gateway integration test that a plain request succeeds through a proxy without drop_params. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): rehydrate metadata JSONB text on ui_view_spend_logs (#29682) Fixes #29674. `/spend/logs/ui` raw-SQL path returns the JSONB metadata column as a string — prisma's query_raw skips the ORM-layer hydration. The UI reads metadata.status / metadata.error_information as object fields, so provider-failure rows look like successes. Fix: json.loads the metadata field right after query_raw, fall back to {} on malformed JSON. 3 existing error-code/error-message tests called json.loads on response.data[0]["metadata"] — they were leaning on the bug. Updated to read the dict directly. Plus 2 new regression tests (failure metadata roundtrip + invalid-json fallback). Reverting the fix makes both new tests fail with AssertionError: metadata should be dict, got <class 'str'>. * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) (#30020) * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) * fix: refund max_parallel_requests on disconnect from outer streaming generators The cancellation refund previously lived in async_post_call_streaming_iterator_hook, but that hook is nested inside the outer streaming generators and a nested async generator only receives GeneratorExit on garbage collection (non-deterministic). With only the v3 limiter enabled, /chat/completions also bypasses the hook entirely (needs_iterator_wrap() is false). Move the release into async_data_generator and async_streaming_data_generator, the generators Starlette closes on client disconnect, so the refund fires deterministically on every streaming route. Warn when no event loop is running, and document the window TTL refresh on the decrement * fix(mcp): propagate model into model_call_details for passthrough tool calls (#30122) * fix(mcp): propagate model into model_call_details for passthrough tool calls The @client decorator on call_mcp_tool creates the logging object via function_setup without a model kwarg, so model_call_details["model"] starts as None. execute_mcp_tool only set logging_obj.model as an instance attribute, which the spend-log writer never reads (it reads kwargs["model"] from model_call_details). MCP passthrough tools/call rows therefore persisted with model="" while list_tools rows showed "MCP: list_tools", degrading the Logs UI display and bucketing all MCP tool spend under an empty model in DailyUserSpend. Propagate the model into model_call_details alongside the existing attribute assignment so the StandardLoggingPayload and SpendLogs writer pick it up. Covers the /mcp passthrough, REST /mcp-rest/tools/call, and orchestrated paths (the latter already passed model into function_setup, so this is a no-op there). * test(mcp): trim regression test docstring * fix(mcp): surface upstream challenges for delegated OAuth (#30124) * fix(mcp): surface upstream challenges for delegated OAuth * docs(mcp): clarify delegated upstream auth comments * perf(benchmarks): add CPU timing metrics to streaming benchmark (#29980) * Add CPU timing metrics to streaming benchmark * Fix spacing around timing sample dataclass * fix(gemini): don't emit empty choices on metadata-only stream chunks (#29167) web_search + reasoning makes Gemini stream mid-chunks that carry only grounding/thought metadata — no content part, no finishReason. _process_candidates skips content-less candidates and the existing fallback only ran when finishReason was set, so choices stayed empty and the downstream streaming handler raised IndexError on choices[0]. Emit an empty-delta choice for content-less chunks regardless of finishReason. Fixes #28884 * fix(key): allow /key/update to clear budget_limits with [] or null (#30085) * Fix /key/update rejecting budget_limits clear requests with HTTP 400 Sending budget_limits: [] or null to /key/update returned HTTP 400, so once a key had budget windows the last one could never be removed. prepare_key_update_data only json.dumps'd budget_limits when the value was truthy, so [] and None passed through raw to the Prisma Json? column; jsonify_object only serializes dicts, and prisma-client-py has no DbNull sentinel for Json? writes, so Prisma rejected both shapes. Serialize the clear case explicitly as the JSON literal null, matching how memory_endpoints encodes metadata for the same column type. Truthy values keep the existing reset_at window initialization path. Fixes #30067. * Require admin access for budget_limits changes on /key/update Clearing budget_limits via [] or null is a budget mutation, but _validate_update_key_data only counted max_budget and spend as budget changes before deciding whether to skip _check_key_admin_access. A non-admin key owner or a team member with /key/update could therefore remove a key's per-window spend caps without admin authorization. Treat any explicit budget_limits value in the request (set, change, or clear) as a budget change so it gates through the same admin check as max_budget. model_fields_set is used because an explicit null is indistinguishable from an omitted field by value alone. * fix(proxy): persist guardrail info in spend logs for /v1/responses (#30092) Pre-call guardrail blocks on /v1/responses wrote guardrail_information as null in LiteLLM_SpendLogs because _handle_logging_proxy_only_error splits request_data by LoggedLiteLLMParams keys and litellm_metadata, where the Responses API stores request metadata including standard_logging_guardrail_information, was not among them. It fell into optional_params, so merge_litellm_metadata never saw it. Add litellm_metadata to LoggedLiteLLMParams so it routes into litellm_params the same way metadata does on the chat completions path Fixes #28971. * fix(proxy): handle non-standard SSE frames in Anthropic passthrough logging (#26000) Some third-party Anthropic-compatible providers emit non-standard SSE frames (OpenAI-style [DONE] sentinels, non-JSON keep-alive lines) in streaming responses. These caused json.JSONDecodeError in _build_complete_streaming_response, breaking the passthrough logging pipeline so the request was never logged or billed. Skip whole-line 'data: [DONE]' sentinels and catch JSONDecodeError per event. Matching the full line (not a substring) keeps a valid chunk whose text payload contains '[DONE]' from being dropped. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(newrelic): Add New Relic extension (#26989) * initial New Relic integration. * Minor fixes for basic observability. * Implemented basic support for the success path. Generates New Relic custom events needed by the AI Monitorin interface. * Supportability metric is sent on first request. * Emit supportability metric every hour instead of once a day. * Add the start/end times to the messages before sending them so that the start time and end time reflect the correct time and both are not set to 'now'. * Make use of `turn_off_message_logging` configuration that is available by default from CustomLogger. * Enabling New Relic agent to be wired when docker container starts if an environment variable is set. * If we cannot find trace information, send the AI events without the trace ID attached. * Use a fake trace_id if we cannot find one. * Implementing a configuration so that users can use litellm configuration to disable sending LLM messages to New Relic. There is a second method to do this via New Relic env var. * Mised file. * Cleaning up logic to turn off recording content via either the LiteLLM configuration or an env var. * Removing debugging. Fixed logic / comments around how often to send supportability metric. * Initial version of public doc for New Relic. * Use a proper name for the doc file. * Updating newrelic.md document. * Updating LiteLLM documentation for New Relic extension. * Moving New Relic imports into the methods to support unit tests. * Adding unit tests for the New Relic extension. * Updating linting and the unit tests that are not running in the CI environment. * Address reviewer feedback on New Relic integration. - Fix _record_error_metric to use app.record_custom_metric() instead of module-level newrelic.agent.record_custom_metric() so the call works outside of an active transaction context - Remove unreachable except ImportError block in _get_trace_context - Update stale "23 hours" comment to "27 hours" (matches 97200s threshold) - Remove commented-out debug code from _process_success - Fix docs typo: NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STOREDA -> NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STORED - Update TestRecordErrorMetric to verify app.record_custom_metric call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Reformating for the linter. * Addressing additional automated feedback. - Removed a legacy comment about the New Relic header - Reordered imports in one file - Switched another file to use the import at the top of the file instead of inline when used - Added unit tests for untested methods that were identified * Addressing new feedback. - Proper handling of time to floats. Created a util method and updated code to use it. - added the missing guard to ensure the app is enabled * Addressing feedback. - When an error occurs, still check if the periodic supportability metric should be emitted - Added a check to ensure the extension is ready in the error handler to match _process_success * Updating the NR event timestamps to more accurately reflect when the messages were generated. * Addressing feedback for potential better practice. * Addressing feedback on accessing default values. Added tests for most of these cases. * Adding a new catch exception block based on feedback. * Addressing feedback about a potential issue around a timestamp for the supportability metric. * Addressing minor feedback on length of generated, fallback traceId. * Addressing feedback. - A few more cases were found where the dictionary access might not return the correct value. - Handling cases where `traceparent` is not lower cased * Addressed feedback where the newrelic options might not apply correctly. * Addressing some feedback. * Addressing feedback. * Validating testing / formatting for our changes. * Updating linting, adding tests, defining data type for UI. * Configuration for the logging callback definition. * Adding a newrelic image for the UI to use. * Putting the New Relic callback in proper alphabetic order. * Copying the logo to a committed output directory so it shows up in a locally built container. * Adding missing definition of new env vars that were causing a build failure. * Addressing automated feedback from greptile. * Adding a few more unit tests to increase the code coverage just a bit more. * Additional unit tests to push coverage to almost 90%. * Adding a custom newrelic docker image build process. This removes the need to add the newrelic agent to the core litellm container or dependencies. * Clarifying message when the New Relic agent is not installed and someone is trying to use the newrelic extension. Either use the proper image when using docker, or install the agent manually when running from source. * Ensuring pip is available to install the New Relic agent. * Updating the definition and handling of traceId (no spanId). Clarifying behavior of env vars vs UI configuration for the newrelic extension. * Removing entries from the New Relic logger configuraiton UI as these values must be set as part of running the image. * Removing a stale doc file that has moved to the litellm-docs repo. Cleanup of Dockerfile to remove a LABEL that was incorrect. * Updating container image name to be the best guess for the new name. * Addressing feedback from greptile. - Added a comment around token_count=0 - Updated the boolean parser to allow a wider set of options which matches existing patterns in other parts of LiteLLM. * Removing option for a separate New Relic container image. The agreement is to handle this in the New Relic integration docs. * Updating error message when New Relic agent is not available. * Wiring in the test message from the LiteLLM callback UX. * Missed saving one of the file conflicts. * Fixed a lint error I introduced. Somehow, I dropped another string and now added it back. * Adding newrelic to the schema definition. * Added an admin check on the call before sending test message as mentioned by the AI code review. * Updating to use should_redact_message_logging(kwargs) as part of the logic to determine if message content should be sent to New Relic or not. This still uses the `record_content` property as well, but both have to be true in order for content to be included. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add Azure AI Foundry DeepSeek V3.1 and V4 Pro/Flash global pricing to cost map (#30134) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(logging): translate Responses bridge result to ModelResponse for spend logs (#28985) PR #29394 fixed the AnthropicResponse.model_validate crash for the streaming anthropic_messages -> OpenAI Responses bridge by unwrapping terminal events and returning the inner ResponsesAPIResponse. The spend_logs row lands and usage/cost are correct, but the row's response field stores the Responses API shape (output[...].content[...].text). The proxy UI Logs tab reads response.choices[0].message via parseMessages in prettyMessagesUtils.ts with no fallback for the Responses shape, so the OutputCard renders "No response data available" for every cross-routed call. The same shape mismatch affects every downstream consumer of spend_logs that assumes the canonical chat-completion shape This change keeps the unwrap from #29394 but routes the resulting ResponsesAPIResponse (and the bare-response non-streaming path) through LiteLLMResponsesTransformationHandler.transform_response, which is the same conversion already used by the chat-completion Responses bridge. Spend_logs now stores a ModelResponse with choices[0].message.content, so the UI and other consumers see the assistant text. On a translation failure (eg. empty output on an incomplete response) the handler falls back to a minimal ModelResponse carrying model and usage so the row still lands rather than being dropped as a Non-Blocking error Also corrects a stale comment in the Responses adapter that implied the call type was reclassified to acompletion; the code preserves anthropic_messages and the success handler translates back to ModelResponse for the row Fixes #28595 * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions (#30024) * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions The `/v1/messages` -> `/v1/chat/completions` streaming adapter (`AnthropicStreamWrapper`) silently dropped the first non-empty delta of every content block that started via a *transition* (e.g. text -> tool_use -> text, text -> thinking). When an upstream chunk both triggers a new content block (its type differs from the active block) and carries that block's first delta, the wrapper emitted `content_block_stop` -> `content_block_start` and then only re-queued the trigger chunk when it was an `input_json_delta` (bundled tool args). The synthesized `content_block_start` always carries an empty body, so the first `text_delta` / `thinking_delta` was lost — the client output started from the second token (e.g. "Hi, how can I help you?" rendered as ", how can I help you?", or text resuming after a tool call lost its first sentence). This is especially visible with Claude Code-style clients that consume Anthropic Messages streaming events strictly. Fix: re-queue the trigger chunk's translated delta whenever it carries non-empty content (text/thinking/signature/tool args), via a shared `_trigger_delta_has_content` helper used by both the sync and async paths. Empty trigger deltas are still suppressed so no spurious empty `content_block_delta` is introduced. Fixes #30014 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(anthropic-adapter): cover all _trigger_delta_has_content branches Add a direct parametrized unit test for the re-emit predicate so every delta type (text/input_json/thinking/signature), the empty-payload guards, and the malformed/non-delta cases are exercised independently of upstream chunk translation. Raises patch coverage for the new helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat: add opt-in healthy_only filter to GET /v1/models (#30130) * feat: add opt-in healthy_only filter to GET /v1/models Adds an opt-in `healthy_only=true` query parameter to GET /v1/models and GET /models that hides models whose backing deployments are all marked unhealthy by background health checks. - Add Router.async_get_fully_unhealthy_model_names(), mirroring the semantics of get_fully_blocked_model_names(): a model is hidden only when every backing deployment is unhealthy and the health state is not stale (fail open otherwise). - Reuses the existing DeploymentHealthCache populated by _run_background_health_check(), so no new health state is introduced. - No-op when allowed_fails_policy is set, mirroring _async_filter_health_check_unhealthy_deployments semantics. - team_public_model_name aliases are aggregated alongside model_name. - Hiding is presentation-only; default behavior is unchanged. Fixes #30128 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: address Greptile review notes - Note team-alias asymmetry vs get_fully_blocked_model_names - Debug-log when healthy_only is set but no health state is available Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Dedupe team soft budget alerts by team_id instead of token (#30097) _team_soft_budget_check sends type="soft_budget" alerts with event_group=TEAM, but SoftBudgetAlert.get_id always returned the request token. The alert cache key was therefore scoped per virtual key, so every active key in a team over its soft budget fired its own alert within budget_alert_ttl. Branch on event_group so team-level alerts dedupe by team_id, matching TeamBudgetAlert, while key and project level alerts keep per-token dedupe. Fixes #27398. * feat(bedrock guardrails): support contextual grounding qualifiers (request-side) (#30057) * test: add failing tests for Bedrock contextual grounding (request-side) Drive the request-side of Bedrock contextual grounding: callers tag message content blocks as grounding_source/query, the post_call hook assembles an ApplyGuardrail(OUTPUT) call carrying source + query + response(guard_content), and the bedrock converse transform must render the tags as prompt text instead of silently dropping them. Non-grounding payloads must stay byte-identical. * feat(bedrock guardrails): support contextual grounding qualifiers Bedrock contextual grounding scores a model response against a reference source and the user query, expressed via a per-content-block `qualifiers` array on ApplyGuardrail. The guardrail hook previously sent plain text only, so grounding could not be driven through it even though the response-side contextualGroundingPolicy parsing already existed. Callers now tag message content blocks `{"type":"grounding_source"}` / `{"type":"query"}` (mirroring the existing `guarded_text` marker). On the generate path the bedrock converse transform renders them as plain text; at post_call the hook harvests them from the request and assembles one ApplyGuardrail(OUTPUT) call carrying grounding_source + query + the response (as guard_content). Requests without these tags produce a byte-identical payload, so existing behaviour is unchanged. * Feat(guardrail): Adding support for custom Ovalix guardrail (#21887) * Feat(guardrail): Adding support for custom Ovalix guardrail * Internal CR comments fixes * greptileai comments fixes * fix conflict * fixes * fix sha256 * clarify Ovalix actor-id hash is for normalization, not PII protection * fix(github_copilot): normalize per-event item_id in /responses streaming (#30072) GitHub Copilot's native /v1/responses stream assigns a different item_id to every event of a single output item (output_item.added, the part.added / delta / done events, and output_item.done). Spec-strict clients like the Vercel AI SDK key streaming parts by item_id and abort with "reasoning part <id> not found" / "text part <id> not found" when a delta references an unregistered id. Override transform_streaming_response in GithubCopilotResponsesAPIConfig to anchor every event of an output item to the id from its output_item.added. Copilot accepts that id paired with the final encrypted_content on the next turn, so multi-turn replay is unaffected. Fixes #30071 * feat: add /model/block and /model/unblock endpoints (#30125) * feat: add /model/block and /model/unblock endpoints Add dedicated proxy-admin POST /model/block and /model/unblock endpoints over the existing blocked flag on LiteLLM_ProxyModelTable, mirroring the /key/block and /key/unblock pattern. Calling a model whose deployments are all blocked now returns a clear 403 "Model is blocked" instead of a generic no-deployment error, including direct-dispatch route types (e.g. eval) via a pre-route guard. Includes audit-log entries for block/unblock and unit tests. Closes #29742 Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * chore: regenerate dashboard API types for model block/unblock endpoints Regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts from the proxy OpenAPI spec (npm run gen:api) so it includes the new endpoints. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: widen router block-helper param type and add direct unit tests Type the _are_all_deployments_blocked deployments parameter to match its callers (DeploymentTypedDict) so mypy passes, and add tests/test_litellm/test_router_block_helpers.py with direct unit tests for the three block helper methods so router_code_coverage recognizes them. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: restore type-ignore on messages arg after black reflow Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * refactor: raise model-block 403 in proxy layer, not SDK Router Keep the SDK Router's documented behavior for blocked deployments (filtered -> "no healthy deployment") and move the 403 PermissionDeniedError into the proxy layer (route_llm_request), where model blocking is an admin concept. This avoids a backwards-incompatible 403 for SDK users who set blocked=True on their own deployments, per maintainer review. Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: add week unit support to get_next_standardized_reset_time (#30100) * fix: add week unit support to get_next_standardized_reset_time The function handled d/h/m/s/mo units but silently fell through to the default next-midnight branch for the w (week) unit. This was inconsistent: _extract_from_regex already accepted w in its character class, and duration_in_seconds already returned value * 604800 for it. Add the missing elif unit == 'w' branch that delegates to _handle_day_reset with value * 7, which reuses the existing Monday- alignment logic for 1w and the generic N-day-from-midnight path for larger multiples. Add test_week_based_resets covering 1w from a Wednesday (expects next Monday) and 2w from a Monday (expects 14 days forward at midnight). Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * test: exercise relative week semantics with non-Monday base dates + add docstring Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * fix: black formatting and remove undocumented MAVVRIK_FOCUS_FREQUENCY env var * fix: black formatting with correct version and sync schema.d.ts for healthy_only param * fix: resolve mypy errors and add transcription_sessions to JSON schema endpoint enum * fix: restore MAVVRIK_FOCUS_FREQUENCY guard and exclude it from docs key scan * fix: address Greptile P2 comments - move constant, use UTC datetime, skip redundant team lookup * revert: restore original team lookup logic in can_key_call_resolved_model --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: nina-hu <nina.huuu@gmail.com> Co-authored-by: Sahith Jagarlamudi <104647530+s-jag@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com> Co-authored-by: alex107ivanov <30668368+alex107ivanov@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com> Co-authored-by: Teo Xian Zhong Augustine <35527068+auggie246@users.noreply.github.com> Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: Saksham Maggo <122939011+SakshamMaggo@users.noreply.github.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Kelvin <leikaiwei@outlook.com> Co-authored-by: Josh Bonczkowski <josh.bonczkowski@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: M. Dennis Turp <mdturp@pm.me> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Piotr Minkina <piotrminkina@users.noreply.github.com> Co-authored-by: Martín Alcalá Rubí <martin@tryolabs.com> Co-authored-by: T. Kobayashi <13004314+nix-tkobayashi@users.noreply.github.com> Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: Shalom <shalom@ovalix.io> Co-authored-by: codgician <15964984+codgician@users.noreply.github.com> Co-authored-by: FugoP <kim@pomsora.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-11 22:30:26 -07:00
Sameer Kankute	9ddf0535b1	fix(proxy): skip double-wrapping unified batch output file ids on retrieve (#30011) ... * fix(proxy): skip double-wrapping unified batch output file ids on retrieve After ensure_batch_response_managed_file_ids normalizes output_file_id, the managed files post-call hook was re-encoding the unified id and storing the nested id as the provider mapping. Use the decoded llm_output_file_id for retrieve and model_mappings instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): guard managed file id parsing for non-output unified formats Only treat decoded unified ids as already-wrapped output files when they contain llm_output_file_id. Skip other litellm_proxy id shapes instead of IndexError on split. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): rename loop variable to satisfy mypy unified file id typing Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-11 22:00:43 -07:00
Sameer Kankute	c30297e98a	fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate (#29946) ... * fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate Map input_audio_buffer.commit/end to Gemini audioStreamEnd (or activityEnd for manual VAD) so burst user audio triggers server_vad after response.done. Use 24kHz PCM MIME for Vertex native-audio sessions. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(gemini-live): honor input_audio_buffer.clear during deferred setup replay Apply clear semantics when buffering and flushing pre-setup audio frames so cleared appends are not forwarded to Gemini Live after setup completes. Co-authored-by: Cursor <cursoragent@cursor.com> * style: black-format realtime_streaming.py for py312 CI Co-authored-by: Cursor <cursoragent@cursor.com> * Fix realtime working with gaurdrails * fix(realtime): remove unused GuardrailEventHooks import Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-11 22:00:15 -07:00
Mateo Wang	250d8d2a96	fix(a2a): forward agent_extra_headers through completion bridge (#28277) ... * fix(a2a): forward agent_extra_headers through completion bridge A2A agents backed by a custom_llm_provider (e.g. langgraph, bedrock_agentcore) silently dropped any per-request headers rewritten from the inbound `x-a2a-{agent}-*` convention or admin-configured `extra_headers`. The headers were correctly extracted in `a2a_endpoints.py` but never passed into `_send_message_via_completion_bridge` or the bridge handler, so the upstream HTTP request reached the agent backend without them. Thread `agent_extra_headers` through: - asend_message / asend_message_streaming -> bridge call sites - _send_message_via_completion_bridge - A2ACompletionBridgeHandler.handle_non_streaming / handle_streaming - Inject as `extra_headers` into the underlying litellm.acompletion() call, and forward to provider configs via kwargs (their **kwargs signature absorbs it harmlessly today). * fix(a2a): forward agent_extra_headers through bridge convenience wrappers Address greptile review on PR #28277: - handle_a2a_completion / handle_a2a_completion_streaming (the public, exported convenience wrappers) now accept agent_extra_headers and forward it to the underlying class methods. Without this, callers going through the public API would still silently drop per-request headers — the exact regression this PR fixes for the class-method path. - Add the missing agent_extra_headers entry to the handle_streaming docstring for parity with handle_non_streaming. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(a2a/bedrock): forward agent_extra_headers to AgentCore HTTP request Address greptile follow-up on PR #28277: BedrockAgentCoreA2AConfig was absorbing agent_extra_headers via **kwargs but never propagating to the underlying HTTP POST, so x-a2a-{agent}-* rewrites and admin extra_headers were silently dropped on the bedrock_agentcore path that bypasses the completion bridge. Thread the parameter through the full Bedrock AgentCore stack: - config.handle_non_streaming / handle_streaming pull agent_extra_headers from kwargs and pass to the handler. - handler.handle_non_streaming / handle_streaming accept it and forward to the transformation layer. - transformation.get_url_and_signed_request merges agent_extra_headers into the headers dict BEFORE signing, so SigV4 covers them in the signature. JWT/Bearer path: AgentCore signer always overwrites Authorization with api_key, so use api_key (not agent_extra_headers) to override the bearer token. Also fix a pre-existing test assertion that was already broken by the parent commit ab70ff6 (test_provider_config_receives_litellm_params didn't include agent_extra_headers in the expected call). Tests: - TestTransformation::test_agent_extra_headers_merged_into_signed_headers_jwt - TestTransformation::test_agent_extra_headers_signed_for_sigv4 - TestNonStreaming::test_agent_extra_headers_forwarded_on_outbound_post Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(a2a/bedrock): drop reserved AWS headers from agent_extra_headers Per veria-ai security review on PR #28277: agent_extra_headers carries values rewritten from the client-controlled x-a2a-{agent}-* convention, so the unconditional 'headers.update(agent_extra_headers)' in BedrockAgentCoreA2ATransformation.get_url_and_signed_request let any caller with access to an agent overwrite headers the proxy sets from trusted server-side config -- most notably X-Amzn-Bedrock-AgentCore-Runtime-User-Id, which AWS treats as the runtime identity. Because the merge happened before SigV4 signing, the spoofed value would also be bound into a valid signature. Strip reserved AWS/AgentCore headers (authorization, host, x-amzn-bedrock-agentcore-runtime-*, x-amz-*) from agent_extra_headers before merging and log a warning when any are dropped. Legitimate per-request headers (e.g. x-mcp-token, x-tenant) still pass through. Adds two tests covering both the JWT path (verifies the spoof does not land on the outbound headers) and the SigV4 path (verifies the signer never sees the spoofed values). Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(a2a): annotate completion_params dict for mypy The dict literal initializing completion_params had heterogeneous value types (str, list, bool), so mypy inferred the value type as a narrow union that did not accept dict[str, str] when assigning extra_headers. Annotate completion_params as Dict[str, Any] in both the non-streaming and streaming bridge handlers so the agent_extra_headers merge type-checks cleanly. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(a2a/pydantic_ai): forward agent_extra_headers to upstream HTTP request * fix(a2a/bridge): admin litellm_params.extra_headers win over caller-rewritten headers agent_extra_headers contains both admin static_headers and caller-derived dynamic headers (from the x-a2a-{agent}-* rewrite). Merging it last would let a caller replace headers that the proxy was configured to send upstream via litellm_params.extra_headers. Flip the merge order so admin-configured headers take precedence on conflict. * fix(a2a/headers): merge_agent_headers compares case-insensitively HTTP header names are case-insensitive, but the previous merge was a case-sensitive dict update. That meant an admin-configured static_headers['Authorization'] (capital A) did not strip a caller-rewritten x-a2a-{agent}-authorization (lowercase, from the inbound header normalization in a2a_endpoints) - both ended up on the outbound request to pydantic_ai / langgraph / etc. Restore the documented 'static wins on conflict' invariant by comparing case-insensitively when overlaying static_headers. Static side's casing is preserved on the output. * fix(a2a/bridge): merge configured extra_headers case-insensitively over caller headers A caller-rewritten lowercase header (e.g. authorization from the x-a2a-{agent}-* convention) could ride alongside an admin-configured case-variant key in litellm_params.extra_headers, sending duplicate Authorization headers upstream. The bridge now reuses merge_agent_headers so configured headers win case-insensitively, in both the non-streaming and streaming paths. merge_agent_headers moved to litellm.interactions.agents.utils (re-exported from the proxy utils) so the SDK-level bridge does not import from litellm.proxy. https://claude.ai/code/session_017cBvda8Y4CLo8wspB2kfSV --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-06-11 21:56:18 -07:00
Yassin Kortam	1828a7c6f0	fix(passthrough): resolve costing model when body model is unknown (#30160)	2026-06-11 14:26:55 -07:00
michelligabriele	8e12d42ea7	fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity (#30151)	2026-06-11 22:32:08 +02:00
Yassin Kortam	a992ed18df	feat(spend_logs): opt-in native Postgres partitioning for SpendLogs retention (#29466) ... High-volume deployments see LiteLLM_SpendLogs grow unbounded because retention via DELETE leaves dead tuples that autovacuum cannot reclaim fast enough. With a range-partitioned table, retention drops whole partitions instead: an instant metadata operation that returns disk to the OS immediately. The feature is gated behind general_settings.use_spend_logs_partitioning (default false). With the flag off, the cleanup job never queries the catalog and behaves exactly as today. With it on, the job verifies the table is partitioned, pre-creates upcoming partitions, and drops expired ones; expired rows the drops cannot reach (DEFAULT partition, partitions spanning the cutoff) are still deleted row-wise so retention is never bypassed. If the table is not partitioned it falls back to batched DELETE only. Converting an existing table is a manual, documented operation in db_scripts/partition_spend_logs.sql; db_scripts/unpartition_spend_logs.sql rolls it back. Both scripts rename the old table's indexes aside before recreating them, since a table rename keeps the schema-unique index names and would otherwise silently skip the CREATE INDEX IF NOT EXISTS block. Granularity and pre-create lookahead are tunable via SPEND_LOG_PARTITION_INTERVAL (day/week/month, invalid values fall back to day) and SPEND_LOG_PARTITION_PRECREATE_AHEAD.	2026-06-11 11:02:42 -07:00
Yassin Kortam	012d9f6c0a	feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker (#30211)	2026-06-11 10:34:26 -07:00
ryan-crabbe-berri	0d120de785	chore(hooks): enforce Conventional Commits and Conventional Branches (#30174) ... * chore(hooks): enforce Conventional Commits and Conventional Branches Adds opt-in local git hooks plus a CI PR-title check: - .githooks/commit-msg validates commit subjects against Conventional Commits 1.0.0 (feat|fix|docs|style|refactor|perf|test|build|ci| chore|revert)(scope)!: subject. Merge/revert/fixup!/squash!/amend! messages pass through; --no-verify still works. - .githooks/pre-push validates branch names against Conventional Branches (feature|bugfix|hotfix|release|chore)/desc. Bypasses main, litellm_internal_staging, dependabot/*, gh-readonly-queue/*. Tag pushes and deletions are skipped. - scripts/install_git_hooks.sh sets core.hooksPath=.githooks and is wired up as 'make install-hooks'. Opt-in — not chained into install-dev. - .github/workflows/conventional-commits.yml validates PR titles via amannn/action-semantic-pull-request pinned to v6.1.1's SHA. This is the actual gate since squash-merge uses the PR title as the commit subject. - tests/test_litellm/test_git_hooks.py exercises both hooks via subprocess for accept / reject / bypass / git-generated-message cases. - CONTRIBUTING.md documents the conventions, the install step, the bypass list, and the --no-verify escape hatch. Resolves LIT-3306 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(hooks): address Greptile review on PR #28703 Resolves two findings from the automated code review: 1. CONTRIBUTING.md: shrink the new Conventional Commits / Branches section to a 2-line pointer at docs.litellm.ai. Per the team convention, the full documentation lives in the litellm-docs repo — see BerriAI/litellm-docs#208 for the companion change that adds the section to docs/extras/contributing_code.md. 2. .githooks/commit-msg: tighten the subject regex to also reject an uppercase first letter in the description. CI's subjectPattern is ^(?![A-Z]).+$ so the previous local hook would accept 'feat: Add thing' which would then fail the PR-title check. The local hook is now the strictly tighter of the two gates. Test cases extended to cover both the new rejection and the digit/symbol-start cases that remain allowed. Resolves LIT-3306 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: trigger ci after branch rename * fix(ci): rerun pr title check when bypass label changes amannn/action-semantic-pull-request only honors ignoreLabels if the workflow retriggers on labeled/unlabeled events; without them a red check stays red after a maintainer applies the bypass label. Also point the CONTRIBUTING.md workflow comments at the conventions section, which now sits above the Development Workflow section. --------- Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MBP.localdomain> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-06-11 10:00:23 -07:00
Mateo Wang	49ca04d8c3	feat(bedrock): aws_bedrock_project_id for bedrock-mantle project / workspace association (#30163) ... * feat(bedrock): support aws_bedrock_project_id for bedrock-mantle project association Adds a litellm_params field to associate bedrock-mantle requests with an Amazon Bedrock project, sent as the OpenAI-Project header on the OpenAI-compatible chat and responses paths and as the anthropic-workspace header on the Anthropic messages paths. This lets a single model entry opt into a project-scoped data retention mode (e.g. provider_data_share for Claude Fable 5) while the account-wide setting stays on default. The param is carried via litellm_params only and is explicitly excluded from optional_params so it can never leak into a request body. Fixes #30070 * chore(ui): regenerate schema.d.ts for aws_bedrock_project_id Generated with npm run gen:api after adding the field to LiteLLM_Params * fix(proxy): ban client-supplied aws_bedrock_project_id in request bodies The deployment pins aws_bedrock_project_id so the project's data retention policy applies to its requests. Without this guard an authenticated caller could supply the field in the request body and, since client kwargs win the router merge, run requests under any project reachable with the deployment's shared AWS credentials. Adds the field to _BANNED_REQUEST_BODY_PARAMS so it is rejected at the auth boundary by default while remaining available through the existing admin opt-ins (allow_client_side_credentials proxy-wide or configurable_clientside_auth_params per deployment).	2026-06-11 10:01:08 +05:30
Mateo Wang	7a96b3490d	[internal copy of #30137] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay (#30142) ... * perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay The GA realtime support added in #27110 made backend_to_client_send_messages parse every backend frame up to three times for beta clients (OpenAI-Beta: realtime=v1), build a discarded Pydantic object per frame for logging, and re-serialize even frames that need no translation. For high-frequency response.output_audio.delta frames carrying multi-KB base64 payloads, that serialized CPU work on the hottest relay path drove the latency regression between v1.83.14 and v1.88.1 for gpt-realtime-1.5 and gpt-realtime-2. This parses each frame once via _parse_backend_event and threads the dict into _handle_raw_backend_message, store_message, and _translate_event_to_beta; short-circuits store_message before the Pydantic build for events not in the logged set; returns the original event unchanged from _translate_event_to_beta when no rename applies so the raw frame is forwarded without re-serialization; and only json.dumps when the type is actually renamed. * fix(realtime): widen store_message type hint to accept plain dict The parse-once refactor passes the dict produced by _parse_backend_event into store_message, but the parameter was typed as str | bytes | OpenAIRealtimeEvents (a union of TypedDicts), which mypy does not consider compatible with a plain dict. Add dict to the accepted union; the body already handles it. --------- Co-authored-by: Miguel Armenta <maarmenta92@gmail.com>	2026-06-11 09:56:35 +05:30
ishaan-berri	4a3860df1f	fix: completion_cost AttributeError on streaming Anthropic web_search responses (#26153) (#27346) ... * fix: coerce server_tool_use dict to ServerToolUse in Usage.__init__ (#26153) * fix: coerce server_tool_use to ServerToolUse in stream_chunk_builder (#26153) * fix: dict/pydantic-tolerant access in tool_call_cost_tracking (#26153) * fix: dict/pydantic-tolerant access in anthropic cost_calculation (#26153) * test: assert ServerToolUse type in existing stream_chunk_builder anthropic web search test * test: regression test for #26153 (stream_chunk_builder server_tool_use type) * test: dict/pydantic safety for tool_call_cost_tracking helper * test: dict/pydantic safety for anthropic web_search cost * refactor: consolidate _get_web_search_requests into shared cost-calc utils * test(realtime): use gpt-realtime; openai retired gpt-4o-realtime-preview OpenAI shut down the gpt-4o-realtime-preview family (incl. the undated alias) on 2026-05-07, causing the live realtime test to fail with a 4000 invalid_request_error.invalid_model close. gpt-realtime is the GA successor; switch the live-call tests to it, matching the base branch. * refactor(types): drop redundant server_tool_use coercion in Usage.__init__ --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-10 21:20:11 -07:00
Sameer Kankute	6068bb7781	fix(proxy): align /v1/model/info with router deployments (#30025) ... * fix(proxy): align /v1/model/info with router deployments Return router model_list entries (including team-scoped models) with team access metadata instead of wildcard-expanded names from get_complete_model_list. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): gate v1 team filter and honor key allowlists Only apply get_all_team_and_direct_access_models for admin or user-bound keys, then intersect with key/team model restrictions to avoid empty lists for service tokens and metadata leaks for restricted keys. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): skip v1 team filter when user row is missing Require a DB-backed user before applying team-access filtering on /v1/model/info, and skip the trailing filter in get_all_team_and_direct_access_models when user context cannot be resolved. Co-authored-by: Cursor <cursoragent@cursor.com> * Revert "fix(proxy): skip v1 team filter when user row is missing" This reverts commit 74e1fbd77a. * fix(proxy): restore legacy v1 model access filtering Keep /v1/model/info on key/team allowlists instead of DB team-membership filtering, while still listing router deployments for team-scoped models. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): drop A2A agent entries from public /v1/model/info list * fix(proxy): scope team BYOK rows on /v1/model/info to caller's teams Listing the full router model_list let any authenticated key without explicit model restrictions enumerate other teams' BYOK deployments (public name, team_id, api_base) via /v1/model/info. Reuse the existing _get_caller_byok_team_scope check so non-admin callers only see global deployments plus their own team's BYOK rows; admins keep the full view. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-10 19:38:21 -07:00
Yassin Kortam	da9d64b4de	fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures (#29986)	2026-06-10 23:48:11 +00:00
Yassin Kortam	dff25fef44	feat(proxy): add option to disable server-side prepared statements for DB lookups (#29984)	2026-06-10 16:06:32 -07:00
Yassin Kortam	3bd3951e37	fix(proxy): recover from cached-plan errors by reconnecting the Prisma client (#29983)	2026-06-10 16:06:01 -07:00
tin-berri	1436ee9092	fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted (#30141)	2026-06-10 15:56:58 -07:00
yuneng-jiang	7899463c6a	fix(callbacks): forward callback_settings to callback initializers and guard consumers against non-dict values (#30161) ... * fix(datadog): pass callback_specific_params so DatadogCostManagementLogger receives cost_tag_keys (#29590) * fix(datadog): pass callback_specific_params so DatadogCostManagementLogger receives cost_tag_keys * test(proxy): regression test that load_config forwards callback_specific_params * fix(proxy): guard lakera_prompt_injection callback_specific_params against non-dict Addresses review feedback: forwarding callback_settings as callback_specific_params (so DatadogCostManagementLogger receives cost_tag_keys) exposed the lakera_prompt_injection branch, which did lakeraAI_Moderation(**callback_specific_params ["lakera_prompt_injection"]) with no type guard. A config like `callback_settings: {lakera_prompt_injection: "any-string"}` then hit `**"any-string"` -> TypeError: argument after ** must be a mapping, not str. Guard the lakera branch with isinstance(dict), matching the existing presidio and datadog_cost_management branches (non-dict values fall back to {}). Add a regression test asserting initialize_callbacks_on_proxy ignores a non-dict value instead of crashing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: inject fake lakera_ai module to avoid importing the real one CI fix for the lakera regression test: it stubbed litellm.proxy.proxy_server with a SimpleNamespace and then monkeypatch.setattr'd the real lakera_ai module, which forces importing it — and lakera_ai does `from litellm.proxy.proxy_server import LiteLLM_TeamTable`, absent on the stub -> ImportError under proxy-infra tests. Inject a fake lakera_ai module into sys.modules instead, so the callbacks branch's `from ...lakera_ai import lakeraAI_Moderation` resolves to the stub without loading the real module. The guard under test (isinstance(dict) in the lakera branch) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(callbacks): guard compression/websearch interceptors against non-dict callback_settings (#30153) #29590 forwards the full callback_settings dict into initialize_callbacks_on_proxy, which activates the compression_interception and websearch_interception consumers. Their initialize_from_proxy_config read the callback_settings subkey without an isinstance(dict) guard, so a non-dict value such as `compression_interception: true` reached from_config_yaml(...).get(...) and aborted proxy startup with AttributeError. #29590 added that guard for lakera_prompt_injection but not for these two Mirror the isinstance(dict) guard already used by the lakera, presidio, and datadog branches so a non-dict value is ignored and the callback initializes with defaults. A parametrized test feeds every callback_settings consumer a non-dict value through initialize_callbacks_on_proxy to catch a future consumer that forgets the guard * fix(callbacks): normalize non-dict callback_specific_params to empty dict A blank callback_settings: key in YAML loads as None, and config.get('callback_settings', {}) returns None because dict.get only falls back to the default when the key is absent. Forwarding that value verbatim to initialize_callbacks_on_proxy made the first '<name>' in callback_specific_params membership test raise TypeError: argument of type 'NoneType' is not iterable, aborting proxy startup. Same failure for any non-dict root such as callback_settings: true. Normalize the value at the function boundary so both callsites (and any future ones) initialize callbacks with their defaults instead of crashing. --------- Co-authored-by: Hedi Daoud <150018939+hdaoud23@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-10 15:22:00 -07:00
Mateo Wang	20e453f698	feat(cli): per-agent lite claude / codex / opencode commands that wrap coding agents through the proxy (#29850) ... * feat(cli): add `litellm-proxy run -- <agent>` to wrap coding agents through the proxy Wraps Claude Code, Codex, OpenCode, and any other coding agent so all of its LLM traffic routes through a LiteLLM proxy, with the agent-vault style of "just works" DX: one `run -- <agent>` command, auto SSO login when interactive, env-key "agent mode" for containers/CI, and a fail-fast key check against the proxy so bad credentials error immediately instead of deep inside the agent. The wrapped binary is detected by name to pick the right variables. Claude Code gets ANTHROPIC_BASE_URL (the bare proxy root, so it appends /v1/messages) and ANTHROPIC_AUTH_TOKEN, with any stray ANTHROPIC_API_KEY cleared so the proxy token wins. Codex and OpenCode get OPENAI_BASE_URL (proxy + /v1) and OPENAI_API_KEY. Unrecognized commands get both sets so they work either way. `litellm-proxy claude-code` remains as a shortcut for `run -- claude`. The core logic is split into dependency-injected helpers (agent_profile, build_agent_env, verify_proxy_key, run_agent) so env wiring, the preflight, and the launch handoff are unit-tested without monkeypatching, alongside CliRunner tests for auth resolution, agent mode, and auto-login. Mutation-tested the env profiles, preflight, and agent-mode branch to confirm the tests fail when the behavior is broken. https://claude.ai/code/session_0154VpLXW7mMvk5wfbgPRJa6 * Make each coding agent its own litellm-proxy command Replace the `run -- <agent>` interface and the `claude-code` shortcut with top-level commands generated per known agent, so launching is just `litellm-proxy claude`, `litellm-proxy codex`, or `litellm-proxy opencode`, with everything after the agent name forwarded straight to it. This drops the ceremony of `run --` and cuts typing. The `--model`/`--small-fast-model` wrapper flags are gone; pass the agent's own model flag instead, or export the model env vars (the wrapper preserves what you already have set), which keeps the surface minimal and avoids intercepting flags the agent owns. Rename the module to agents.py to match. * fix(cli): route `litellm-proxy codex` through the proxy via a custom provider Codex ignores OPENAI_BASE_URL (it always dials api.openai.com over the Responses WebSocket transport), so the OpenAI env profile alone left `litellm-proxy codex` talking to OpenAI directly instead of the proxy. Point Codex at the proxy with a custom provider passed as `-c` config overrides, and force the HTTP/SSE Responses transport with supports_websockets=false since the proxy does not speak the Responses WebSocket protocol. The provider reads its key from OPENAI_API_KEY, which the agent env already exports. The overrides are injected ahead of the user's args so they precede Codex's subcommand. Claude Code and OpenCode are unaffected; they honor the exported env vars. Adds regression tests for the per-agent launch args and the injection ordering. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * Rename litellm-proxy CLI command to lite The proxy management CLI was invoked as litellm-proxy, which is a lot to type for an everyday command. Rename the console script entry point to lite and update the in-CLI usage examples, help text, error messages and docs to match. * fix(sso): stop CLI auth success page from hanging on "Closing..." The CLI opens the SSO success page with webbrowser.open, so the tab is not script-opened and the browser refuses window.close(). The countdown would end on "Closing..." and the tab would sit there forever. Drop the countdown and just show "You can now close this window and return to your terminal." from the start, while still attempting window.close() once so the tab auto-closes in the rare case the browser allows it. Add a regression test asserting the manual-close instruction is always present and the misleading countdown/"Closing..." text is gone. * fix(cli): reattach controlling terminal after SSO login, keep litellm-proxy alias When the first `lite claude` has to log in via browser SSO, completing the login could leave stdin detached from the terminal, so a TUI agent like Claude Code would start in non-interactive mode and exit with "Input must be provided". The wrapper now reopens the controlling terminal onto stdin just before handoff when the session started interactively; piped or redirected input is detected up front and left alone, so agent-mode and non-interactive use are unchanged. Also keep the `litellm-proxy` console script as an alias for `lite` so existing scripts and CI that invoke `litellm-proxy` keep working; both names map to the same CLI. * feat(install): make the curl installer need only curl, not a pre-existing Python The installer now lets uv provision a managed Python 3.13 when no suitable interpreter is found, instead of aborting. The minimum is also bumped from 3.9 to 3.10 to match the package's requires-python (>=3.10), so a system Python 3.9 is no longer selected only for uv tool install to reject it. * feat(cli): add thin litellm[cli] install path (install-cli.sh + brew) for the lite CLI On a developer laptop the `lite` CLI only needs `lite login` and running coding agents through a proxy, but the sole install path was `litellm[proxy]`, which drags in the whole server tree (fastapi, uvicorn, boto3, polars, cryptography, litellm-enterprise). The CLI's heavy imports are all guarded, so it runs on the base SDK plus just rich, pyyaml and requests. Add a `cli` extra carrying exactly those three, a `scripts/install-cli.sh` curl one-liner that installs `litellm[cli]`, and a `BerriAI/homebrew-litellm` tap formula with a release runbook under `packaging/homebrew/`. The installer passes no `--python`, so uv honours litellm's requires-python and provisions a managed interpreter, skipping a too-old (3.9) or too-new (3.14+) system Python instead of failing to resolve. A pyproject thin-contract test asserts the `cli` extra keeps the deps the CLI imports and never leaks a server-only dependency from `proxy`, so the laptop install cannot silently re-bloat * fix(install): let uv pick the Python via --python-preference system Both installers detected a system Python with a floor-only check and forced it with `uv tool install --python <interp>`. On a host whose only Python is outside litellm's requires-python (a too-old 3.9 or, increasingly, a too-new 3.14) that forced an incompatible interpreter and the resolve failed. Drop the detection and pass `--python-preference system`: uv reuses a compatible system Python when present and downloads a managed one otherwise, always honouring requires-python * test(router): filter aiohttp unclosed-session gc noise in test_async_fallbacks test_async_fallbacks asserts the last three captured log records are the router's fallback messages. Under the litellm_router_testing job (pytest -k router -n 4) many router tests share the module-level in_memory_llm_clients_cache (max 200, ttl 3600s). Older cached OpenAI/Azure clients get evicted while their aiohttp ClientSession is still open, and when the gc reclaims them aiohttp emits "Unclosed client session"/"Unclosed connector" through the asyncio logger. Those records land in caplog mid-test and push the expected router logs out of the last-three window, so the assertion flips to failing non-deterministically. These warnings are async cleanup noise, not router debug logs, so filter them out exactly like the existing leaked-task warnings before asserting order. The assertion on the three router fallback messages is unchanged. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-06-10 13:52:26 -07:00
Mateo Wang	a4a3348801	[internal copy of #28007] Fix/gcp model garden streaming (#28363) ... * fix(vertex): stream Model Garden Gemma/Qwen responses correctly through /v1/messages * test(vertex): cover _CombinedChunkSplitter defensive branches * test(databricks): rename test file to avoid duplicate basename collision * fix(databricks,anthropic): defensive token defaults; document single-mode splitter Address greptile P2 concerns: - databricks: default usage token fields to 0 when constructing ChatCompletionUsageBlock from a partially populated usage block — matches the defensive pattern used in ollama/vertex_ai/cohere/bedrock. - _CombinedChunkSplitter: clarify in the docstring that an instance is single-mode (sync or async, not both), since the two iteration paths hold independent upstream iterator references. Co-authored-by: Claude <claude@anthropic.com> --------- Co-authored-by: Steven Kessler <9701252+stvnksslr@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com>	2026-06-10 12:31:00 -07:00
Yassin Kortam	410b892f77	fix(register_model): preserve built-in cache pricing when registering custom overrides under unmapped keys (#30044) ... * fix(spend-tracking): fall back to direct spend-counter increment when reservation reconcile fails When the reservation-reconcile path in `_reconcile_budget_reservation_for_counter_update` hits a Redis error, it now correctly returns an empty set so that `increment_spend_counters` re-runs the direct increment for the affected counters. Previously, the function logged the failure, invalidated the reserved counters, and still returned the reserved counter keys, which caused the caller to skip the direct increment. With the increment skipped and the counter deleted, the next request reseeded the counter from `LiteLLM_VerificationToken.spend`, a column the batched flusher only updates every few seconds, so the enforced cross-pod spend value collapsed to a stale snapshot and budget gating stopped firing for affected keys. Adds a regression test that exercises the failure path with a flaky redis backend and asserts the actual response cost lands in the shared counter. * fix(register_model): preserve built-in cache pricing when registering custom overrides under unmapped keys When a custom-priced model is registered under a key shape that get_model_info cannot resolve (e.g. litellm_params.model set to bedrock/bedrock/us.anthropic.claude-sonnet-4-6 or another non-canonical alias), register_model previously fell back to an empty existing_model. The merged entry then carried only the fields the user set explicitly (input/output cost, provider) and dropped cache pricing. Downstream the cost calculator defaulted cache_creation_input_token_cost and cache_read_input_token_cost to 0, silently dropping the bulk of the bill for cache-heavy Anthropic traffic. register_model now attempts to resolve a canonical built-in entry by stripping provider prefixes, region prefixes, and provider-specific suffixes before giving up. When a variant resolves, its defaults (notably cache pricing) are inherited while the user's explicit overrides still win. When nothing resolves and the user supplied no cache pricing, it logs a warning instead of silently under-billing. * fix(router): inherit built-in cache pricing on deployments with partial custom pricing A deployment configured with only input_cost_per_token and output_cost_per_token under model_info was being registered under its model_info.id with no cache cost fields. The cost calculator then defaulted cache_creation_input_token_cost and cache_read_input_token_cost to 0, silently billing cache_read and cache_creation tokens at zero. For cache-heavy Anthropic traffic this drops the bulk of the bill. When the deployment's litellm_params.model resolves to a built-in cost-map entry, pull the cache pricing fields from there before registering. User-specified cache fields still win on merge; only missing fields are inherited. Pairs with the register_model fallback added earlier in this branch: that handles unmapped key shapes like bedrock/bedrock/x, this handles deploy-id keys whose backend model is mapped. * fix(register_model): inherit only cache pricing on unmapped-key fallback, not provider The unmapped-key fallback in register_model copied the entire resolved built-in entry, so registering openai/command-r-plus inherited the cohere built-in's litellm_provider and get_model_info(custom_llm_provider=openai) could no longer resolve it. Restrict the fallback to the cache-pricing fields, matching the router-side _inherit_builtin_cache_pricing, so the cache-cost dropout stays fixed without clobbering the registered provider. Add a direct unit test for Router._inherit_builtin_cache_pricing so the router coverage check sees it, and pin the fixed spend-counter contract: when reservation reconcile fails the counter must hold the directly incremented cost rather than being left at None.	2026-06-10 12:11:03 -07:00
michelligabriele	f9293d40c4	fix(proxy): self-heal startup/reload prisma reads on engine disconnect (#28803)	2026-06-10 20:16:58 +02:00
Sameer Kankute	3b40ac987f	Litellm oss 090626 (#30021) ... * fix(mcp): report scoped server name during initialize (#29865) * fix mcp scoped server name * Update litellm/proxy/_experimental/mcp_server/mcp_context.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(mcp): cover scoped server name in the SSE initialize handler --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): show all session logs in the drawer, not just the first 50 (#29795) * fix(ui): show newest session logs first * test(ui): keep session log pagination coverage * fix(ui): show all session logs in the drawer, not just the first page The session detail drawer fetched session logs via sessionSpendLogsCall without page/page_size, so it only ever received the backend default of one page (50 rows). Sessions with more than 50 calls had the rest unreachable in the UI (#29153). sessionSpendLogsCall now takes page/page_size, and the drawer fetches the first page, reads total_pages, then fetches the remaining pages and accumulates them before the existing client-side sort. This keeps the single continuous list (and the selected-log lookup and keyboard navigation, which all assume the full session) correct. Fetching is bounded by a page cap, and the sidebar shows a "showing most recent N" note if a session exceeds it. The rows are lightweight metadata (the endpoint excludes messages/response), so the full set is small; request/response bodies are still loaded per log on demand. * fix(ui): default session drawer to most recent log, newest first Open a session with its most recent log selected, and order the sidebar newest-first to match the all-sessions logs overview. MCP calls stay grouped last. The latest log by time is computed explicitly, since the MCP grouping means it is not always the first row. * Apply fetching pages in batches suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): derive session total from accumulated rows when backend omits it Compute the session total after all pages are fetched, falling back to the accumulated row count rather than the first page's. Guards the truncation note against a backend response that omits total but spans multiple pages. --------- Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): handle Mistral multipart passthrough (#29927) * fix(proxy): handle Mistral multipart passthrough * chore: satisfy passthrough ci formatting * test(proxy): cover Mistral passthrough in CI shard * fix(vertex_ai): use REP host for context caching on eu/us multi-region endpoints (#29573) Context caching built the cachedContents URL as https://{location}-aiplatform.googleapis.com, which is an invalid host for the eu/us multi-region endpoints and returns 404. The inference path already resolves these to the REP host (https://aiplatform.{geo}.rep.googleapis.com) via get_vertex_base_url(); reuse that helper in _get_token_and_url_context_caching so caching uses the same host as inference. Adds tests covering the eu/us multi-region cachedContents URLs (v1 and v1beta1). Fixes #29571 * Support per-model encrypted content affinity config (#29760) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: propagate upstream status code in proxy API exception handler (#29402) * fix: propagate upstream status code in proxy API exception handler When Google GenAI / Vertex returns a 404 for deprecated or missing models via streamGenerateContent, the exception was falling through to a generic handler that defaulted to 500. Now provider exceptions carrying a valid HTTP status_code correctly propagate it through to the ProxyException. * fix: apply black formatting to common_request_processing.py * fix: tighten status code range to 400-599 and deduplicate ProxyException raise * fix(tests): use valid vertex_location in context caching tests Replace "test_location" (contains underscore) with "us-central1" so tests pass the regex validation added in get_vertex_base_url(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sdk): add xAI OAuth provider (#29866) * Add xAI OAuth provider * Update oauth.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix xAI OAuth CI failures * Add xAI OAuth coverage tests * Move xAI OAuth coverage tests to core utils * Address xAI OAuth review comments * Prevent xAI OAuth api_base token exfiltration * Treat blank xAI OAuth api keys as absent * Wrap invalid xAI OAuth JSON responses * Use xAI OAuth behind explicit flag --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy) #27734 allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update (#27751) * fix(proxy): allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update Fixes #27734 Sending null for budget_duration, team_member_budget, team_member_budget_duration, team_member_rpm_limit, or team_member_tpm_limit via /key/update or /team/update returned 200 OK but silently ignored the null value. The fields remained unchanged in the database. Root causes: - /key/update: prepare_key_update_data() popped budget_duration from the update dict but never re-added it (or budget_reset_at) when the value was None. - /team/update: _set_budget_reset_at() only acted when budget_duration was non-None, leaving a stale budget_reset_at in the DB. - /team/update: team_member_* null values bypassed the budget table update entirely because should_create_budget() requires at least one non-None field. * test(proxy): cover no-budget-row path in clear_team_member_budget_fields * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes (#30028) * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes When output_parse_pii=true on the Anthropic native path (anthropic/claude-*), response chunks arrive as raw bytes in SSE format. _stream_pii_unmasking was yielding those bytes unchanged, so <PERSON_1> tokens were never replaced with the original values before reaching the caller. Add _unmask_sse_bytes_chunk to parse each data: line, find content_block_delta / text_delta events, and apply _unmask_pii_text before re-encoding. Wire it into _stream_pii_unmasking so bytes chunks are unmasked when pii_tokens exist. * fix(presidio): handle CRLF line endings and non-ASCII PII in SSE unmask Strip trailing \r before the [DONE] guard so CRLF-terminated SSE chunks don't bypass it and silently swallow a JSONDecodeError. Add ensure_ascii=False to json.dumps so non-ASCII replacement values like accented names are preserved as UTF-8 on the wire rather than being \uXXXX-escaped. Add regression tests for both cases. * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) (#29925) * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) Bedrock Mantle serves the Responses API on two upstream paths: - gpt frontier models (gpt-5.5 / gpt-5.4) on /openai/v1/responses - every other Responses-capable model (e.g. gpt-oss) on the standard /v1/responses BedrockMantleResponsesAPIConfig gains a `use_openai_path` flag; the provider gate in utils.py picks the path per model: openai.gpt-* (non gpt-oss) -> /openai/v1/responses; any model declared mode=responses (price-map entry or user model_info) -> /v1/responses; everything else returns None and keeps the existing chat-completions emulation. Adds gpt-5.5 / gpt-5.4 price-map entries, registry wiring, and the routing-matrix tests. * feat(bedrock_mantle): data-driven frontier routing via use_openai_responses_path Addresses the Greptile review point that frontier detection should be a price-map field rather than a hardcoded name match. The gate now routes a model to /openai/v1/responses when its price-map entry declares use_openai_responses_path, so a frontier model whose name does not follow the openai.gpt- convention can be onboarded by JSON alone. The name-convention check is kept as a fallback that needs no price-map entry, which preserves zero-change routing for a future gpt-6 before its entry loads. gpt-5.5 / gpt-5.4 get the flag in both price maps. Adds tests for the data-driven flag path and for the flag presence on the gpt-5.x entries; both branches are mutation-tested. * test(model_prices): allow use_openai_responses_path in price-map schema The model_prices_and_context_window.json schema validator (test_aaamodel_prices_and_context_window_json_is_valid) enforces additionalProperties: false, so the new use_openai_responses_path flag on the gpt-5.5 / gpt-5.4 entries failed validation. Add it to the schema as a boolean, alongside the other supports_* / capability flags. * Add Tensormesh serverless models to the model cost map (#30037) * Add Tensormesh serverless models to the model cost map * Flag reasoning support on the Tensormesh models that expose thinking mode * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update (#30001) * fix(proxy): reconcile stale key spend counter after budget reset * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update * fix(proxy): remove read-time stale counter reconciliation to prevent budget bypass * revert: undo unrelated formatting changes in enterprise directory * test(proxy): add unit test for key spend update invalidating counter * test(proxy): fix mocked update_data and hash token expectations in unit test * fix(proxy): use Responses-API transformer in pass-through cost tracking (#29728) The `elif is_responses:` branch of `openai_passthrough_handler` was calling the chat-completions `transform_response` on a Responses API payload. The chat-completions transformer expects `choices: [...]` in the raw response; the Responses API uses `output: [...]` and `usage.input_tokens` / `usage.output_tokens` (not `prompt_tokens` / `completion_tokens`). The result was a KeyError 'choices' deep inside `convert_to_model_response_object`, swallowed by the surrounding `except Exception` in the handler, and the SpendLogs row was written by the fallback path with zeroed-out tokens, spend, and model. This bug silently undercounts cost for every successful pass-through call to either OpenAI's `/v1/responses` or Azure's `/openai/v1/responses` (deployments configured for the Responses API). Reproduced 2026-06-04 against a real Azure OpenAI Responses API deployment proxied through LiteLLM v1.88.0. Fix: use the dedicated `OpenAIResponsesAPIConfig.transform_response_api_response` for the Responses branch. This transformer already exists in LiteLLM (`litellm/llms/openai/responses/transformation.py`) and knows the Responses-API on-the-wire shape. `litellm.completion_cost` already handles `ResponsesAPIResponse` natively with `call_type="responses"`, so no downstream changes are needed. Tests: test_responses_api_uses_responses_transformer_not_chat_completions NEW. Real regression test — exercises the openai_passthrough_handler with a real-shaped Responses payload (no `choices`, has `output` and Responses-API `usage` keys) and NO mocked `get_provider_config`. Pre-fix: raises KeyError 'choices' inside the chat-completions transformer (the bug). Post-fix: returns a ResponsesAPIResponse, completion_cost is called with call_type="responses" and a ResponsesAPIResponse instance (asserted). Verified to fail on un-fixed handler + pass on fixed handler before commit. test_responses_api_cost_tracking UPDATED. Old test mocked `get_provider_config` (no longer called in the responses branch post-fix). Now mocks the Responses transformer directly (`OpenAIResponsesAPIConfig.transform_response_api_response`) to test the downstream cost-calc contract. Out of scope for this PR (separate followup): - Recognizing *.cognitiveservices.azure.com (the newer Azure OpenAI hostname) in the is_openai_*_route checks. Separate PR. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(skills): execute DB skills by matching the litellm_skill_ tool name prefix (#30116) Skill IDs are generated as litellm_skill_<uuid> and the model-facing tool name is the sanitized skill ID, but the post-call execution gates in SkillsInjectionHook only ran tools whose name starts with "skill_", so DB skills were silently returned to the client as raw tool calls. Fixes #28122. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): synthesize content_block_start when Responses stream omits output_item.added (#30115) * fix(team): reserve team budget raises for proxy admins on /team/update (#30030) The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974) * test(ui): add a data-driven App Router migration E2E smoke Add a growing Playwright smoke for migrated pages: for each segment it deep-links to the path route, asserts the URL and that the dashboard shell rendered, then clicks off to a legacy page and asserts navigation still works. Driven by e2e_tests/fixtures/migratedPages.ts, so adding a page is one line. Runs in two situations against the same proxy: the default mount (npm run e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root). globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage state is valid under a prefix. Seeded with api-reference; append the rest as their migrations merge. * test(ui): support headed slow-motion + watch pauses in the migration smoke Honor SLOWMO in the server-root-path config (the default config already did), and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state. Both are no-ops by default, so CI behavior is unchanged. * test(ui): make the migration smoke a sidebar-click user journey Rework the smoke from deep-linking to a real navigation journey: start at the landing page, click the migrated page in the sidebar (expanding submenus for nested items), assert the path route rendered, reload it (the check a wrong server_root_path breaks), bounce to a legacy page and back, and — once two pages are migrated — navigate directly between two migrated pages. Verifies via URL + shell render, driven by the same fixture list. * test(ui): address review on the migration smoke Escape ROOT and segment before interpolating them into RegExp URL matchers so a future segment containing regex metacharacters can't silently widen the match. Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead of silently re-running the default mount and passing without exercising the prefix. * test(ui): drop unused watch helper and fix stale smoke README * test(ui): run the migration smoke under a server root path in CI * test(ui): harden + instrument the server-root-path proxy reboot in CI * test(ui): run the server-root-path migration smoke as its own CI job Replace the in-place proxy reboot in e2e_ui_testing with a dedicated e2e_ui_testing_server_root_path job that boots the proxy once with SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the config gets its own job rather than killing and relaunching the live proxy. The reboot was failing deterministically: after pkill -9 and relaunch the prefixed proxy never came back up on :4000 (connection refused), so the smoke never ran. The readiness step that was supposed to surface the cause could never reach its boot-log tail because CircleCI runs steps under bash -eo pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's exit 7. Booting the proxy as the job's own background step lets any boot crash land in that step's log instead of being swallowed. The default e2e_ui_testing job is unchanged aside from dropping the reboot, prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at the root mount there via the default Playwright config. * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests * chore(ui): remove dead App Router route stubs under (dashboard) (#30045) models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them * fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000) * fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss * fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation * docs(security): require a reproduction video for vulnerability reports (#30048) (#30063) With AI models capable of automated vulnerability discovery now publicly available, we expect a large increase in report volume, much of it unverified. Requiring a video of the exploit running against a live instance raises the bar for submissions and keeps triage focused on reproducible issues. Reports without a video will be closed and reopened if one is added later. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> * feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. * chore(ui): remove dead dashboard files and unused dependencies (#30047) * chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit 30d2e96f77. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064) * Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context, 128K output, adaptive thinking only) on the Anthropic API, Bedrock converse (base, global, and us/eu geo inference profiles at the 10% regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which serves Fable 5 with the full 1M context window unlike Opus 4.8). Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the model in the setup wizard, and extends the reasoning effort e2e grid. The Bedrock, Vertex, and Azure grid cells carry fail_reason markers until the CI accounts are provisioned: Bedrock needs the provider data sharing opt-in Fable 5 requires, and the Foundry resource needs a claude-fable-5 deployment. The first-party entry carries provider_specific_entry {us: 1.1} for the inference_geo premium and deliberately no fast multiplier since Fable 5 has no fast mode. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drop removed sampling params for Claude 4.7+ when drop_params is set Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was forwarding them even with drop_params enabled because the Anthropic and Bedrock converse transformations passed temperature/top_p through unconditionally. Mirror the GPT-5/o-series handling: temperature=1 still passes through, other values and any top_p are dropped when drop_params is set, and without drop_params a clean client-side UnsupportedParamsError tells the caller how to opt in, instead of surfacing the raw provider error. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drive sampling param gating from the cost map and cover top_k Greptile review follow-ups on the sampling param fix: the restriction for Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false on every affected cost map entry (perplexity excluded; that route is OpenAI-compatible and maps sampling params upstream) and read back through a tri-state map lookup, keeping the name check only as a fallback for provider-routed ids whose hosted map entries predate the flag, the same layering supports_adaptive_thinking uses. top_k bypasses map_openai_params as a provider-specific kwarg, so it is gated at the shared AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex, Azure) and in the Bedrock converse _handle_top_k_value path, with drop_params threaded through the converse transform helpers. Also updates the reasoning effort grid cell count assertion for the four Fable 5 rows added on this branch (29 x 11 cells). https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Declare supports_sampling_params in the cost map schema The model map validation schema uses additionalProperties: false, so the new flag must be declared for the 28 entries that carry it; this was the one failing job (misc / Run tests) on the previous commit. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix(bedrock): gate top_k=0 on converse to match Anthropic boundary Truthiness check let top_k=0 silently disappear on models that removed sampling params, while AnthropicConfig.transform_request treats 0 as present and raises UnsupportedParamsError (or drops when drop_params is set). Switch to 'is not None' so converse, direct Anthropic, invoke, Vertex, and Azure all behave the same for top_k=0. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> * fix(anthropic): avoid index -1 content_block_delta in messages stream When a /v1/messages request is routed through the Responses API adapter, AnthropicResponsesStreamWrapper only emits content_block_start on response.output_item.added. Some upstreams (LMStudio for example) never send that event, so the text delta handler fell back to _current_block_index, which starts at -1, and clients received content_block_delta events with index -1 and no preceding content_block_start. Anthropic SDKs then fail with "text part -1 not found" The text delta handler now synthesizes a content_block_start with a fresh block index whenever the delta references an unregistered item_id or no block is open yet, and registers the item_id so follow-up deltas reuse the same index Addresses the /v1/messages defect in #27442 * Make test sys.path shim resolve relative to the file, not the CWD os.path.abspath("../../../../../../..") depends on where pytest is invoked from; anchoring on os.path.dirname(__file__) makes the import work from any working directory. Also corrects the depth: the repo root is six levels above this file, not seven. --------- Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: tin-berri <tin@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> * fix: enable compact-2026-01-12 beta header for vertex_ai provider (#30114) * fix(team): reserve team budget raises for proxy admins on /team/update (#30030) The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974) * test(ui): add a data-driven App Router migration E2E smoke Add a growing Playwright smoke for migrated pages: for each segment it deep-links to the path route, asserts the URL and that the dashboard shell rendered, then clicks off to a legacy page and asserts navigation still works. Driven by e2e_tests/fixtures/migratedPages.ts, so adding a page is one line. Runs in two situations against the same proxy: the default mount (npm run e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root). globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage state is valid under a prefix. Seeded with api-reference; append the rest as their migrations merge. * test(ui): support headed slow-motion + watch pauses in the migration smoke Honor SLOWMO in the server-root-path config (the default config already did), and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state. Both are no-ops by default, so CI behavior is unchanged. * test(ui): make the migration smoke a sidebar-click user journey Rework the smoke from deep-linking to a real navigation journey: start at the landing page, click the migrated page in the sidebar (expanding submenus for nested items), assert the path route rendered, reload it (the check a wrong server_root_path breaks), bounce to a legacy page and back, and — once two pages are migrated — navigate directly between two migrated pages. Verifies via URL + shell render, driven by the same fixture list. * test(ui): address review on the migration smoke Escape ROOT and segment before interpolating them into RegExp URL matchers so a future segment containing regex metacharacters can't silently widen the match. Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead of silently re-running the default mount and passing without exercising the prefix. * test(ui): drop unused watch helper and fix stale smoke README * test(ui): run the migration smoke under a server root path in CI * test(ui): harden + instrument the server-root-path proxy reboot in CI * test(ui): run the server-root-path migration smoke as its own CI job Replace the in-place proxy reboot in e2e_ui_testing with a dedicated e2e_ui_testing_server_root_path job that boots the proxy once with SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the config gets its own job rather than killing and relaunching the live proxy. The reboot was failing deterministically: after pkill -9 and relaunch the prefixed proxy never came back up on :4000 (connection refused), so the smoke never ran. The readiness step that was supposed to surface the cause could never reach its boot-log tail because CircleCI runs steps under bash -eo pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's exit 7. Booting the proxy as the job's own background step lets any boot crash land in that step's log instead of being swallowed. The default e2e_ui_testing job is unchanged aside from dropping the reboot, prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at the root mount there via the default Playwright config. * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests * chore(ui): remove dead App Router route stubs under (dashboard) (#30045) models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them * fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000) * fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss * fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation * docs(security): require a reproduction video for vulnerability reports (#30048) (#30063) With AI models capable of automated vulnerability discovery now publicly available, we expect a large increase in report volume, much of it unverified. Requiring a video of the exploit running against a live instance raises the bar for submissions and keeps triage focused on reproducible issues. Reports without a video will be closed and reopened if one is added later. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> * feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. * chore(ui): remove dead dashboard files and unused dependencies (#30047) * chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit 30d2e96f77. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064) * Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context, 128K output, adaptive thinking only) on the Anthropic API, Bedrock converse (base, global, and us/eu geo inference profiles at the 10% regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which serves Fable 5 with the full 1M context window unlike Opus 4.8). Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the model in the setup wizard, and extends the reasoning effort e2e grid. The Bedrock, Vertex, and Azure grid cells carry fail_reason markers until the CI accounts are provisioned: Bedrock needs the provider data sharing opt-in Fable 5 requires, and the Foundry resource needs a claude-fable-5 deployment. The first-party entry carries provider_specific_entry {us: 1.1} for the inference_geo premium and deliberately no fast multiplier since Fable 5 has no fast mode. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drop removed sampling params for Claude 4.7+ when drop_params is set Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was forwarding them even with drop_params enabled because the Anthropic and Bedrock converse transformations passed temperature/top_p through unconditionally. Mirror the GPT-5/o-series handling: temperature=1 still passes through, other values and any top_p are dropped when drop_params is set, and without drop_params a clean client-side UnsupportedParamsError tells the caller how to opt in, instead of surfacing the raw provider error. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drive sampling param gating from the cost map and cover top_k Greptile review follow-ups on the sampling param fix: the restriction for Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false on every affected cost map entry (perplexity excluded; that route is OpenAI-compatible and maps sampling params upstream) and read back through a tri-state map lookup, keeping the name check only as a fallback for provider-routed ids whose hosted map entries predate the flag, the same layering supports_adaptive_thinking uses. top_k bypasses map_openai_params as a provider-specific kwarg, so it is gated at the shared AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex, Azure) and in the Bedrock converse _handle_top_k_value path, with drop_params threaded through the converse transform helpers. Also updates the reasoning effort grid cell count assertion for the four Fable 5 rows added on this branch (29 x 11 cells). https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Declare supports_sampling_params in the cost map schema The model map validation schema uses additionalProperties: false, so the new flag must be declared for the 28 entries that carry it; this was the one failing job (misc / Run tests) on the previous commit. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix(bedrock): gate top_k=0 on converse to match Anthropic boundary Truthiness check let top_k=0 silently disappear on models that removed sampling params, while AnthropicConfig.transform_request treats 0 as present and raises UnsupportedParamsError (or drops when drop_params is set). Switch to 'is not None' so converse, direct Anthropic, invoke, Vertex, and Azure all behave the same for top_k=0. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> * fix: enable compact-2026-01-12 beta header for vertex_ai provider The vertex_ai block in anthropic_beta_headers_config.json mapped compact-2026-01-12 to null, so update_headers_with_filtered_beta stripped the header before the request reached Vertex while the compact_20260112 context edit stayed in the body, and Vertex rejected the request with HTTP 400. Vertex rawPredict accepts the header, and the bedrock and databricks blocks already forward it. Mirrors #21867, which enabled context-1m-2025-08-07 for vertex_ai the same way. Fixes #27290. --------- Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: tin-berri <tin@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> * fix(proxy): coerce litellm_settings.max_budget env var to float (#30113) * fix(team): reserve team budget raises for proxy admins on /team/update (#30030) The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974) * test(ui): add a data-driven App Router migration E2E smoke Add a growing Playwright smoke for migrated pages: for each segment it deep-links to the path route, asserts the URL and that the dashboard shell rendered, then clicks off to a legacy page and asserts navigation still works. Driven by e2e_tests/fixtures/migratedPages.ts, so adding a page is one line. Runs in two situations against the same proxy: the default mount (npm run e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root). globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage state is valid under a prefix. Seeded with api-reference; append the rest as their migrations merge. * test(ui): support headed slow-motion + watch pauses in the migration smoke Honor SLOWMO in the server-root-path config (the default config already did), and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state. Both are no-ops by default, so CI behavior is unchanged. * test(ui): make the migration smoke a sidebar-click user journey Rework the smoke from deep-linking to a real navigation journey: start at the landing page, click the migrated page in the sidebar (expanding submenus for nested items), assert the path route rendered, reload it (the check a wrong server_root_path breaks), bounce to a legacy page and back, and — once two pages are migrated — navigate directly between two migrated pages. Verifies via URL + shell render, driven by the same fixture list. * test(ui): address review on the migration smoke Escape ROOT and segment before interpolating them into RegExp URL matchers so a future segment containing regex metacharacters can't silently widen the match. Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead of silently re-running the default mount and passing without exercising the prefix. * test(ui): drop unused watch helper and fix stale smoke README * test(ui): run the migration smoke under a server root path in CI * test(ui): harden + instrument the server-root-path proxy reboot in CI * test(ui): run the server-root-path migration smoke as its own CI job Replace the in-place proxy reboot in e2e_ui_testing with a dedicated e2e_ui_testing_server_root_path job that boots the proxy once with SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the config gets its own job rather than killing and relaunching the live proxy. The reboot was failing deterministically: after pkill -9 and relaunch the prefixed proxy never came back up on :4000 (connection refused), so the smoke never ran. The readiness step that was supposed to surface the cause could never reach its boot-log tail because CircleCI runs steps under bash -eo pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's exit 7. Booting the proxy as the job's own background step lets any boot crash land in that step's log instead of being swallowed. The default e2e_ui_testing job is unchanged aside from dropping the reboot, prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at the root mount there via the default Playwright config. * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests * chore(ui): remove dead App Router route stubs under (dashboard) (#30045) models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them * fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000) * fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss * fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation * docs(security): require a reproduction video for vulnerability reports (#30048) (#30063) With AI models capable of automated vulnerability discovery now publicly available, we expect a large increase in report volume, much of it unverified. Requiring a video of the exploit running against a live instance raises the bar for submissions and keeps triage focused on reproducible issues. Reports without a video will be closed and reopened if one is added later. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> * feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. * chore(ui): remove dead dashboard files and unused dependencies (#30047) * chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit 30d2e96f77. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064) * Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context, 128K output, adaptive thinking only) on the Anthropic API, Bedrock converse (base, global, and us/eu geo inference profiles at the 10% regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which serves Fable 5 with the full 1M context window unlike Opus 4.8). Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the model in the setup wizard, and extends the reasoning effort e2e grid. The Bedrock, Vertex, and Azure grid cells carry fail_reason markers until the CI accounts are provisioned: Bedrock needs the provider data sharing opt-in Fable 5 requires, and the Foundry resource needs a claude-fable-5 deployment. The first-party entry carries provider_specific_entry {us: 1.1} for the inference_geo premium and deliberately no fast multiplier since Fable 5 has no fast mode. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drop removed sampling params for Claude 4.7+ when drop_params is set Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was forwarding them even with drop_params enabled because the Anthropic and Bedrock converse transformations passed temperature/top_p through unconditionally. Mirror the GPT-5/o-series handling: temperature=1 still passes through, other values and any top_p are dropped when drop_params is set, and without drop_params a clean client-side UnsupportedParamsError tells the caller how to opt in, instead of surfacing the raw provider error. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drive sampling param gating from the cost map and cover top_k Greptile review follow-ups on the sampling param fix: the restriction for Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false on every affected cost map entry (perplexity excluded; that route is OpenAI-compatible and maps sampling params upstream) and read back through a tri-state map lookup, keeping the name check only as a fallback for provider-routed ids whose hosted map entries predate the flag, the same layering supports_adaptive_thinking uses. top_k bypasses map_openai_params as a provider-specific kwarg, so it is gated at the shared AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex, Azure) and in the Bedrock converse _handle_top_k_value path, with drop_params threaded through the converse transform helpers. Also updates the reasoning effort grid cell count assertion for the four Fable 5 rows added on this branch (29 x 11 cells). https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Declare supports_sampling_params in the cost map schema The model map validation schema uses additionalProperties: false, so the new flag must be declared for the 28 entries that carry it; this was the one failing job (misc / Run tests) on the previous commit. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix(bedrock): gate top_k=0 on converse to match Anthropic boundary Truthiness check let top_k=0 silently disappear on models that removed sampling params, while AnthropicConfig.transform_request treats 0 as present and raises UnsupportedParamsError (or drops when drop_params is set). Switch to 'is not None' so converse, direct Anthropic, invoke, Vertex, and Azure all behave the same for top_k=0. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> * fix(proxy): coerce litellm_settings.max_budget env var to float When max_budget is set in litellm_settings via os.environ/MAX_BUDGET, the env var resolves to a string and the generic setattr branch in ProxyConfig.load_config stored it as-is, so the startup check litellm.max_budget > 0 raised TypeError. The earlier fix (#23855) only covered the CLI initialize() path. Coerce the value to float in the settings loop, matching the existing max_internal_user_budget handling. Fixes #26696. --------- Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: tin-berri <tin@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> * fix(router): don't drop bedrock pass-through deployments using IAM credentials (#30111) * Fix Bedrock passthrough deployment dropped when using IAM credentials Bedrock deployments with use_in_pass_through enabled and IAM/OIDC auth (aws_role_name, no api_key) hit the generic pass-through branch in Router._initialize_deployment_for_pass_through, which calls set_pass_through_credentials and raises "api_key is required". The exception drops the deployment from the router entirely, breaking both passthrough and normal routing for that model. Skip the credential store write when no api_key is set; the bedrock passthrough route resolves AWS credentials at request time via BedrockConverseLLM.get_credentials(), not the passthrough credential store, so there is nothing to register here. Fixes #27728. * Reset passthrough credentials singleton before api_key credential test The test reads the module-level passthrough_endpoint_router singleton, so a stale "openai" entry written by an earlier test in the same process could make the assertion pass without exercising the code path. Clearing the credentials dict up front makes the test order-independent. * fix(sdk): stop mirroring reasoning_content in provider_specific_fields (#30110) The dict-to-response conversion path mirrored reasoning_content into provider_specific_fields, while live provider transforms (Anthropic's _build_provider_specific_fields) only set it top-level on the Message. Cache-replayed messages therefore serialized differently from live ones, breaking disk cache key stability for multi-turn conversations with extended thinking. The mirror was added for DeepSeek before Message.reasoning_content existed as a top-level attribute. The top-level field is still set by the converter, so DeepSeek's request-side promotion is unaffected. Fixes #27337. * fix(mcp): coerce mcp_server_cost_info values to float at ingest (#30109) * fix(mcp): coerce mcp_server_cost_info values to float at ingest YAML 1.1 parses scientific notation without a decimal point (e.g. 7e-05) as a string, and MCPServerCostInfo is a TypedDict with no runtime validation, so a string-typed default_cost_per_query from config.yaml flowed through the proxy untouched and crashed the MCP server settings page with '.toFixed is not a function'. Normalize mcp_server_cost_info on both the config and DB load paths, dropping non-numeric values with a warning instead of failing the server load. Fixes #27097. * fix(mcp): drop non-numeric default_cost_per_query instead of nulling it Keeping the key with a None value still exposes a null to the UI, which can crash .toFixed formatting when the consumer checks key existence rather than truthiness. Delete the key on coercion failure, matching how non-numeric per-tool cost entries are already omitted. * fix(proxy): count embedding and text completion tokens toward TPM limits (#30105) * fix(proxy): count embedding and text completion tokens toward TPM limits The parallel request limiters only read token usage off ModelResponse, so EmbeddingResponse and TextCompletionResponse objects left total_tokens at 0 and the per key, user, team, and end user TPM counters never incremented. Requests to /v1/embeddings and /v1/completions were effectively free against any tpm_limit. In the v3 limiter this was worse: the post-call reconciliation computed actual usage as 0 and refunded the pre-call reservation made at request time. Broaden the isinstance checks to accept EmbeddingResponse and TextCompletionResponse, which both expose a Usage object, at the four per-scope sites in parallel_request_limiter.py and at the usage extraction in parallel_request_limiter_v3.py. ResponsesAPIResponse was already covered in v3 via BaseLiteLLMOpenAIResponseObject. Fixes #27738. * test(proxy): cover v1 limiter TPM counting for embedding and text completion responses Exercise the broadened isinstance sites in parallel_request_limiter.py by asserting that async_log_success_event adds total_tokens to the per key, user, team, and end user TPM counters for EmbeddingResponse and TextCompletionResponse objects. The counters are pre-seeded at zero so the assertion is exactly the increment; on the pre-fix code these responses left total_tokens at 0 and the test fails. * fix(openai): forward client headers on the text completion path (#30103) * fix(openai): forward client headers on the text completion path litellm.completion() merges caller headers with extra_headers, but the text-completion-openai branch never passed the merged dict to openai_text_completions.completion(), and the handler only used its headers argument for logging. Pass the merged headers through the call site and set them as extra_headers on the outgoing request, mirroring the chat completion handler, so x-* client headers forwarded by the proxy reach the provider on /v1/completions. Fixes #27410. * Drop redundant extra_headers assignment and fix test module collision completion() merges extra_headers into headers before the text-completion-openai branch, and the handler now sets the merged headers as extra_headers on the request, so the branch-local optional_params["extra_headers"] assignment was a dead duplicate. Removing it keeps the assignment in one place while both entry paths (litellm.text_completion and direct handler callers) still forward headers; a new regression test pins the extra_headers kwarg path. Also rename the test module to test_completion_handler.py since its basename collided with tests/test_litellm/llms/bedrock/batches/ test_handler.py and broke pytest collection. * fix(bedrock): route Anthropic-shape count_tokens to InvokeModel and base64-encode the body (#30102) * fix(bedrock): route Anthropic-shape count_tokens to InvokeModel POST /v1/messages/count_tokens with Anthropic content blocks ({"type": "text"|"tool_use"|...}) was routed to the Converse input of the Bedrock CountTokens API. The Converse transform copies list content through verbatim, so Bedrock rejected the request with a 400 and the caller silently fell back to the local tokenizer, returning counts that can be off by ~50% on tool-heavy payloads. _detect_input_type now routes messages whose content blocks carry a "type" key (Anthropic shape) to the invokeModel input, which forwards the body verbatim. The invokeModel body is now base64-encoded as the CountTokens API requires (InvokeModelTokensRequest.body is a base64-encoded blob), and Anthropic Messages bodies get the anthropic_version and max_tokens fields Bedrock validates against. Fixes #27632. * refactor(bedrock): name the CountTokens max_tokens placeholder Replace the magic 1024 with a module-level DEFAULT_ANTHROPIC_INVOKE_MODEL_MAX_TOKENS constant so the intent is explicit and there is a single place to update if Bedrock's InvokeModel schema ever changes. Module-local rather than litellm/constants.py because the value is only a schema-validation placeholder for token counting, not a user-tunable generation default. * Add above-512k pricing tier for MiniMax-M3 and correct its base rates (#30095) * Add above-512k pricing tier support for MiniMax-M3 MiniMax-M3 doubles its per-token rates once a prompt exceeds 512k input tokens. The tiered cost parser already handles arbitrary thresholds, but get_model_info only copies whitelisted keys from ModelInfoBase, which had no 512k variants, so above_512k keys were silently dropped and long-context requests were priced at the flat rate. Add the input, output, and cache-read above_512k_tokens fields to ModelInfoBase and pass them through in get_model_info. Update the minimax/MiniMax-M3 entry with the tiered rates and correct the base rates, which matched the above-512k tier instead of the published base tier (https://platform.minimax.io/docs/guides/pricing-paygo). Fixes #29663. * Add above-512k keys to pricing schema, set MiniMax-M3 context to 1M Register the three new above_512k_tokens cost keys in the INTENDED_SCHEMA of test_aaamodel_prices_and_context_window_json_is_valid, declared the same way as the existing above_200k/above_272k tier keys, so the schema check accepts the MiniMax-M3 tiered pricing entry. Also raise MiniMax-M3 max_input_tokens from 512000 to 1000000 in both pricing JSONs. The MiniMax API docs (https://platform.minimax.io/docs/guides/text-generation) state the model supports a 1,000,000-token context window, and the pay-as-you-go pricing page (https://platform.minimax.io/docs/guides/pricing-paygo) prices input above 512k tokens, which only makes sense if inputs beyond 512k are accepted. This makes the above-512k pricing tier reachable. * fix(bedrock): make document names unique across conversation turns (#30093) * fix(bedrock): make document names unique across conversation turns PR #16275 derived Bedrock document names purely from a content hash so that names stay deterministic for prompt caching. When the same PDF or document appears in more than one conversation turn, every occurrence gets the identical name and Bedrock rejects the request with "Messages can not contain duplicate document names". Add _rename_duplicate_bedrock_document_names, a post-pass over the assembled message blocks that keeps the first occurrence's hash-based name and appends a positional suffix (_2, _3, ...) to later occurrences. Apply it in both _bedrock_converse_messages_pt and _bedrock_converse_messages_pt_async. Names remain deterministic across requests and the first occurrence is unchanged, so prompt cache prefixes stay stable. Fixes #29418. * fix(bedrock): avoid suffix collisions with organic document names A renamed duplicate could collide with a document whose hash-derived name already ends in the same positional suffix (e.g. an organic report_2 next to two documents named report). Collect every document name up front and bump the suffix until the candidate is unused, so renames can collide neither with organic names nor with each other. * fix(_types): remove ResponsesAPIResponse from PassThroughEndpointLoggingResultValues The import of ResponsesAPIResponse was removed from the file but a usage was left in the Union type, causing a NameError on import and breaking all CI tests. Remove the stale reference to match the cleanup intent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(_types): restore ResponsesAPIResponse import and add use_xai_oauth to filter list Two related fixes: 1. Re-add ResponsesAPIResponse import in _types.py — it was removed but still needed in PassThroughEndpointLoggingResultValues (used in openai_passthrough_logging_handler.py). 2. Add use_xai_oauth to all_litellm_params so it is filtered before forwarding kwargs to providers like OpenAI that do not recognize it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Hari <kancharla.ha@northeastern.edu> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Ceder Dens <ceder.dens@uantwerpen.be> Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: 冯基魁 <56265583+fengjikui@users.noreply.github.com> Co-authored-by: victoruce <161634297+victoruce@users.noreply.github.com> Co-authored-by: kejunleng <33445544+silencedoctor@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Tyson Cung <45380903+tysoncung@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: Daan <255322319+daanhendrio@users.noreply.github.com> Co-authored-by: Avani Prajapati <143805019+Avani-prajapati@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: Dimitris Spachos <dspachos@gmail.com> Co-authored-by: Liam Scott <liam@uilliam.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: tin-berri <tin@berri.ai> Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>	2026-06-10 10:34:07 -07:00
michelligabriele	2fe9feda71	fix(caching): restore stored prompt_tokens on embedding cache hits instead of recomputing (#30046)	2026-06-10 15:49:20 +05:30
Mateo Wang	e15b37a18e	Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064) ... * Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context, 128K output, adaptive thinking only) on the Anthropic API, Bedrock converse (base, global, and us/eu geo inference profiles at the 10% regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which serves Fable 5 with the full 1M context window unlike Opus 4.8). Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the model in the setup wizard, and extends the reasoning effort e2e grid. The Bedrock, Vertex, and Azure grid cells carry fail_reason markers until the CI accounts are provisioned: Bedrock needs the provider data sharing opt-in Fable 5 requires, and the Foundry resource needs a claude-fable-5 deployment. The first-party entry carries provider_specific_entry {us: 1.1} for the inference_geo premium and deliberately no fast multiplier since Fable 5 has no fast mode. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drop removed sampling params for Claude 4.7+ when drop_params is set Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was forwarding them even with drop_params enabled because the Anthropic and Bedrock converse transformations passed temperature/top_p through unconditionally. Mirror the GPT-5/o-series handling: temperature=1 still passes through, other values and any top_p are dropped when drop_params is set, and without drop_params a clean client-side UnsupportedParamsError tells the caller how to opt in, instead of surfacing the raw provider error. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Drive sampling param gating from the cost map and cover top_k Greptile review follow-ups on the sampling param fix: the restriction for Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false on every affected cost map entry (perplexity excluded; that route is OpenAI-compatible and maps sampling params upstream) and read back through a tri-state map lookup, keeping the name check only as a fallback for provider-routed ids whose hosted map entries predate the flag, the same layering supports_adaptive_thinking uses. top_k bypasses map_openai_params as a provider-specific kwarg, so it is gated at the shared AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex, Azure) and in the Bedrock converse _handle_top_k_value path, with drop_params threaded through the converse transform helpers. Also updates the reasoning effort grid cell count assertion for the four Fable 5 rows added on this branch (29 x 11 cells). https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Declare supports_sampling_params in the cost map schema The model map validation schema uses additionalProperties: false, so the new flag must be declared for the 28 entries that carry it; this was the one failing job (misc / Run tests) on the previous commit. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix(bedrock): gate top_k=0 on converse to match Anthropic boundary Truthiness check let top_k=0 silently disappear on models that removed sampling params, while AnthropicConfig.transform_request treats 0 as present and raises UnsupportedParamsError (or drops when drop_params is set). Switch to 'is not None' so converse, direct Anthropic, invoke, Vertex, and Azure all behave the same for top_k=0. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-10 08:50:15 +05:30
Sameer Kankute	2cd7e87485	fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) ... * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit 30d2e96f77. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-10 08:22:15 +05:30
ryan-crabbe-berri	248176112e	feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) ... * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves.	2026-06-09 17:45:42 -07:00
shivam	5e2df556d8	fix(cost): store cost breakdown for /v1/realtime sessions ... Realtime cost calculation computed totals but never populated logging_obj.cost_breakdown, so spend logs and the UI Metrics/Cost Breakdown showed no input/output cost details. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-09 17:10:15 -07:00
tin-berri	5b7063d194	fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) ... * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation	2026-06-09 14:19:11 -07:00
michelligabriele	fe60f9d0f1	fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) ... * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests	2026-06-09 22:10:23 +02:00
milan-berri	d84499e0f2	fix(team): reserve team budget raises for proxy admins on /team/update (#30030) ... The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-09 09:19:15 -07:00
tin-berri	51ba6e39cd	fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path (#29960) ... * fix(ui): load MCP tool configuration tools via the OBO/passthrough-aware GET path * fix(mcp): admin-only include_disabled_tools so the settings UI shows toggled-off tools * fix(ui): repopulate MCP server edit form when server data loads after mount (OAuth return) * fix(ui): persist MCP OAuth token on save and return to the Settings tab after authorize * fix(ui): scope MCP OAuth callback to the initiating form so create and edit flows don't cross-talk * fix(ui): derive OAuth-return Settings tab via lazy state init instead of setState-in-effect * Fix MCP OAuth edit token handling --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-08 19:58:51 -07:00
Sameer Kankute	424db6a980	feat(azure_ai): add MAI-Image-2.5 image generation support (#29688) ... * feat(azure_ai): add MAI-Image-2.5 image generation support Route azure_ai MAI models to /mai/v1/images/generations and map OpenAI size to width/height for the serverless API. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(azure_ai): address MAI image generation review feedback Validate unsupported size values, default width/height independently, add MAI-Image-2.5 pricing, and expand test coverage. @greptileai Co-authored-by: Cursor <cursoragent@cursor.com> * feat(azure_ai): add MAI image edit and expand model cost map Add MAI image edit support with usage normalization for Azure response format, and register MAI-Image-2.5-Flash and MAI-Image-2e pricing in the model map. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(azure_ai): validate MAI edit size by consuming map iterator Greptile: lazy map() never evaluated int() so values like 1024xabc passed through. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(azure_ai): normalize MAI usage in generation response handler Apply normalize_mai_image_usage before building ImageResponse so token-based cost calculation works when Azure returns num_output_tokens fields. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(azure_ai): narrow MAI edit size param type for mypy Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Azure MAI image response handling * Fix MAI image generation base model routing * fix(azure_ai): preserve zero num_output_tokens in MAI usage normalization * fix(azure_ai): wrap MAI generation response JSON parsing in error handling * fix(azure_ai): build MAI image edit URL correctly for /mai/ root bases * fix(azure_ai): build MAI image generation URL correctly for /mai/ root bases --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-08 18:27:04 -07:00
tin-berri	92817cb65b	changing expires_in default to use actual slack return details (#29951)	2026-06-08 18:13:06 -07:00
yuneng-jiang	1bbaf1c39d	fix(guardrails): read CrowdStrike AIDR identity from both metadata bags (#29991) ... Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first	2026-06-08 17:46:28 -07:00
milan-berri	411bd3da5b	feat(vantage): include organization metadata in FOCUS Tags export (#28184) ... * feat(vantage): include organization metadata in FOCUS Tags export Join LiteLLM_OrganizationTable when building Vantage/FOCUS export rows so organization_id and organization_alias appear in Tags for org-level filtering. Co-authored-by: Cursor <cursoragent@cursor.com> * test(focus): include api_requests in organization Tags tests FocusTransformer now requires api_requests after staging merge; add the column to test fixtures so integrations CI can run the Tags assertions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-09 02:59:21 +03:00
yuneng-jiang	c24a3603d9	fix(team-management): delete a team's BYOK models when the team is deleted (#29977) ... A team's BYOK models (rows in LiteLLM_ProxyModelTable with model_info.team_id set) were left orphaned when the team was deleted; they lingered in the database and kept showing on the Models + Endpoints page. delete_team now removes them via a new delete_team_models helper that deletes the rows in one transaction and syncs the in-memory router only after that transaction commits, run before the team rows are deleted so a mid-flight failure never leaves the team gone with its models orphaned	2026-06-08 16:55:35 -07:00
Sameer Kankute	dfd6cbc514	fix(vertex): propagate Vertex AI metadata in streaming success callbacks (#29899) ... * fix(vertex): propagate Vertex AI metadata in streaming success callbacks Streaming calls assembled via stream_chunk_builder were missing vertex_ai_grounding_metadata and vertex_ai_url_context_metadata in standard_logging_object.response. Merge metadata from chunks into the assembled response and mirror non-streaming hidden_params on Gemini chunks. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(vertex): move streaming metadata merge into provider config hook Address review feedback by delegating assembled-stream metadata propagation to VertexGeminiConfig via BaseConfig.apply_assembled_streaming_response_metadata, and only write chunk hidden_params when metadata is non-empty. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(redaction): scrub Vertex provider metadata when message logging is off Clear vertex_ai_grounding_metadata and related fields from standard logging responses and assembled streaming ModelResponse objects so turn_off_message_logging cannot leak prompt-derived web search queries. Co-authored-by: Cursor <cursoragent@cursor.com> * Use assembled model for streaming metadata hook * Fix Vertex metadata redaction bypass in logging callbacks. Scrub Vertex provider fields from litellm_params.metadata.hidden_params during perform_redaction so streaming success_handler merges do not leak prompt-derived metadata when message logging is disabled. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Vertex streaming metadata from hidden params * fix(vertex): mirror vertex_ai_safety_results on assembled streaming responses The non-streaming transform_response stores safety data under vertex_ai_safety_results, but the streaming path only wrote vertex_ai_safety_ratings. Assembled streaming responses therefore never carried vertex_ai_safety_results, so any consumer reading that field saw a silent difference between streaming and non-streaming calls. Set vertex_ai_safety_results alongside vertex_ai_safety_ratings in the shared stream metadata setter and add it to the assembled metadata field list so it propagates through stream_chunk_builder. * fix(streaming): log provider streaming metadata hook failures instead of swallowing them * refactor(vertex): share single Vertex metadata field tuple across redaction and streaming * refactor(vertex): move Vertex metadata redaction helpers into llms/vertex_ai --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-08 16:14:30 -07:00
milan-berri	1c881eee5d	fix(fireworks): enable tool calling for glm-5p1 in model cost map (#29697) ... glm-5p1 supports native tools on Fireworks; explicit false flags caused drop_params to strip tools and tool_choice before the provider request. Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-08 15:54:19 -07:00