1745 commits

Author	SHA1	Message	Date
yuneng-jiang	b3086ccd74	chore(release): backport 11 staging PRs onto patch-1.92.0rc2 for the 1.92.0 stable cut (#32959) ... * fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389) * fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) * test(register_model): use a triple provider prefix as the unresolvable-key fixture get_model_info now resolves bedrock/bedrock/... like a routing prefix, so the double-prefix fixture stopped exercising the register_model fallback path. Lock the new double-prefix resolution in as a model-info regression test (cherry picked from commit 734fd29e00) * fix(guardrails): walk Responses-API text taxonomy in shared content helpers (#32542) * fix(guardrails): walk Responses-API text taxonomy in shared content helpers Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently drops all text on the /v1/responses path. AIM turns it into a loud 422 ( {"error":"No messages in the request"}); every other guardrail (Lakera v2, Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret detection) scans an empty payload and lets the request through unscanned. Three defects, all in _content_utils.py: 1. _iter_text_parts_in_content recognised only part.type == "text", but the Responses API uses input_text (request) and output_text (assistant). 2. _coerce_input_to_messages gated on "every item has a role key"; any Responses input list containing a function_call or function_call_output item failed the check and was wrapped as one opaque blob. 3. build_inspection_messages forwarded any role through, including a bare tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze reject with a schema error. Fix walks the actual Responses item taxonomy (message, function_call, function_call_output, bare content parts and strings), recognises {text, input_text, output_text} everywhere, and coerces any role outside {system, user, assistant} to user in the outbound inspection payload. * style: ruff-format changed guardrail files * test(guardrails): cover function_call_output string form; drop em-dash in new docstring * fix(guardrails): map function_call_output straight to user role Avoids ever materialising a schema-invalid bare tool message. The downstream role-safety coercion in build_inspection_messages still guards genuinely caller-supplied non-standard roles (developer, function, custom values); add a regression test covering that path so the coercion has real coverage after this simplification. * test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages * docs(test): soften AIM-specific claims in LIT-4294 test docstrings Ryan's review flagged that several test docstrings assert AIM's /fw/v1/analyze validates + rejects specific schema violations. That behavior is customer-reported in the LIT-4294 writeup, not directly verified by us. Rephrase to attribute the AIM 422 to the customer's writeup and describe the underlying constraint as the OpenAI chat schema; any downstream API that validates against that schema rejects the same shape. * refactor(guardrails): move unsupported-role coercion into AIM only The generic coercion in build_inspection_messages collapsed any role outside {system, user, assistant} to user for every caller of the helper. Combined with the pre-existing apply_redacted_messages_back write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400 on chat-completions tool-message masking into a silent semantic corruption of the outbound request (role tool with tool_call_id got rewritten to bare role user, dropping the assistant + tool_calls sibling). AIM specifically requires the coercion because its /fw/v1/analyze validates the payload against the OpenAI chat schema; other guardrails either do not validate roles or do their own reconstruction. Move the coercion to AimGuardrail._build_aim_inspection_messages so the shared helper keeps caller roles intact and no new cross-guardrail role corruption is introduced. The pre-existing apply_redacted_messages_back structural flatten remains as separate follow-up work. function_call_output items still synthesise role user in the shared helper because they have no natural role field, which is a different concern from coercing a caller-supplied role. * refactor(guardrails): preserve role fidelity in shared _content_utils Shared inspection helpers should extract text and preserve semantic role signals; role coercion for third-party schema safety stays inside the guardrail that needs it (AIM). Three shared-helper changes: - Bare content-part dicts (input_text/output_text) with an explicit role keep it; only role-less parts default to user. - Responses message items already had their role preserved; the behavior is now covered by an explicit test. - function_call_output items default to role tool (semantic equivalent of the chat-completions tool message shape) instead of role user, so Responses and chat completions produce symmetric inspection payloads. A caller-supplied role on the item is still preserved. AIM's schema-safe coercion in _build_aim_inspection_messages already handles the resulting role tool: it collapses to user before the POST to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the bare tool message (no tool_call_id can survive the flatten). Added a regression test in test_aim.py covering that path. (cherry picked from commit e84a19acd5) * feat: add Meta Model API provider and muse-spark-1.1 (day-0) (#32701) (cherry picked from commit d82645d163) * fix(bedrock): keep mid-conversation system messages in place for Claude Invoke (#32578) Hoisting every role system entry into the top-level system field mutates the cache prefix whenever a client such as Claude Code appends a new mid-conversation system message, invalidating the prompt cache for the entire message history on Bedrock Invoke. Bedrock only rejects a system entry at messages.0, so hoist just the leading run and forward the rest in place (cherry picked from commit cc36d5469c) * feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls (#32655) * feat(otel): emit the gen_ai.client.operation.exception event on failed LLM calls The GenAI semantic conventions record failures of a GenAI client operation as a log-based event named gen_ai.client.operation.exception, carrying the exception.type / exception.message / exception.stacktrace trio at severity WARN and correlated to the failed span. OTel v2 never emitted it: a failed LLM call produced only the deprecated error.* span attributes, a generic exception span event without a stacktrace, and the stacktrace under the vendor key litellm.provider.error.stack_trace. Build the logs pipeline (LoggerProvider + console/OTLP log exporters mirroring the metrics plumbing) and record the event behind the enable_events flag, which until now was defined but consumed nowhere. An operator-configured LoggerProvider global is reused so the events ride their existing logs pipeline; an explicit NoOpLoggerProvider global is honored as an opt-out and builds no recorder at all. The existing span-side error surface (error.type, error.message, the exception span event, and the litellm.provider.error.* detail keys) is untouched for backwards compatibility. * fix(otel): always ride the semconv-required exception pair on the GenAI event Filtering the event attributes on truthiness conflated "absent" with "empty", so an empty exception.type or exception.message would have been dropped, leaving an event with neither semconv-required field. Build the attributes so the pair is unconditional and only the recommended stacktrace is omitted when the payload carries none. * docs(otel): document the events plumbing module in the package README * test(otel): cover the log exporter selection and logs endpoint normalization The new logs plumbing had no coverage for exporter-kind selection, the console fallback for an unrecognized kind, the /v1/logs signal-path rewriting that lets one OTEL_ENDPOINT serve every signal, or the simple-vs-batch processor split. (cherry picked from commit 99b4c5ed3e) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831) * fix(bedrock): gate in-place system role messages on model support for Claude Invoke * feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule (cherry picked from commit 5e23a5ab05) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867) * fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface (thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5) get the effort translated to a legacy thinking budget_tokens. Models with no reasoning support have thinking/effort dropped under drop_params. And because adaptive thinking carries no budget while the legacy form must satisfy Anthropic's max_tokens > budget_tokens rule, the translated budget is capped below max_tokens, dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass through untouched. This matters because clients like Claude Code speak native Anthropic /v1/messages and send the adaptive interface unconditionally, regardless of the routed model. The native passthrough previously only capability-gated the OpenAI-style reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so a pre-4.6 model rejected it with "This model does not support the effort parameter" and the request failed. Claude Code already gets drop_params auto-set, so its requests now succeed. * test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests Addresses review feedback on the max_tokens-too-small branch. Previously a thinking-capable model whose max_tokens could not fit the minimum thinking budget had thinking silently dropped regardless of drop_params, while a residual output_config field in the same call still raised when drop_params was off. Gate both consistently on drop_params: raise a clear error (naming max_tokens for the undersized case) when drop_params is off, drop otherwise. Claude Code gets drop_params auto-set, so it still succeeds. Adds tests for the undersized-max_tokens raise, the residual output_config raise, and the no-adaptive-interface passthrough on a non-adaptive model. * fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts The previous raise-when-not-drop_params behavior broke existing bedrock and vertex messages tests: those providers already silently strip unsupported output_config for pre-4.6 models (issue #22797) with no drop_params required, and the shared parent transform raising pre-empted that. It also conflicted with the goal of keeping requests working rather than failing them. Make the reshape silent: translate effort to legacy thinking for thinking-capable models, drop thinking for non-reasoning models, and remove only the consumed effort key from output_config, leaving any residual (e.g. format) for provider subclasses (bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the review note about inconsistent drop_params handling by making every path uniform. Updates the tests to assert the silent behavior and residual output_config preservation. * fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5) Greptile caught a real bug: the early-return guard treated supports_output_config as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises supports_output_config (it accepts output_config.effort) but is not adaptive, so it rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking block raw, reproducing the exact failure the fix is meant to prevent. thinking:{type:adaptive} and output_config.effort are independent capabilities. Only early-return for adaptive-thinking models. For a model that supports output_config.effort but is not adaptive, keep the native effort and drop only the unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code payload now returns 200 instead of 400. Adds regression tests for Opus 4.5 with and without adaptive thinking. * fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models Claude Opus 4.5 advertises supports_output_config but not adaptive thinking, so the early-return guard forwarded thinking.type=adaptive raw and Anthropic rejected it. The guard now only skips true adaptive models; effort-only requests on effort-capable models still pass through untouched. The _map_reasoning_effort call is wrapped to surface unrecognized effort values as a clean 400, matching _translate_reasoning_effort_to_anthropic * fix(anthropic): fall back to legacy thinking when effort level unsupported Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code defaults to xhigh on newer models, so preserving that level raw gets rejected by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model and fall through to the budget translation for unsupported levels * fix(anthropic): keep effort-only requests untouched for provider normalization The xhigh fall-through consumed effort-only requests on effort-capable models, breaking bedrock invoke's own normalization which clamps xhigh to the model's ceiling after the base transform runs (test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict the fall-through to requests that carry adaptive thinking; effort-only requests pass through so provider subclasses keep owning level clamping --------- Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> (cherry picked from commit 3a62e5428f) * fix(bedrock): flag mapped Claude 4.8+ entries with supports_mid_conversation_system (#32882) Exact cost-map hits resolve before fallback-generalization rules, so the mapped Sonnet 5, Fable 5 and jp Opus 4.8 Bedrock entries bypassed the bedrock-anthropic-claude-mid-conversation-system rule and hoisted mid-conversation system messages, invalidating the prompt cache. (cherry picked from commit c15891fc98) * Merge pull request #32873 from BerriAI/litellm_fallback_rules_routing_split refactor(fallback-generalizations): split rules into routing and provider-neutral capability kinds (cherry picked from commit 45d3644408) * Merge pull request #32874 from BerriAI/litellm_thread_provider_capability_probes fix(anthropic): thread real provider through capability probes instead of pinning anthropic (cherry picked from commit ead7ad3804) * test: add /v1/messages to supported_endpoints schema enum (#32739) (cherry picked from commit bf02a4a47f) --------- Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com> Co-authored-by: tin-berri <tin@berri.ai>	2026-07-11 16:29:55 -07:00
yucheng-berri	07b9ea8c3b	fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool (#32093) ... * fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool The advisor_20260301 interceptor honored a caller-supplied api_base once allow_client_side_credentials was enabled, even without a caller-supplied api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials plus the conversation history got sent to a caller-chosen destination _resolve_advisor_credentials() now only honors api_base alongside a non-empty caller-supplied api_key, requires the https scheme, and validates api_base via validate_url() before use, mirroring check_complete_credentials in auth_utils.py. https is required because validate_url only DNS-pins the connection for http; for https with TLS verification on it returns the URL unchanged and relies on certificate validation to block DNS rebinding * fix(anthropic): also reject advisor api_base when ssl_verify is disabled validate_url only DNS-pins the connection for http, or for https with litellm.ssl_verify disabled; the previous https-only check missed the ssl_verify=False case, where validate_url's rewritten URL was still being discarded, per Greptile's review of this PR. Reject api_base outright when ssl_verify is False so the discarded rewrite can no longer matter	2026-07-04 12:06:09 -07:00
Mateo Wang	2e1d8d2928	fix(anthropic): keep context_management working when drop_params is enabled (#32020) ... * fix(anthropic): keep context_management working when drop_params is enabled drop_params (proxy-wide or per-request) silently disabled the in-gateway context_management polyfill on the /v1/messages -> chat completions adapter path, even though context_management is a LiteLLM-supported param (native on Anthropic, polyfilled elsewhere). Gate the polyfill on an explicit additional_drop_params: ["context_management"] opt-out instead, which also makes that escape hatch actually work on the adapter path. * test(anthropic): cover sync adapter polyfill gate for global drop_params and additional_drop_params	2026-07-03 14:12:32 -07:00
tin-berri	d0e785140c	feat(tinyfish): make search provider permissive, attribute errors (#31997) ... * feat(tinyfish): make search provider permissive, attribute errors Reshapes the TinyFish search provider so LiteLLM mirrors the TinyFish Search API surface instead of maintaining a parallel cherry-pick. Request side: - Drop misleading request TypedDict - Stop sending max_results on wire (TinyFish ignores it); clamp to [1,10] client-side via self-threaded state - Guard non-numeric max_results from bare ValueError - Auto-JSON-encode dict params; lowercase bool serialization for ux-labs Response side: - Drop both Pydantic response models; parse directly into SearchResponse so per-result extras flow through via extra="allow" - Default missing title/url/snippet to "" instead of failing the call - Read top-level parameter_warnings and re-fire as verbose_logger.warning (pre-wired for upcoming TinyFish-side rollout; no-op today) Error handling: - Attributed _wrap_error helper at 3 call sites in transform_search_response ("TinyFish Search: <msg>. See https://docs.tinyfish.ai/search-api for details.") - Dispatch non-2xx responses through _wrap_error (fixes pre-existing bug where 4xx/5xx silently returned empty SearchResponse) - Wrap json.JSONDecodeError on 200 bodies - Wrap pydantic.ValidationError for envelope-shape mismatches Bug fix worth flagging: 4xx/5xx responses now raise an attributed BaseLLMException instead of silently returning SearchResponse(results=[]). Follow-up to #30634. * fix(tinyfish): apply ruff format; guard OverflowError in max_results clamp - Run ruff format on the touched files (CI lint job rejected the prior commit's formatting). - Add OverflowError to the except clause in the max_results clamp so callers passing math.inf (or other non-finite floats) get the same warn-and-ignore behavior as other malformed values. Greptile spotted this in the first-pass review. - Add test_max_results_infinity_float_warns_and_skips covering the inf case. * fix(tinyfish): apply --line-length 88 ruff format to match CI CI uses 'ruff format --check --line-length 88'; my prior format pass used the default line length, leaving several lines unwrapped. No behavior change — purely whitespace. * fix(tinyfish): reduce transform_search_response complexity; sort imports CI's ruff strict-rule budget rejected the prior commit with: - C901: transform_search_response complexity 16 > 10 (cap exceeded by 1) - I001: import sort violation (cap exceeded by 1) Extract two module-level helpers from transform_search_response to drop its cyclomatic complexity: - _default_missing_result_fields: in-place title/url/snippet defaulting - _emit_parameter_warnings: defensive parameter_warnings reader Auto-fix the import sort via ruff --fix. No behavior change; the 59 existing tests still pass. * test(tinyfish): cover defensive branches in _default_missing_result_fields Codecov flagged 97.61% patch coverage (2 lines missing). The uncovered lines were the non-dict raw_json and non-dict per-result item early-exits in _default_missing_result_fields. Add two unit tests on the helper directly to bring patch coverage to 100%. * chore(tinyfish): apply ruff format to fix lint after staging merge --------- Co-authored-by: Chenlu Ji <jichenlulu@gmail.com>	2026-07-03 10:17:11 -07:00
Mateo Wang	ae356cf1fa	fix(azure_ai): preserve content, tables, and keyValuePairs in doc-intelligence /v1/ocr (#32018) ... * fix(azure_ai): preserve content, tables, and keyValuePairs in doc-intelligence /v1/ocr Azure Document Intelligence analyzeResult.content, .tables, and .keyValuePairs were dropped when normalizing to the Mistral OCR schema. They are now passed through verbatim as top-level response fields, and the duplicated sync/async response parsing is consolidated into one pydantic-validated helper. Also adds the Azure DI features query param (list[str] or comma-separated string, e.g. features=keyValuePairs) which Azure requires for keyValuePairs extraction. * test(azure_ai): replace fastapi jsonable_encoder with model_dump in ocr unit tests	2026-07-02 22:38:36 -07:00
michelligabriele	cbe23d6184	fix(a2a): populate response usage in a2a chat transformation (#31980) ... * fix(a2a): populate response usage in a2a chat transformation * test(a2a): rename to avoid module basename collision with fastcrw's test_transformation.py	2026-07-03 09:28:36 +05:30
Mateo Wang	138a69bd61	fix(bedrock): map guardrailConfig to InvokeModel guardrail headers (#31985) ... * fix(bedrock): map guardrailConfig to InvokeModel guardrail headers The InvokeModel API takes the guardrail identifier, version and trace as X-Amzn-Bedrock-* request headers, unlike Converse which takes them in the request body. The invoke transformer never set these headers, so guardrailConfig was silently dropped (or leaked into the request body) and Bedrock guardrails never ran on invoke-route models. Pop guardrailConfig in AmazonInvokeConfig.validate_environment, validate it, and set the headers before SigV4 signing; explicitly passed headers keep winning over guardrailConfig so existing workarounds are unaffected * fix(bedrock): reject guardrailConfig missing guardrailIdentifier A guardrailConfig without guardrailIdentifier (e.g. an empty dict) would validate, produce no guardrail headers, and let the request proceed with guardrails silently not applied; that silent skip is the exact failure mode this fix exists to remove, so fail fast with a 400 instead	2026-07-02 18:58:30 -07:00
Mateo Wang	8bb4e62412	feat(tencent): add Tencent TokenHub as a provider (#31903) ... * feat(tencent): add Tencent TokenHub as a provider Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a new provider: TencentChatConfig routes /v1/chat/completions and gates the thinking/reasoning_effort params behind supports_reasoning, and TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API. Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider endpoint support metadata. * test(tencent): add unit tests for Tencent TokenHub provider Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig (messages API) across transformation, param mapping, URL building, and header validation, plus get_optional_params routing. Tests mock supports_reasoning to stay independent of remote model cost data. * fix(tencent): correct max_output_tokens and reuse parent messages env validation Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling * fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix * fix(tencent): register tencent in models_by_provider The provider was added to the LlmProviders enum and cost map but not to the models_by_provider lookup, so test_models_by_provider (which asserts every litellm_provider present in the cost map is registered) failed once the tencent models were loaded. Add the tencent_models set, populate it from the cost map, and expose it under the tencent key, mirroring deepseek. * fix(tencent): import generic_cost_per_token from its canonical module Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils instead of the top-level litellm.cost_calculator dispatcher, which imports the tencent cost module at load time. Removing the back-reference avoids the circular import and matches how deepseek and the other providers source the helper. --------- Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-07-02 18:31:59 -07:00
Shivam Rawat	1543725916	fix(bedrock): honor ttl for tool_config cache injection points (#31929) ... * fix(bedrock): honor ttl for tool_config cache injection points Pass cache_control_injection_points control.ttl through to Bedrock toolConfig cachePoint blocks, matching message/system cache behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a manual update for every new Claude release (it already silently missed Sonnet 5 and Fable 5). Replace it with a lookup against cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json, which AWS docs confirm tracks the same 1h-TTL-capable model set. Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried that pricing field (their own regional variants didn't have it), which would have made the JSON-driven check wrongly grant them 1h TTL support. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id (...-20250514-v1:0) that never shipped. This passed under the old regex-based is_claude_4_5_on_bedrock, which matched on substring alone, but fails now that it looks up cache_creation_input_token_cost_above_1hr in litellm.model_cost, since the fake id has no pricing entry. Also force the bundled local cost map in both tests so ttl eligibility reads this branch's pricing data instead of the network-fetched main copy, which lacks the fix until merge. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): restore cache and tool config compatibility * fix(bedrock): preserve Sonnet 5 parallel tool config * fix(bedrock): decouple parallel tool support from cache ttl * refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and bedrock_converse_supports_strict_tool_schemas (dead code) with a supports_parallel_tool_use_config key in model_prices_and_context_window.json, matching how is_claude_4_5_on_bedrock already reads cache_creation_input_token_cost_above_1hr from the pricing JSON. New models pick up parallel tool use support automatically when their pricing entry ships with the key set, with no code change required Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in model_prices_and_context_window.json, so bedrock_converse_supports_parallel_tool_use_config returned False and the test died with KeyError on additionalModelRequestFields. Use jp.anthropic.claude-opus-4-7, a real entry that carries supports_parallel_tool_use_config without 1h-TTL cache pricing, which is exactly the decoupling this test exists to cover * test(utils): allow supports_parallel_tool_use_config in pricing schema The misc unit test job validates model_prices_and_context_window.json against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects unknown keys. Add the supports_parallel_tool_use_config key this PR introduced so test_aaamodel_prices_and_context_window_json_is_valid passes again * fix(bedrock): preserve ttl for regional claude models * fix(bedrock): fall back to base model entry when regional pricing lacks capability fields Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit cache_creation_input_token_cost_above_1hr shadowed the base entry that has it, so is_claude_4_5_on_bedrock returned False and requested cache ttl values were dropped for those deployments. Both capability lookups now consult the full model id and the region-stripped base entry, matching the coverage of the old name-pattern list. Also restores ToolBlock keyword construction for the tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every supported Python version --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-07-02 16:30:06 -07:00
Sameer Kankute	a16d9c6f9e	test(e2e): add live batches suite across providers and routing scenarios (#30958) ... * tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * fix: p0 issues, added types and shared functions for each test suite * style: carry clearer status_code comparison into renamed e2e dir * refactor: migrate to gateway client * fix: add new tests, split gateway * test(e2e): add live batches suite across providers and routing scenarios * test(batches): cover real cost tracking on completed batch retrieve * test(e2e): assert managed vs raw file and batch id shapes per routing scenario * test(e2e): assert full response shape of each batches and files endpoint * test(e2e): only accept transitional statuses for a freshly created batch * test(prompt-factory): make test_convert_url deterministic with a data URL picsum.photos is down (HTTP 522), so test_convert_url failed on every run. Swap the live external image for an inline data: URL and assert the round-trip through convert_url_to_base64 genuinely. A data URL is already inline base64 image data, so convert_url_to_base64 now short-circuits it instead of attempting an impossible HTTP fetch; add a regression for that branch in the mapped image_handling test * fix: pass through async image data urls * fix(image-handling): short-circuit data URLs in async path too Bugbot flagged that convert_url_to_base64 returns data: base64 URLs unchanged but async_convert_url_to_base64 still tried to fetch them, so async OCR flows (Bedrock, Azure) would reject inline images the sync path accepts. Add the same guard to the async function and a regression test that asserts the async path returns the data URL without touching the HTTP client * Fix: openai batches lifecycle * Fix: add e2e azure openai tests * Fix e2e for vertex ai * Add all models for testing * test(managed-files): assert idempotent upsert in store_unified_file_id store_unified_file_id switched from create to upsert to avoid UniqueViolationError when re-storing the same unified_file_id (e.g. batch output files stored before metadata is available). Update the unit test to assert the upsert call and its create payload instead of the removed create call. * test(batches): reconcile vertex_ai native batch-id comment with fallback guard * fix(test-config): keep rust-ocr models in model_list by moving files_settings after it * fix(test-config): move batch models after OCR block to keep merge with internal_staging clean * fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: ruff format transformation.py and endpoints.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition * test(vertex-ai/batches): align completion_window assertion to 24h * fix: update managed file metadata on upsert --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-02 08:05:23 -07:00
Sameer Kankute	a2a951a1e9	feat(vertex_ai): pass full imageConfig dict for Gemini image generation (#31811) ... * feat(vertex_ai): pass full imageConfig dict for Gemini image generation Support all ImageConfig fields (aspectRatio, imageSize, personGeneration, imageOutputOptions) when calling Vertex AI Gemini image generation endpoints. Previously only aspectRatio and imageSize were extracted; other fields were silently dropped. Co-authored-by: Cursor <cursoragent@cursor.com> * style: ruff format vertex_gemini_transformation Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex_ai): warn on non-dict imageConfig instead of silently dropping Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-07-01 23:29:59 -07:00
Mateo Wang	c4f28ce287	fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging (#31924) ... * fix(bedrock): trigger Nova Sonic generation on response.create so realtime sessions stop hanging (LIT-2239) * fix(bedrock): reopen audio content at client sample rate after trigger block * test(bedrock): cover realtime handler disconnect flush and stream-end guard * fix(bedrock): always close realtime input stream even if close flush fails * fix(lint): use contextlib.suppress in bedrock realtime cleanup to satisfy BLE001 budget * fix(bedrock): suppress bedrock close send errors per-message so promptEnd/sessionEnd still flush --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-07-01 19:11:18 -07:00
Mateo Wang	fde4c7c97a	feat(gdc): implement Google Distributed Cloud (GDC) Gemini provider (#31895) ... * feat(gdc): add Google Distributed Cloud Gemini provider support Introduce support for the Google Distributed Cloud (GDC) Gemini provider by adding "gdc" to the list of chat providers and enabling the gdc/ model prefix. The implementation defines a new GDCGeminiConfig class which handles authentication via Google Distributed Cloud service account credentials, manages token generation, formats GDC Gemini request URLs, and transforms request structures accordingly The PreProcessNonDefaultParams class is also updated to exclude vertex parameters from filtering when the custom LLM provider is GDC, allowing vertex parameters to be passed properly during GDC initialization * fix: resolve issues identified in PR #30702 * fix(gdc): harden credentials, fix vertex param filtering, add tests The supports_vertex_params branch regressed vertex_ai and vertex_ai_beta: the `if custom_llm_provider in [...]: pass` was a no-op, so those providers fell through to the config lookup, found no supports_vertex_params, and had their vertex_ params stripped. The check is now a single _provider_supports_vertex_params helper that keeps vertex_ params for the vertex family and for any config that opts in, and only swallows the expected ValueError from an unknown provider string instead of a blanket except GDC project and location now resolve from the deployment's litellm_params and the litellm.vertex_project / litellm.vertex_location globals before falling back to request optional_params, matching how vertex_ai resolves them, so a proxy caller can no longer route a request to a project the deployment did not expose A request api_key is no longer treated as a filesystem path, so a caller can't make the host open a local service-account file; api_key must be a literal service-account JSON string or a bearer token The opt-in token cache is hardened: the lock and cache dict are created in __init__ instead of via a racy hasattr lazy-init, the token is read inside the lock, and the audience is stripped of a trailing slash once so the cached and non-cached paths agree Also declares gdc_api_base, switches the lazy-import entry to the relative path every other entry uses, adds the missing trailing comma in the provider config map, and drops the api_base fallback that only ran when api_key was None Adds unit tests covering the vertex-param filter, deployment-over-request precedence, the api_key file-path rejection, URL construction branches, environment validation, token caching, and the gdc completion dispatch; transformation.py is fully covered * fix(gdc): prefer GDC-specific config, honor vertex_ai aliases, harden URL and bool parsing * fix(gdc): mint the GDCH token audience from the host, not the full base When api_base embedded /v1/projects/... and the deployment set project/location, get_complete_url rebuilt the request URL from the host while validate_environment still derived the token audience from the full original api_base, so the bearer token could target a different audience than the URL actually called. The audience is now the scheme://host of api_base in every case, matching the host get_complete_url builds against * fix(gdc): restrict JSON api_key to GDCH service accounts Only accept a credential whose type is gdch_service_account before calling google.auth.load_credentials_from_dict, so a caller-supplied external_account/identity_pool/pluggable credential carrying arbitrary token or credential_source endpoints is rejected before any token refresh runs. GDC only ever uses GDCH service accounts, and non-GDCH credentials could not have completed auth anyway (with_gdch_audience is GDCH-only), so this narrows the credential-refresh surface without changing valid GDC behavior. * fix(gdc): validate project and location as plain identifiers vertex_project and vertex_location can come from request params and were interpolated as raw path text into the GDC request URL and the x-goog-user-project header. A caller-supplied value containing / ? # or .. could reshape the path and make the proxy send its GDC-authorized request to a different endpoint under the configured host. Validate both against a strict identifier pattern before building the URL or header and raise an auth error otherwise; GCP project ids and locations are plain identifiers so valid deployments are unaffected. * fix(gdc): bind x-goog-user-project quota header to the deployment The quota project header was resolved with request-level vertex_project taking effect, so with a preformed deployment api_base a caller could set vertex_project to a different project and have it sent under the proxy's GDC credential, misattributing quota or billing. Resolve the header project the same way the URL is resolved: a preformed api_base without a deployment override binds to the project embedded in the URL, otherwise deployment and global config win over request params. This keeps the URL and the quota header consistent. * fix(gdc): always rebind x-goog-user-project, stripping caller-forwarded values The quota project header was only set when absent, so with client header forwarding an authenticated caller could send their own x-goog-user-project (any casing) and have it ride on the proxy's GDC credential, bypassing the deployment-derived binding. Strip every casing of the header and always set it from _effective_project before the request is signed. * fix(gdc): make a preformed api_base authoritative for project routing get_litellm_params copies caller-supplied vertex_project and vertex_location into litellm_params via OPTIONAL_KWARGS_KEYS, so litellm_params cannot be treated as a deployment-only source. The previous _deployment_overrides_path inference let an authenticated caller flip a pinned preformed api_base such as /v1/projects/pinned/... to /v1/projects/attacker/..., driving requests to a caller-chosen project with the proxy's configured GDC credentials and quota header A preformed /v1/projects/ api_base is now authoritative; get_complete_url returns it unchanged and _effective_project binds the x-goog-user-project quota header to the project embedded in that URL, so a caller can no longer redirect a pinned deployment or move the quota header off it. The two tests that asserted the override behavior are now regression tests that fail if the rewrite is reintroduced * fix(gdc): make a preformed api_base self-sufficient in get_complete_url get_complete_url resolved and required a params-derived vertex_project before returning a preformed /v1/projects/ api_base, so a deployment that pins its project in the api_base path was forced to also pass vertex_project or hit 'project is required'. validate_environment already extracts the project from a preformed URL and needs no such param, so the two paths disagreed The preformed-URL early return now runs before project/location resolution, matching validate_environment: a preformed api_base is returned as-is with no redundant param, and non-preformed bases still require vertex_project and vertex_location as before. Adds a regression test that a preformed base with no project/location params returns the URL unchanged --------- Co-authored-by: Paige O'Connor <lostpaige@google.com> Co-authored-by: Tim Laubach <tlaubach@google.com>	2026-07-01 17:31:07 -07:00
Mateo Wang	0b0fd6a4d1	feat(github_copilot): route /v1/messages to Copilot native Anthropic endpoint (#31802) ... * feat(github_copilot): route /v1/messages to Copilot native Anthropic endpoint Add a GitHub Copilot Anthropic Messages transformation that routes supported Claude models through the native /v1/messages endpoint. This covers request URL construction, default headers, and supported model metadata. * fix(github_copilot): address PR review feedback Tighten the Anthropic Messages environment validation and web search interception behavior after review feedback. Avoid treating non-web-search requests as web-search-only paths. * style(github_copilot): apply black formatting Apply Black formatting to the GitHub Copilot Anthropic Messages tests. * test(github_copilot): cover ProviderConfigManager dispatch for Anthropic Messages Add coverage for ProviderConfigManager dispatch when GitHub Copilot models use the Anthropic Messages API, including non-Anthropic models returning no config. * fix(github_copilot): apply messages-proxy intent header to /v1/messages Set the messages-proxy interaction header for GitHub Copilot Anthropic Messages requests so /v1/messages uses the expected Copilot intent. * fix(github_copilot): use modern generic annotations Replace legacy typing generics in the GitHub Copilot Anthropic Messages transformation so the strict Ruff budget gate stays within its ceiling. * refactor(github_copilot): decouple web-search short-circuit and harden /v1/messages URL Address review feedback on the Copilot native Anthropic messages path. Replace the hardcoded LlmProviders.GITHUB_COPILOT check in the web-search interception handler with a handles_web_search_natively() method on BaseAnthropicMessagesConfig (default True), overridden to False in GithubCopilotAnthropicMessagesConfig. Provider-specific behavior now lives in llms/ and the handler stays provider-agnostic, so a future provider in the same situation needs no carve-out here. In get_complete_url, reuse the already-resolved api_base returned by validate_anthropic_messages_environment instead of reading the authenticator a second time, removing redundant I/O and the mid-request inconsistency window. The caller-supplied base is still discarded in validate, which is the security boundary. Normalize a trailing slash on the base in both methods so a tenant-specific host never yields a double-slash //v1/messages URL. * fix(github_copilot): forward anthropic-beta headers on /v1/messages The Copilot config inherited should_filter_anthropic_beta_headers()==True from BaseAnthropicMessagesConfig, so update_headers_with_filtered_beta stripped every anthropic-beta value after validate_anthropic_messages_environment injected them (github_copilot has no mapping in anthropic_beta_headers_config.json). That silently disabled header-gated features like context_management and structured outputs on the native passthrough. Override the hook to False, matching OpenAILikeAnthropicMessagesConfig. * test(github_copilot): remove dead branch in beta-header regression test The anthropic-beta filtering test guarded the fix with an if branch on should_filter_anthropic_beta_headers(), which is always False, so the branch was unreachable. Replace it with a direct assertion that running the provider-scoped filter for github_copilot strips every beta value, proving why the override is load-bearing and catching a regression that flips it back on. --------- Co-authored-by: ririnto <ririnto@kakao.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-07-01 17:16:21 -07:00
devin-ai-integration[bot]	7e993446d8	feat(bedrock_mantle): add xai.grok-4.3 to model cost map for SigV4 auth (#31916) ... Register bedrock_mantle/xai.grok-4.3 with /v1/responses in supported_endpoints so the data-driven gate routes it through BedrockMantleResponsesAPIConfig (which inherits SigV4 signing via BedrockMantleAuthMixin). Without this entry the model falls through to None and forces bearer-token-only auth. Pricing sourced from AWS Bedrock pricing page. Closes #31196 Co-authored-by: unknown <> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-01 15:44:30 -07:00
Krrish Dholakia	ada9ef88ac	fix(websearch): websearch_interception agentic loop fixes for chat completions and anthropic messages (#31669) ... * fix(websearch): wire chat completion agentic loop to correct hooks maybe_run_chat_completion_agentic_loop was calling async_should_run_agentic_loop (Anthropic format) and async_run_agentic_loop (Anthropic path) instead of the chat-completion variants. This meant WebSearchInterceptionLogger never intercepted chat completion requests — the LLM returned a litellm_web_search tool_call but the agentic loop never executed, so the raw tool_calls response was returned to the caller. Fix: gate on async_should_run_chat_completion_agentic_loop override, call that hook and async_build_chat_completion_agentic_loop_plan / async_run_chat_completion_agentic_loop in the execution path. Regression test added. * fix(websearch): strip tool_choice from follow-up request When the original request forces tool_choice to litellm_web_search, the follow-up request after search execution inherited that tool_choice, causing the model to call the search tool again instead of synthesizing an answer from the results. * fix(websearch): inject api_key into agentic hook kwargs for anthropic messages Follow-up calls inside async_run_agentic_loop (e.g. websearch interception's synthesis call after executing Exa/Perplexity searches) were missing api_key because the named api_key param in async_anthropic_messages_handler was never merged into the kwargs dict forwarded downstream. Result: every /v1/messages websearch follow-up failed with "x-api-key header is required" and the caller received the raw tool_use response instead of the synthesized answer. * ci: trigger CI run * fix(websearch): support unified agentic hooks alongside chat-completion-specific hooks CodeInterpreterInterceptionLogger uses async_should_run_agentic_loop with _agentic_loop_api_surface to handle both surfaces from one hook. The chat completion loop must also check _gate_overridden so callbacks using the unified hook pattern still fire for chat completions. * fix(websearch): strip tool_choice from legacy chat completion follow-up call The _execute_chat_completion_agentic_loop path merged original optional_params (which includes forced tool_choice) into follow-up params without explicit removal. _build_chat_completion_request_patch already excluded tool_choice from its optional_params output, but dict.update() with a missing key leaves the original value intact. Explicit pop after the merge removes it. * fix(websearch): always strip tool_choice from plan-path follow-up params The tool_choice removal was gated on patch.tools is not None. WebSearch sets tools via patch.optional_params not patch.tools, so the gate was False and forced tool_choice from the original request survived into the synthesis call. Move the pop outside the patch.tools branch so it applies unconditionally.	2026-07-01 09:36:40 +08:00
Mateo Wang	6d828e5759	feat(messages): passthrough /v1/messages to native endpoints via supported_endpoints (#31685) ... * feat(messages): passthrough /v1/messages to native endpoints via supported_endpoints The unified /v1/messages proxy endpoint always translated inbound Anthropic requests down to /v1/chat/completions (or the Responses API for openai) when the deployment's provider lacked a native Anthropic-messages config, dropping Anthropic-only features like cache_control and thinking. Some customers run OpenAI-compatible servers (self-hosted vLLM, DeepSeek's Anthropic endpoint, etc.) that also natively expose /v1/messages and want the raw Anthropic payload forwarded untranslated, while keeping provider openai so /v1/chat/completions to the same deployment stays native. Opt in per deployment via model_info.supported_endpoints containing /v1/messages. When present, the gate routes to a generic, provider-agnostic OpenAILikeAnthropicMessagesConfig that POSTs the Anthropic payload to {api_base}/v1/messages with Bearer auth, instead of translating. Default behavior is unchanged. Generalizes and supersedes the hosted_vllm-only, env-var-toggled PR #28745. * fix(messages): preserve standard-cased caller headers in native passthrough The OpenAI-like Anthropic passthrough config only checked for lowercase header names before injecting Bearer auth, anthropic-version, and content-type defaults. A caller sending standard-cased Authorization, Anthropic-Version, or Content-Type was treated as missing those headers, so LiteLLM added duplicate lowercase variants and overwrote the caller's credential/version at the HTTP layer. Header presence is now checked case-insensitively and the merge no longer mutates the caller dict. Also moves the feature docs out of the main repo (docs live in litellm-docs). * fix(openai_like/messages): delegate to parent transform and inject anthropic-beta headers The passthrough config bypassed the parent transform and skipped header beta injection. Both gaps cause native /v1/messages features (context management, advisor tool, fast mode, structured outputs, reasoning_effort, advisor stripping) to silently degrade on opted-in deployments. Reuse the parent's pipeline and call _update_headers_with_anthropic_beta after merging defaults * fix: normalize anthropic-beta header key case before beta injection * style: collapse anthropic-beta header normalization to single line ruff format --check requires the comprehension on one line (it fits within the 120 char limit); fixes the lint job failure on the bugbot autofix commit * fix(messages): forward anthropic-beta to native passthrough upstream The shared anthropic_messages HTTP handler ran update_headers_with_filtered_beta with the deployment's custom_llm_provider after validate. For the native /v1/messages passthrough that provider is openai, which has no beta-header mapping, so every anthropic-beta value (caller-supplied or feature-derived for speed/context_management/etc.) was stripped to empty before the upstream request, breaking beta passthrough to the Anthropic-compatible endpoint. Beta filtering only makes sense on cross-provider translation paths where the upstream cannot understand Anthropic betas. Gate it on a new should_filter_anthropic_beta_headers() that defaults to True (bedrock, vertex_ai, native anthropic unchanged) and is overridden to False by OpenAILikeAnthropicMessagesConfig, whose upstream is a native Anthropic endpoint, so betas pass through verbatim. * chore: remove accidentally committed local QA logs and config --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-30 12:17:33 -07:00
Mateo Wang	26ee5dd597	fix(passthrough): drop top-level additional_drop_params on /v1/messages (#31645) ... * fix(passthrough): drop top-level additional_drop_params on /v1/messages On the Anthropic Messages pass-through path, additional_drop_params only stripped nested dotted paths, so plain top-level keys like `thinking` and `context_management` were forwarded to the provider. Bedrock rejects these with "Extra inputs are not permitted", returning a 400 to Claude App/CLI even when the user configured `additional_drop_params: ["thinking"]`. delete_nested_value already handles plain top-level fields, so route every drop param through it and remove the nested-only filter. Fixes #25931. * fix(passthrough): drop thinking for bedrock inference-profile ARNs on /v1/messages Opaque Bedrock Application Inference Profile ARNs contain neither "anthropic" nor "claude", so is_anthropic_claude_model returned False and the thinking param was rewritten to reasoning_effort before additional_drop_params ran. That made additional_drop_params: ["thinking"] a no-op for the converse-ARN form, and the Bedrock Converse transform re-expanded reasoning_effort back into additionalModelRequestFields.thinking, so the request 400'd. Extend the thinking-translation gates to also accept bedrock ARNs via the existing is_bedrock_arn_model helper, mirroring the cache_control path, so thinking is preserved as thinking and additional_drop_params can drop it.	2026-06-29 18:17:12 -07:00
mubashir1osmani	85840aef51	fix(vertex_ai/files): single media upload for batch files to fix 499s on large uploads (#31653) ... * fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads PR #31036 switched the vertex batch file upload from a single GCS media upload to a chunked resumable session. The resumable path sends the body as many sequential PUTs, each waiting a full round-trip to GCS before the next, so a multi-GB upload accumulates hundreds of round-trips and overruns the client/load-balancer request timeout, surfacing as 499s (client closed connection) on files as small as 500MB. This was a regression from the last-known-good commit, where the upload completed as one continuous request. Revert the batch upload to a single uploadType=media request, but stage the transformed payload to a temp file first so peak memory stays bounded (the goal of the resumable rewrite) without the per-chunk round-trips. The temp file is closed deterministically (TemporaryFile unlinks on close), not left to the GC. The now-unused resumable chunked-upload plumbing is removed. Also swap the per-row transform's stdlib json for orjson (parse + serialize), which is ~4x faster on this hot path; the streaming body now emits compact orjson bytes. The request stays synchronous, so the returned file object is real and POST /v1/batches keeps working immediately against the uploaded object. Tests: single media request carries the whole payload with a real Content-Length (no chunked transfer-encoding); failed upload raises; the staged temp file is closed deterministically; byte-for-byte transform parity. * test(vertex_ai/files): mock single media upload POST instead of removed resumable method test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type. * fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports Forward the per-request timeout through _stage_and_upload_media / _astage_and_upload_media to the GCS POST. Every other upload branch forwards it; the new media path was dropping it, so a caller-provided timeout was silently ignored (the files path passes 600s by default, but a custom request_timeout would not have reached this upload). Regression test asserts the resolved timeout reaches the request (mutation-verified). Revert the orjson swap in the batch transform: importing orjson at module load in this core-path file broke `import litellm` on environments without orjson (the Windows import test). Back to stdlib json; the upload leg dominates large uploads anyway, so the transform-side win was marginal. Fix import ordering in llm_http_handler.py (I001) introduced by the new imports. * fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file Addresses a disk-exhaustion concern: staging the full transformed batch body to a local temp file before the GCS request meant an authenticated user could fill the proxy's temp volume with large concurrent uploads (on top of Starlette's input spool). GCS's simple/media upload accepts chunked transfer-encoding, so stream the transform straight to the single media request instead. Each block is produced on a worker thread (the transform never runs on the event loop) and sent chunked, so the body is neither buffered in memory nor written to disk, and the upload is still one continuous request (no per-chunk round-trips, no 499). Drops the temp-file staging, the tempfile/IO imports, and Content-Length computation. Regression test asserts the upload streams (chunked transfer-encoding, no Content-Length) and creates no temp file; mutation-verified that reintroducing staging fails it.	2026-06-29 17:31:32 -07:00
Mateo Wang	20dabb781a	fix(databricks): split parallel tool calls so each tool message follows tool_calls (#31633) ... * fix(databricks): split parallel tool calls so each tool message follows tool_calls Databricks OpenAI-compatible serving (e.g. GPT models) 400s with "messages with role 'tool' must be a response to a preceeding message with 'tool_calls'" when an assistant turn makes parallel tool calls. LiteLLM faithfully sends one assistant message holding all tool_calls followed by one 'tool' message per result, so every result after the first is preceded by another 'tool' message rather than the assistant tool_calls message, which Databricks rejects. Re-emit each result immediately after an assistant message that carries only its matching tool_call, turning assistant(tool_calls=[A, B]), tool(A), tool(B) into assistant(tool_calls=[A]), tool(A), assistant(tool_calls=[B]), tool(B). The rewrite is a no-op when the turn is already valid (single call), the group is incomplete, or ids don't line up, so no tool call is ever dropped. Scoped to non-Claude models, matching the existing OpenAI-shaped transformation path. * style(databricks): use builtin list generics in parallel tool-call split Switch the List[...] annotations introduced by _split_parallel_tool_calls to lowercase list[...] so the UP006 strict-rule budget stays within its ceiling.	2026-06-29 13:46:53 -07:00
Sameer Kankute	2cf565ae28	test(batches): add 1:1 test file scaffold for batches component paths (#30529) ... * test(batches): add 1:1 test file scaffold for batches component paths Co-authored-by: Cursor <cursoragent@cursor.com> * Add harness test for create batch endpoint * Add retrieve endpoint harness tests * Add list endpoint harness tests * Add cancel endpoint harness tests * Add cancel endpoint harness tests * Add test for litellm/batches/main.py * Add test for litellm/tests/test_litellm/batches/test_batch_utils.py * Add handler and transformation tests for all providers * Fix: run batches tests in cicd * fix(tests): remove azure/__init__.py that shadowed azure namespace package Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to insert tests/test_litellm/llms/ into sys.path[0], making our empty azure/ dir shadow the real azure-identity namespace package. Any test that patched azure.identity.* would then fail with AttributeError. * style(tests): apply ruff format to test_batch_utils.py Base migrated the formatter from black to ruff format (#31317); reformat the batches scaffold test file to match. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-29 09:22:58 +05:30
Mateo Wang	b76a858826	feat: declarative fallback generalizations for unknown models (#29718) ... * feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match	2026-06-27 21:01:19 -07:00
mubashir1osmani	ef5d05f137	fix(realtime): stop second Gemini Live setup, retry hung handshake, close guardrail bypass (#31519) ... * fix(realtime): stop sending a second Gemini Live setup on follow-up session.update Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client message; a second setup closes the socket with 1007 Request contains an invalid argument. The AI Studio Gemini path forwarded every client session.update after the first as a follow-up setup, and GA clients (pipecat) send several while configuring the session, so the second one tore the session down before the first turn. Callers saw silence after the first response, exponential per-turn latency from reconnect/retry churn, and intermittent 1011 errors. Drop subsequent session.updates instead of resending setup, matching what the Vertex subclass already does. Tools and instructions must ride on the first session.update before any conversation content. Adds regression tests covering the plain follow-up, a follow-up that adds tools (the case the previous identical-only dedup still forwarded), and the guardrail create_response=False warning path. * fix(realtime): retry the backend open handshake instead of failing with 1011 The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs; waiting longer never recovers a hung attempt, but a fresh attempt almost always connects in ~1s. The proxy opened the backend websocket once with the default open_timeout and no retry, so a single slow handshake surfaced to the caller as a fatal 1011 internal error and dropped the call. Bound each open attempt with a short open_timeout and retry; a bounded attempt that already timed out spaces out the next try, so no backoff is needed. Deterministic handshake-status rejections (auth/4xx) are not retried, and the retry only ever wraps the open, never a live session. Adds tests for retry-then-succeed, raise-after-max-attempts, and no-retry-on-auth-failure. * fix(realtime): close guardrail bypass + surface handshake status; drop obsolete tests Three review fixes on the Gemini Live realtime path. Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once the initial setup is sent the guardrail's automaticActivityDetection.disabled=true can no longer be delivered as a follow-up session.update. With that follow-up now dropped, the model's auto-response stayed enabled and a realtime_input_transcription guardrail was bypassed (the model answered before the proxy could gate the turn). Fold the disable into the one-and-only setup instead: the handler injects it into the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it into the deferred first setup. OpenAI sessions accept follow-up updates and are left untouched. Backend handshake status: the open-retry treated only InvalidStatusCode as deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake, so a 401/403 fell into the broad WebSocketException branch and was retried before the caller closed the client with 1011 instead of the upstream status. Treat both as non-retryable. Obsolete tests: the four tests asserting a follow-up session.update is merged and re-sent as a second setup asserted behavior that crashes Gemini Live with 1007 (verified directly against the API). Removed; the drop is covered by new regression tests. * style(realtime): reformat changed files to ruff line-length 120 Post-merge with litellm_internal_staging, which unified ruff format width to 120 (#31518). The realtime change set was formatted at 88, so the changed lines tripped the whole-file ruff format check. Reformat with ruff 0.15.3 at the repo's 120 width; no logic changes.	2026-06-28 08:52:20 +05:30
Shivam Rawat	d515e5bf05	fix(vertex_ai): append rawPredict suffix for custom api_base on /v1/messages (#31529)	2026-06-27 14:54:57 -07:00
Mateo Wang	64d8d7f8cb	fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke (#31364) ... * fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke * style(bedrock): use builtin generics in new Invoke helpers to clear UP006 gate * fix(bedrock): honor explicit thinking budget_tokens=0 in clear_thinking conversion The clear_thinking_20251015 -> adaptive conversion resolved the thinking budget with `thinking.get("budget_tokens") or BEDROCK_MIN_THINKING_BUDGET_TOKENS`, which treats a caller-supplied `budget_tokens=0` as missing and silently substitutes the Bedrock minimum. Resolve the budget with an explicit `is not None` check so an explicit 0 is honored. * fix(bedrock): gate Fable 5 into clear_thinking adaptive injection on Invoke _ensure_thinking_for_clear_thinking_context_management returns early when _supports_extended_thinking_on_bedrock(model) is False, so the adaptive-thinking injection never runs for models absent from that gate. Opus 4.8 slips through on the incidental "opus-4" substring, but Fable 5 had no matching pattern, so a clear_thinking_20251015 request on Fable 5 reached Bedrock with an unsupported context-management edit and no thinking field; the exact 400 this path exists to prevent. Add the fable-5 patterns to the gate so Fable 5 (mapped ids and unmapped aliases) gets thinking.type=adaptive + output_config.effort like the other adaptive models. Extend the adaptive-injection regression test to cover Fable 5 (a mapped id and an unmapped alias) so it fails without the gate entry, and add focused coverage for the budget->effort tiers, the disabled/enabled/adaptive thinking branches, output_config.effort preservation, and list/dict system-role normalization. Also normalize the Invoke transformation module and its test to line-length 88 so ruff format --check (CI format-check) passes. * refactor(anthropic): make supports_adaptive_thinking flag authoritative for thinking detection Replace the per-version name helpers (_is_claude_4_6/4_7/4_8_model, _is_claude_fable_5_model) with cost-map-flag-first detection. _is_adaptive_thinking_model now reads supports_adaptive_thinking from the model cost map and falls back to a single generalized family-version regex (_claude_version_at_least(model, 4, 6)) only when a model is unmapped, instead of hard-coding each new Claude release. Wire supports_adaptive_thinking through ProviderSpecificModelInfo and ModelInfo so the cost map flag actually surfaces at lookup time. Reroute the Bedrock Invoke extended-thinking gate and the two anthropic/chat/transformation.py call sites through _is_adaptive_thinking_model. Known gap left to the fallback_generalizations work (#29718): unmapped Fable 5 aliases have no parseable minor version, so they defer to the cost map and are not detected until a mapped entry or a generalization rule exists. Covered by an explicit regression test. * refactor(anthropic): drop name-based version fallback; resolve adaptive thinking from cost map only The prior commit kept a regex (_claude_version_at_least) as a fallback when an id resolved to no cost-map entry. Remove it: _is_adaptive_thinking_model now reads supports_adaptive_thinking and nothing else, so "which Claude versions think adaptively" lives entirely in the model cost map, and a new adaptive release is a JSON edit rather than a Python edit. To keep the flag authoritative across the id forms the Bedrock Invoke and anthropic paths actually see, backfill supports_adaptive_thinking=true on every adaptive Claude entry that was missing it (Opus 4.6/4.7 and Sonnet 4.6 across region/provider aliases) in both the root and bundled cost maps, and generalize _model_map_lookup_candidates to normalize an id to its base cost-map key: strip a Bedrock version suffix (-v1:0 fully, or just the :0 inference-profile minor so the -v1-keyed 4.6 entries resolve), strip a dated-release suffix (-20260219), and rewrite a dotted family version (4.6 -> 4-6). This is id normalization feeding the lookup, not capability-by-name. Tests load the PR-local cost map (the flags are not on main until merge) and cover each normalization path plus the unmapped-alias deferral to fallback_generalizations (#29718). * refactor(reasoning_effort): single-source effort<->thinking-budget mappings Route every reasoning_effort <-> thinking-budget conversion through the DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants so the numbers stay in sync across providers. The five constants are now 2000/5000/10000/20000/40000 Add reasoning_effort_from_thinking_budget() in litellm_core_utils/reasoning_effort_utils.py and route the three OpenAI-style forward maps (anthropic adapters, responses adapters, hosted_vllm) through it. The bedrock invoke and experimental messages adaptive maps now reference the constants directly; the only behavior change is the xhigh threshold moving from 24000 to 20000. Reverse maps and the cross-provider test grid read the same constants * test(reasoning_effort): lift budget-mode max_tokens above the new high budget The single-sourced DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET thresholds moved high from 4096 to 10000. The live reasoning_effort grid sends budget-mode requests with max_tokens=8192, so reasoning_effort=high now produces budget_tokens=10000 > max_tokens and every provider returns 'max_tokens must be greater than thinking.budget_tokens'. Derive a shared BUDGET_MODE_MAX_TOKENS (2x the high budget) for the spec and the request builder so the ceiling always clears the largest 200-expected tier. Also resolve the inherited base test_reasoning_effort assertion off the same high-budget constant instead of the stale 4096 literal so it tracks the source of truth. * fix(reasoning_effort): keep effort<->budget thresholds at pre-PR values The single-sourcing refactor moved the shared effort<->budget thresholds up (low 1024->2000, medium 2048->5000, high 4096->10000, xhigh 8192->20000, max 16384->40000). That silently changes the effort->budget direction: a caller who sets reasoning_effort together with a max_tokens that used to sit above the old per-tier budget but below the new one now trips the provider's "max_tokens must be greater than thinking.budget_tokens" 400. It spans every backend that derives a budget from an effort (Anthropic, Gemini/Vertex, hosted vLLM), not just Bedrock. Restore the constants to their pre-PR values while keeping every backend reading from the shared DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, so the mapping stays single-sourced without the behavior change. Tests that pinned the raised thresholds now derive their boundaries from the same constants. * test(reasoning_effort): derive high effort->budget assertions from the shared constant The cross-provider translation tests pinned reasoning_effort="high" to a literal budget_tokens=10000, the raised value. Point them at DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET so they track the single source instead of a magic number. * fix(anthropic): resolve adaptive flag for combined dated+versioned Bedrock ids The model-map candidate normalization applied each suffix strip independently to the original id, so the real Bedrock shape "<base>-<YYYYMMDD>-v1:0" never reduced to its base cost-map key: stripping the version left the date, and the dated-suffix regex is anchored to the end so it could not fire while the version was still present. An adaptive Claude model invoked by its full dated+versioned id (e.g. us.anthropic.claude-sonnet-4-6-20251101-v1:0) therefore resolved to supports_adaptive_thinking=null and was treated as non-adaptive, reaching Bedrock with the rejected thinking.type=enabled shape, the exact 400 this path prevents. Add a composed normalization that rewrites the dotted family version, then peels the -vN:rev version suffix, then the -YYYYMMDD dated suffix, so the combined form resolves to its base key. Regression tests pin the combined suffix on sonnet-4-6 and opus-4-8 across provider/region prefixes. * fix(reasoning_effort): align budget<->effort tests with reverted constants and format common_utils The constant revert restored the effort<->budget thresholds to their pre-PR values (1024/2048/4096/8192/16384) and single-sourced the reverse budget->effort ladder through reasoning_effort_from_thinking_budget, but several tests still pinned the briefly-raised values and the old hardcoded reverse buckets, so the "All Other Providers" shard failed Derive the anthropic chat effort->budget assertions from the shared DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, and update the experimental pass-through and responses adapter expectations to the single-sourced reverse ladder (budget 1024 -> low, 5000 -> high) Also run ruff format --line-length 88 over anthropic/common_utils.py so the CI format-check, which checks the whole changed file, passes	2026-06-27 11:35:36 -07:00
Yassin Kortam	bd5e046464	fix(bedrock): surface web identity token aud/iss on InvalidIdentityToken (#31412) ... When STS rejects a web identity token with InvalidIdentityToken (the "Incorrect token audience" case), litellm propagated the raw botocore error, which never names the aud LiteLLM actually sent. Diagnosing an audience mismatch then required enabling LITELLM_LOG=DEBUG on the prod instance, which degrades performance. _auth_with_web_identity_token now catches InvalidIdentityTokenException, decodes the public aud/iss claims of the resolved JWT without verifying its signature (no secret is read), and raises an AwsAuthError that preserves the STS reason and names the token audience and issuer, so the mismatch is visible from the error alone. Resolves LIT-4026	2026-06-26 13:03:09 -07:00
Mateo Wang	5a1c7839be	feat(mistral): add mistral/mistral-ocr-2512 (OCR 3) to cost map (#31463) ... Adds the OCR 3 model (mistral-ocr-2512) released 2025-12-18 to both the root and bundled backup cost maps at $2 / 1000 pages and $3 / 1000 annotated pages, mirroring the existing Mistral OCR entries. Regresses the pricing in both maps and verifies completion_cost scales per page.	2026-06-26 10:29:07 -07:00
Sameer Kankute	4476923ac4	test: add realtime proxy e2e suite across providers (#30960) ... * tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * test: multi-window budgets coverage * fix: p0 issues, added types and shared functions for each test suite * chore: add config.yml * test: passthrough endpoints stream/non-stream e2e * style: carry clearer status_code comparison into renamed e2e dir * fix: rename cost breakdown function * fix: pydantic validation for budget info, dont allow explicit type cast * refactor: migrate to gateway client * test: add custom pricing tests * chore: change master key * test(e2e): address greptile review feedback Remove the duplicate cache/cache_params block in the gateway config so the two can't silently diverge under future edits. Reorder the soft-budget test to assert the call isn't a budget block before require_successful_call, since that helper hard-fails any non-2xx and left the budget-block check unreachable; the misleading "skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it so a failed delete doesn't leak a budget on the shared proxy. Scope the spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup import so a broader "pytest tests/" run isn't left with a mutated path. * test(e2e): drop misleading skip comment on require_successful_call require_successful_call fails hard, it does not skip; the trailing comment was factually wrong. The function name already states intent, so the comment is removed in both per-model and tag budget helpers. * test(e2e): assert budget-isolation invariant before success check On the should-still-succeed path of the per-model and tag isolation tests, check is_budget_block before require_successful_call. If the isolation bug fires the unaffected model/tag is blocked, so asserting the specific 'blocked by X' invariant first yields the diagnostic message instead of a generic upstream-failure. Matches the ordering in test_soft_budget_e2e.py. * fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows * fix(e2e): run case init() inside try so partial-init failures tear down run_case called case.init() outside the try/finally that runs teardown(), so a case that registers cleanups progressively (create team, then user, then key) and then fails partway through init() would leak the already-created entities on the long-lived shared proxy. Move init() inside the try so teardown always runs. Add a regression test that registers a cleanup then raises mid-init and asserts the resource is still released. * test(e2e): mark known pricing-leak isolation test xfail(strict) test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy gap (a deployment's custom per-token pricing leaks into the shared cost map for sibling deployments of the same underlying model) and was left unconditionally failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True) so the suite stays green while the leak persists and turns into a failure the moment isolation is fixed, prompting the marker's removal. * refactor(e2e): make suite pass its shipped strict basedpyright config The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright --project tests reported four errors in it: three reportAny on the parametrize ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed autouse fixture _require_live_proxy. Replace the untyped lambda with a typed _case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and rename the fixture to require_live_proxy so basedpyright no longer treats it as an unused private function (it is referenced only by pytest's autouse machinery). basedpyright --project tests now reports zero errors. * fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory * test(e2e): run harness unit tests without a live proxy The autouse session fixture skipped the whole tests/e2e session when no proxy answered, which also skipped test_lifecycle.py, a pure unit test of run_case that never touches the proxy. A regression test that silently skips gives no signal, so the skip now lives in pytest_runtest_setup gated on the same e2e marker the spend-log truncate guard already uses: live tests skip when no proxy is up while harness unit coverage always runs. The liveness probe is cached with lru_cache so it still runs once per session * test(e2e): clean up gateway config comment debris Fix the typo on the header comment and drop the orphaned namespace/ttl comment remnants left indented under cache_params; the active values are already set above. Flagged by greptile review. * fix: add new tests, split gateway * test(e2e): type the redis spend-counter probe for strict basedpyright The new cold-counter reseed test drove its redis client untyped, so the strict tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the file landed: scan_iter/get came back unknown and the pool.map lambda had an untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING import (the runtime import stays lazy so the suite still skips, not errors, when redis is absent), which resolves scan_iter to Iterator[str] and get to str | None, and replace the lambda with a typed inner function mirroring _burst. basedpyright --project tests is back to zero errors. * test(e2e): xfail the known team multi-window failure and isolate member teardown Greptile flagged two issues in the mirrored split-gateway commit. The team multi-window budget test documents a real /team/new write bug (budget_limits go straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and /team/update paths) and was left as an unconditional hard failure, which would turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing isolation test so the suite stays green while the bug persists and flips to a failure the moment the write is fixed and the marker should go. The class-scoped member fixture in test_team_member_budget_e2e.py tore down its key, user, and team sequentially with no exception isolation, so a failed delete_key would strand the user and team on the long-lived shared proxy. Route cleanup through a ResourceManager: register each delete progressively and run them LIFO best-effort in a finally, so a partial-setup failure still releases what came before and one failed delete never blocks the rest. * test: add realtime proxy e2e suite across providers Add tests/realtime_e2e covering the proxy realtime websocket endpoint end to end against live providers (openai, azure, gemini, vertex_ai, bedrock, xai). Two layers: a raw-websocket suite asserting the normalized OpenAI GA event sequence, delta/transcript consistency, usage, and a full tool-call round-trip; and a pipecat smoke driving the proxy through the GA OpenAIRealtimeLLMService. Tests carry a new realtime_e2e marker and skip cleanly when the proxy or provider creds are absent, so they stay out of the default unit run. * test: move realtime e2e suite into tests/e2e harness Replace the standalone tests/realtime_e2e with a tests/e2e/realtime suite that follows the existing e2e conventions: a session-scoped client fixture, a frozen-dataclass RealtimeClient wrapping the shared Gateway, pydantic models for every sent and received event, and the e2e marker with the parent harness's liveness skip. The suite opens the proxy realtime websocket (websockets.sync to stay synchronous like the rest of the harness) and asserts the normalized OpenAI GA event sequence for a text conversation plus a full tool-call round-trip, parametrized across providers. A provider whose realtime alias is not configured on the proxy skips via /model/info. Adds a gemini realtime model to the gateway config and fixes the openai realtime model id. * test: add pipecat realism layer to realtime e2e suite Add test_realtime_pipecat_e2e driving the same providers through pipecat's GA OpenAIRealtimeLLMService with base_url pointed at the proxy, as a coarse realism check on top of the raw-websocket suite. Each test stays synchronous and runs the async pipecat pipeline via asyncio.run, and the module skips unless pipecat-ai is installed. Lift the shared provider matrix, ws-url helper, and skip helper into realtime_client so both suites use them. * fix(e2e): parse GA realtime transcript events in e2e client The realtime e2e client speaks the GA protocol, but transcript() only aggregated beta delta event names. Handle GA deltas, fall back to response.done output, and accept nested usage details on response.done. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): address realtime code-review findings - Use the real openai/gpt-4o-realtime-preview model ID in the gateway config (gpt-realtime-2 does not exist and would fail every live test) - Pass a bare base_url to pipecat's OpenAIRealtimeLLMService so pipecat can append ?model= itself; the previous realtime_ws_url already contained ?model= causing a malformed duplicated query parameter - Wrap connection.recv() in a try/except TimeoutError in collect_until so a deadline expiry inside recv preserves the collected-events diagnostic instead of raising a bare, message-free exception Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): filter configured_models to mode:realtime entries only ModelInfoEntry.model_info used CustomPricing (extra="ignore") so the mode field from /model/info was silently dropped, making it impossible to distinguish realtime from non-realtime deployments. Add an optional mode field to CustomPricing and filter configured_models() to entries whose model_info.mode == "realtime" so skip_if_unconfigured never accidentally skips a realtime test due to a naming-pattern collision with a non-realtime deployment. Co-authored-by: Cursor <cursoragent@cursor.com> * Update litellm-config.yml * fix(e2e): use TypeVar instead of PEP 695 generic in realtime parse_last PEP 695 type-parameter syntax (def f[T: Bound](...)) is only parseable on Python 3.12+, but the project declares requires-python >=3.10. Importing the realtime e2e client on 3.10/3.11 raised a SyntaxError before any test could run. Switch parse_last to the backport-safe TypeVar idiom so the suite imports across the full supported range. * fix(e2e/realtime): use GA openai/gpt-realtime model id The realtime gateway config used openai/gpt-realtime-2, which is not a real OpenAI model id and would 404 once live OpenAI realtime credentials are wired in. The GA speech-to-speech model is openai/gpt-realtime (snapshot gpt-realtime-2025-08-28); switch the openai-realtime alias to it. * fix(realtime): harden Gemini/Vertex Live for audio-native e2e Coerce TEXT responseModalities to AUDIO on native-audio and flash-live models, suppress the orphan turnComplete response.done that arrives immediately after tool results, omit function_response.id on Vertex, stop appending client query params to Gemini/Vertex WSS URLs, and add regression tests for these paths. Co-authored-by: Cursor <cursoragent@cursor.com> * Add xai full compatibility * Add working vertex ai realtime tests * Add audio + server vad e2e tests * Add config for e2e testing models * Add fix xai server vad * fix: use correct OpenAI realtime model ID in e2e gateway config openai/gpt-realtime is not a valid model; replace with the correct openai/gpt-4o-realtime-preview model ID to prevent model-not-found errors when running the openai-realtime e2e tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert: restore openai/gpt-realtime model ID gpt-realtime is a valid model; reverting the unnecessary change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve UP006 violations, mock test failures, and stale spec field - Guard gemini setup-without-tools deferral with litellm.gemini_live_defer_setup flag so the default (False) path sends setup immediately, fixing two failing mock tests: test_client_ack_caches_setup_to_prevent_duplicate_session_update_setup and test_deferred_setup_sends_session_update_before_buffered_audio - Replace deprecated typing generics (Dict, List, Tuple, Optional) with builtin equivalents in xai/realtime/transformation.py, gemini/realtime/transformation.py, and realtime_streaming.py to satisfy the UP006 ruff-strict ceiling - Remove 'role' from OpenAPI compliance test expected fields; Google removed it from the Interaction schema in their live spec Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use Optional[dict] in xai normalizer to preserve Black line-split dict[str, Any] | None is shorter than Optional[Dict[str, Any]] by enough that Black collapses the _normalize_usage signature to a single line (86 chars), conflicting with the existing multiline format. Using Optional[dict[str, Any]] keeps the line at 90 chars (> 88 limit) so Black preserves the multiline shape, while still satisfying UP006 by replacing Dict with dict. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove proxy-level setup-tools deferral, delegate to transformer The _gemini_setup_deferred / _gemini_pre_setup_buffer block in _send_to_backend was double-deferring: GeminiRealtimeConfig already handles the session.update-to-setup mapping internally and always returns a ready-to-send setup on the first session.update call (session_configuration_request=None). The proxy layer was incorrectly holding back that setup waiting for tools that the transformer had already incorporated. Removing the block fixes two failing tests: test_client_ack_caches_setup_to_prevent_duplicate_session_update_setup test_deferred_setup_sends_session_update_before_buffered_audio Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: abstract Gemini protocol keys out of core and use cost map for live model detection Move Gemini-specific message key knowledge (setup, realtimeInput, clientContent, toolResponse) out of the core RealTimeStreaming module into provider-level methods. BaseRealtimeConfig gains is_setup_message and is_content_message (both default False); GeminiRealtimeConfig overrides them with the actual Gemini key checks. Add gemini_native_audio and gemini_audio_only_live capability flags to the 10 affected model entries in the cost map. _is_audio_only_live_model and _is_native_audio_model now read from the cost map first and fall back to the existing string markers for models not in the map. * fix: apply black formatting and register gemini capability fields in schema * refactor: drop string-marker fallback; resolve audio-only live models via cost map only * fix: use registered cost-map model name in vertex realtime tests * fix: patch cost map in tests so they don't depend on remote main branch state * fix: align gateway config vertex-realtime model ID with cost-map registered name * fix: patch gemini-2.5-flash-native-audio in cost map fixture for CI * fix(e2e): use correct OpenAI realtime model id in gateway config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(e2e): add budget rescheduler short intervals to gateway config Without proxy_budget_rescheduler_min/max_time set, the rescheduler defaults to ~600s, causing all budget-reset e2e tests to timeout before the reset fires. Set to 5–10s so tests complete within 90s. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(e2e): strip non-realtime files from PR scope Restore budget, spend-tracking, and custom-pricing test files to their litellm_internal_staging state. Keep the mode field addition to CustomPricing in models.py (needed by realtime configured_models filter). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): restore async_realtime regression test and add missing fixture - Restore the end-to-end async_realtime regression test for Vertex query-param forwarding; the previous unit-only version did not exercise the code path where the original bug lived - Add patch_gemini_audio_cost_map_entries fixture to test_gemini_audio_only_live_models_drop_text_from_text_audio_combo so it does not depend on the cost map having gemini_audio_only_live set in CI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): resolve ANN401 violations in realtime streaming code Define RealtimeEventNormalizer Protocol and replace bare Any annotations with typed alternatives (object for event/value params, the Protocol for the normalizer) to stay within the strict-rule budget. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: black format realtime_streaming.py * fix(tests): add gemini_native_audio and gemini_audio_only_live to model prices schema * fix(lint): fix I001 import sort order in realtime_streaming.py * fix(lint): restore import litellm to correct position before from-litellm imports * undo budget removal * test(e2e): pin explicit credentials for gemini and vertex realtime models * test(e2e): share keepalive-safe LiteLLMRealtimeLLMService across pipecat suites The pipecat smoke test drove the proxy through the stock OpenAIRealtimeLLMService, which sends websocket keepalive pings at its default interval. The proxy does not answer them, so the connection is closed with a 1011 before the run completes. Move the proxy-aware LiteLLMRealtimeLLMService (keepalive disabled) into a shared pipecat_service module and use it from both the smoke and audio suites. * test(e2e): document that LiteLLMRealtimeLLMService._connect keeps the ?model= param The proxy routes realtime websockets on the ?model= query param, and pipecat's OpenAIRealtimeLLMService.__init__ bakes it into self.base_url before _connect runs. Passing self.base_url through preserves it; spell that out so the override is not misread as dropping the param. * fix(realtime): set _content_sent_after_setup only after the backend send succeeds A failed content send used to flip _content_sent_after_setup to True before the send was confirmed, mirroring the correct-on-failure ordering the adjacent session-config cache already follows. If the send raised, the flag stayed True and a later session.update that produced a setup frame was silently dropped even though the backend never received any content. Set the flag after the send succeeds and add a regression test that fails if the ordering is reverted. * fix: normalize realtime passthrough events * refactor(realtime): declare patch_outgoing_session on normalizer Protocol; fix wav chunk return type The RealtimeEventNormalizer Protocol only declared should_drop and normalize, so the outgoing session.update patch went through a getattr(..., None) lookup even though should_drop/normalize are called directly. The sole implementer (XAIRealtimeNormalizer) already provides patch_outgoing_session, so declare it on the Protocol and call it directly for consistent, fully-typed dispatch. Also correct _load_wav_chunks' return annotation from list[bytes] to tuple[list[bytes], int]; it returns (chunks, sample_rate) and the caller unpacks both. --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-26 09:36:49 -07:00
Sameer Kankute	133da06aa3	chore: litellm oss staging (#31185) ... * fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped The Total Tokens Over Time and Total Requests Over Time AreaCharts on the Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow once totals pass the hundred-million mark — leading digits of labels like "100.00M" / "4500.00M" got clipped against the chart edge. The requests chart was worse: it formatted with toLocaleString(), so billion-scale request counts produced "1,000,000,000" (13 chars) and overflowed immediately. Fix in two places so neither alone has to carry the whole margin: - activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and switch the requests chart to the shared valueFormatter so it uses the same compact k/M/B suffixes as the tokens chart. - value_formatters.tsx: add a >= 1e9 branch to valueFormatter / valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping every formatted label at most 7 chars. Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com> * Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * docs(readme): add Deploy on AWS/GCP with Terraform section Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): add 1-click deploy buttons for AWS + GCP GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): move AWS + GCP deploy buttons next to Render button * docs(readme): unify deploy button sizes and badge styles * docs(readme): bump deploy button height to 48 to match Render/Railway * docs(readme): bump AWS/GCP badge height to compensate for SVG padding * docs(readme): bump AWS/GCP badge height to 72 * docs(readme): bump AWS/GCP badge height to 84 * fix(readme): make deploy buttons same height (48px) https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc * docs(readme): flag GCP project ID substitution in image_registry * docs(readme): equalize deploy button heights and fix Cloud Shell button font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface. * docs(readme): collapse Railway deploy anchor to a single line The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline. * Add Claude Fable 5 cost map entries as a data-only hotfix Backports only the model map changes from #30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano Three bugs in model_prices_and_context_window.json: 1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K max output, but the values were set as max_input=128000, max_tokens=272000. This caused token limit errors when sending prompts over 128K tokens to GPT-5 Pro. 2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was 272000, but GPT-5.4 Mini shares the same 1,050,000 token context window as GPT-5.4. This was inconsistent with the azure/ variants which already correctly had 1,050,000. 3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini, max_input_tokens was 272000 instead of 1,050,000. Source: OpenAI model documentation and contextwindows.dev which aggregates official context window sizes. Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini should be 400K; their 272K values are correct per OpenAI docs) * fix: also correct max_output_tokens for gpt-5-pro (272000→128000) Per reviewer feedback, max_output_tokens was left at 272000 while max_tokens was corrected to 128000, causing an internal inconsistency. Both should be 128000 per OpenAI docs. * fix(cost): price gpt-image generated output tokens as image tokens (#31147) The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return usage with no output token breakdown — litellm's `ImageUsage` has no `output_tokens_details` field — so generated-image OUTPUT tokens were priced at the text rate (`output_cost_per_token`) instead of the image rate (`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x undercount on the dominant cost component (image output is ~74% of spend). This also affects azure gpt-image, which shares this calculator. The OpenAI gpt-image cost calculator re-implemented usage handling instead of reusing `calculate_image_response_cost_from_usage`, the shared helper that azure_ai/gemini/vertex_ai already use. That helper classifies generated output tokens as image tokens when the provider does not itemize output, and splits text/image when it does. Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage` (pre-transformed chat Usage objects are still costed directly). Adds a regression test for the no-breakdown ImageUsage case (gpt-image-2). * fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098) A bare application-inference-profile ARN passed as bedrock/arn:... fell through to the invoke route, which cannot derive a provider from the opaque profile id and raised 'Unknown provider=None'. The converse route needs no provider, so detect these ARNs in get_bedrock_route and route them to converse, matching the behavior of the already-documented bedrock/converse/arn:... workaround. Explicit invoke/ prefixes still win, and they remain a dead end for these ARNs by design (no provider derivable). System-defined inference-profile ARNs that embed a known model, and other opaque ARN types (provisioned-model, imported-model, custom-model-deployment) that are frequently invoke-only, are deliberately left on their current routes; tests guard both boundaries. * fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060) _add_tool_choice_required_message appended the "select a tool" prompt to the caller's messages list in place, so transform_request corrupted the caller's conversation history and appended a duplicate prompt on every retry. Build and return a new list instead so the call stays idempotent. Adds a regression test asserting the input messages list is unchanged across repeated transform_request calls. Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> * fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996) gpt-4o-transcribe and compatible ASR backends return a diarized_json response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8. TranscriptionUsageDurationObject typed seconds as int, so parsing the response raised a pydantic ValidationError (int_from_float). That error surfaces as an APIConnectionError which the router treats as retryable, so it keeps re-calling the upstream (200 every time) until the upstream rate-limits and returns 429 to the caller. OpenAI specs this field as a float (see openai SDK UsageDuration.seconds), so widen seconds to float. With the parse succeeding there is no exception left to retry, which removes the loop. Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> * fix(deepseek): drop non-function tools before chat completions call (#30910) * fix(deepseek): drop non-function tools before chat completions call DeepSeek's /chat/completions only accepts tools of type "function". Requests bridged from /v1/responses can carry responses-API-native tool types, for example a Codex CLI tool typed "namespace", which DeepSeek rejects with "unknown variant 'namespace', expected 'function'" so the whole request fails (issue #30722). Filter unsupported tool types in the DeepSeek request transform so the function tools still go through; when nothing callable remains, also drop the now-dangling tool_choice and parallel_tool_calls Fixes #30722 * test(deepseek): cover async tool filtering and document tool_choice assumption Add an async_transform_request regression test so the sync and async tool filtering paths cannot silently diverge, and document in _drop_unsupported_tools that only non-function tools are dropped, so a function-named tool_choice always references a surviving tool * feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840) * feat(ui): surface team budget on key overview when key has no own budget (#30801) * feat(ui): surface team budget on key overview when key has no own budget * fix(ui): replace IIFE with derived variable and use find() for team budget display * fix(anthropic): emit replayable streaming thinking blocks (#31022) * feat(proxy): read cold-storage prompts back in the logs detail view (#30364) * feat(proxy): read cold-storage prompts back in the logs detail view When a deployment offloads prompts and responses to cold storage instead of Postgres, the spend-log row holds only "{}" placeholders plus a metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed nothing. The detail endpoint only read the placeholder columns and never fetched the object back. Resolve the payload per row based on actual content, not a config flag: if Postgres has content, return it; otherwise read the exact stored object key and fetch from the configured cold storage backend through ColdStorageHandler. Reading the persisted key is a single GET. The key embeds a microsecond timestamp that cannot be reconstructed from the millisecond-precision startTime column, and listing the day's prefix to match on request_id would be too expensive for this per-open path. Also teach the detail drawer's pretty-view parser to accept a bare messages array. The cold storage payload carries the prompt as a top-level messages list with no proxy_server_request, so without this the output rendered while the input stayed blank. ColdStorageHandler gains an optional injected logger so the resolver can be unit tested without monkeypatching. Postgres-stored prompts are unaffected: the fast path returns the existing columns and the request-body object still renders the same way. * Update litellm/proxy/spend_tracking/spend_management_endpoints.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure Add unit tests for ColdStorageHandler (injected logger, graceful None when no logger is configured, and resolution of a configured logger from the callback registry) and a regression test asserting a cold storage backend exception degrades to the Postgres values instead of surfacing a 500. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068) * fix(mavvrik): advance metricsMarker after upload + fix scheduler startup Two bugs fixed: 1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a successful GCS upload, so metricsMarker stayed at 0 and every daily run re-exported the same dates in an infinite catch-up loop. Fix: add _update_metrics_marker(date_epoch) called at the end of deliver() after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS file is already committed). A 410 raises consistent with the rest of the destination. 2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call has triggered lazy instantiation of MavvrikFocusLogger, so it found no logger instance and silently skipped registering the daily export job. Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call _init_custom_logger_compatible_class to force instantiation before the APScheduler job is registered. * fix(mavvrik): catch up from earliest window when metricsMarker=0 When the connector is freshly registered, metricsMarker=0 parses to None. The catch-up block was guarded by `if last_ingested and ...` which skipped it entirely for None, so only yesterday was exported instead of the full _MAX_CATCHUP_DAYS window. Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup). The existing > 7 day warning only fires for non-None markers that are old. * fix(mavvrik): use now as end_time for yesterday's export window LiteLLM_DailyUserSpend rows for a given date get their updated_at bumped by the spend flush job throughout the next morning. The core database query filters on updated_at, so capping end_time at midnight (yesterday + 1 day) missed any spend rows flushed after midnight. Fix: pass now (cron fire time) as end_time for the daily "yesterday" window so all fully-settled rows are captured regardless of when the flush job ran. Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per row in the exported FOCUS CSV. * fix(mavvrik): also use now as end_time for catch-up windows * fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class Calling it with only logging_integration raised TypeError at proxy startup because internal_usage_cache and llm_router have no defaults. Also fix test name to reflect the actual status code (5xx not 4xx) used in the mock. * ci: retrigger CI run * feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757) * Add optional `instruction` passthrough to the rerank API vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction` field (folded into the model's chat_template_kwargs and consumed by the chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently dropped it: RerankRequest / OptionalRerankParams had no such field, so the outgoing body was rebuilt without it. Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(), get_optional_rerank_params, and the hosted_vllm transformation into the request body, only when non-None. When callers omit it, model_dump(exclude_none) drops the field and the outgoing request is byte-for-byte unchanged — fully backward-compatible. (DeepInfra already forwards `instruction` via non_default_params; this formalizes the field in the shared types.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: thread `instruction` as a typed param + cover rerank_utils Per PR review (greptile P2 + codecov): - Make `instruction` a typed, named argument on the rerank provider interface instead of recovering it from the opaque `non_default_params` blob. Adds `instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params` and every provider override, and forwards it explicitly from `get_optional_rerank_params`. hosted_vllm now reads the named param directly. It is still also surfaced in `non_default_params` so providers that read it there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction` as a named param rather than leaving it in **kwargs. - Add get_optional_rerank_params unit tests (present + absent) to cover the previously-uncovered threading line flagged by codecov. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan rerank `instruction` through request guardrails The rerank guardrail translation (CohereRerankHandler.process_input_messages) only scanned `query`, so the newly added `instruction` field reached the backend model unscanned. Since instruction-aware rerankers (hosted vLLM / Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller could place content there to bypass configured rerank request guardrails. Generalize the handler to scan every user-controlled text field (`query` and `instruction`) in one apply_guardrail call and write each sanitized value back by index. Query-only requests are unchanged (single-element list at index 0); non-string fields are left untouched. Adds tests covering instruction scanning, PII masking write-back, and the non-string case. Addresses the Veria AI security review on PR #30757. * test: narrow Optional results before len() to satisfy basedpyright budget The lint gate (basedpyright delta-vs-base budget) flagged one new reportArgumentType: len(result.results) where results is List[RerankResponseResult] | None. Assert results is not None first to narrow the type before len()/indexing. * fix: read rerank `instruction` from kwargs to satisfy basedpyright budget The basedpyright delta-vs-base gate flagged one new reportArgumentType: the Router forwards rerank calls via an untyped `**kwargs` unpack (`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a typed named param on the public `rerank`/`arerank` entrypoints made pyright check that key against `str | None`, adding an error at router.py with no real safety gain. Read `instruction` from kwargs in `rerank` instead. It remains fully typed where it matters - threaded as a typed argument through `get_optional_rerank_params` and each provider's `map_cohere_rerank_params` (the original Greptile P2 ask). Whole-repo reportArgumentType is back to the base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(github_copilot): synthesize empty choices at the provider seam (#30929) Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with choices=[], either carrying Anthropic-native content blocks or, for the max_tokens=1 probe Claude Code sends, no content at all. github_copilot is dispatched through the OpenAI SDK handler, which calls convert_to_model_response_object directly and never invokes GithubCopilotConfig.transform_response, so the empty-choices guard there surfaced as a 500 Instead of synthesizing choices inside the shared convert_to_model_response_object (which would silently turn empty choices into a fabricated success for every provider), add a no-op transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig overrides it to synthesize choices from Anthropic-native content, reusing its existing parsing, and the OpenAI SDK handler routes its parsed response through the hook before generic conversion. The core utility keeps treating empty choices as an error for all other providers Fixes: https://github.com/BerriAI/litellm/issues/30927 Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> * fix(router): stop fallback lookups from mutating the router fallbacks config (#30624) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens * test: scope local cost map env var with monkeypatch to avoid test pollution * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764) * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold _mask_value did partial reveal by showing the first visible_prefix and last visible_suffix characters, but for a value whose length was at or below visible_prefix + visible_suffix (8 by default) it returned the value verbatim. A value of exactly 8 chars fell through the length guard and computed masked_length == 0, reconstructing the original string with no mask characters; anything shorter hit the early return. Either way short credentials were emitted in plaintext. mask_dict routes real secrets through this path, so an 8-char-or-shorter redis password, api key, or token could be written to logs and the UI unmasked. The sibling helper mask_sensitive_keys already guards this case; _mask_value now does the same by fully masking any value at or below the threshold. * fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers Fully masking short values is the right default for secret masking, but CooldownCache reuses the masker purely to truncate exception messages to the first 50 characters, and it relies on short messages being returned readable. Masking those blanked out short exception text and broke its tests. Add a mask_short_values flag (default True, secure) and have CooldownCache pass False so it keeps the truncation behavior, while every secret-masking caller still gets short values fully masked. * fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview MCPDebug uses the masker to preview auth tokens in debug headers and documents that values of 10 chars or fewer are shown unchanged so token types stay distinguishable. Pass mask_short_values=False so that diagnostic behavior is preserved while secret maskers keep masking short values. * fix(mcp_debug): mask short auth values in debug headers instead of echoing them Earlier this masker opted out of short-value masking to keep a token preview, but that echoes short authorization and token values verbatim in debug response headers, which is the same leak this change is meant to close. Auth material should never be emitted in full, so mask short values here too; the first/last character preview still applies to longer tokens. Only CooldownCache keeps the opt-out, since it truncates exception text rather than masking secrets. * test(mcp_debug): assert masked short value preserves length * refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917) Fireworks AI deprecated audio inference on 2026-06-10 (https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation). Live API testing confirms the endpoint is already non-functional: a valid Fireworks API key receives HTTP 401 "Unauthorized" from api.fireworks.ai/inference/v1/audio/transcriptions for every request, regardless of payload. The audio-prod.api.fireworks.ai host referenced in the test suite returns 401 for every path; the entire host is decommissioned. Remove the dead FireworksAIAudioTranscriptionConfig class and every reference to it across the codebase: - Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line config class that inherited from OpenAIWhisperAudioTranscriptionConfig) - Remove the Fireworks branch from ProviderConfigManager.get_provider_audio_transcription_config() in litellm/utils.py; update the stale comment in get_optional_params_transcription that referenced fireworks ai - Remove the FireworksAIAudioTranscriptionConfig entries from LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in litellm/_lazy_imports_registry.py - Remove the TYPE_CHECKING re-export in litellm/__init__.py - Remove the transcription branch in the fireworks_ai case of get_supported_openai_params() in litellm/litellm_core_utils/get_supported_openai_params.py - Remove the whisper-v3 and whisper-v3-turbo entries from model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json (both had mode: audio_transcription and zero-cost pricing) - Remove the TestFireworksAIAudioTranscription test class and its imports from tests/llm_translation/test_fireworks_ai_translation.py No other provider is affected. The openai_compatible_providers list, FireworksAIMixin, and the OpenAI Whisper transcription handler all stay because they are shared with other Fireworks endpoints and other providers. The provider_endpoints_support.json registry already had audio_transcriptions set to false for fireworks_ai. * feat: add darkbloom provider (#30876) * feat: add darkbloom provider * fix: document darkbloom provider endpoints * fix: address darkbloom review feedback * fix: update darkbloom tool metadata * fix: fail fast for non-Postgres database URLs (#30883) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging * Validate DIRECT_URL alongside DATABASE_URL startup guards * fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946) * fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608) * test(bedrock): mid-stream server errors trigger streaming fallback (#24608) * style(bedrock): black-format stream-error helper (#24608) * fix(mcp): re-land native tool preservation with typed annotations (#30645) * fix(mcp): preserve native tools in semantic filter hook with typed annotations * fix(mcp): tighten _is_mcp_tool Chat Completions shape check * fix(sambanova): return embeddings supported params instead of dropping them (#30937) * fix(router): send fallback metadata when streaming (#30914) When a streaming request triggers a fallback, there was previously no way to know it happened. This commit addresses this in a few ways: 1. The response now correctly populates the fallback headers (`x-litellm-attempted-fallbacks`) so callers know a fallback happened. 2. The correct model ID is passed in the streaming chunks. 3. A streaming chunk with the fallback error can be optionally sent back to the client (opt-in) by passing `include_fallback_errors: true` in the request. The format of the fallback errors while streaming is intentionally OpenAI compatible to not break existing libraries that parse these events. It was tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not delieved unexpectedly to callers by default. * fix(mistral): drop output-only reasoning fields from input messages (#30884) LiteLLM attaches reasoning_content and thinking_blocks to assistant responses. Replaying those assistant turns verbatim forwarded the fields back to Mistral, whose input schema forbids unknown keys, so the whole request failed with a 422 extra_forbidden and reasoning models became unusable across multiple turns. Strip both fields from assistant messages before the request is built, in a spot that runs ahead of the image/file branch so it applies on every path. Fixes #30835 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652) * fix(perplexity): bill search queries at the per-request price, not 1/1000 The fallback cost calculator divided search_context_cost_per_query by 1000, but that field stores the per-request price in USD: sonar is {low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published $5/$8/$12 per 1,000 requests expressed per request. The gemini cost calculator reads the same field per request with no division (its docstring calls it "the per-request cost"). The division understated search cost by 1000x on every Perplexity call that falls back to manual calculation (i.e. when the API does not return a pre-computed usage.cost). Use the value directly. Update the tests that had encoded the /1000 factor in their expectations, and drop an unused import flagged by ruff in the touched test file. * test(perplexity): update integration test search-cost expectations to per-request The integration tests still encoded the old /1000 search-cost factor, so they failed once the fallback calculator was corrected to bill search_context_cost_per_query per request. Update the four expected-cost computations (and the high-volume dollar-value comments) to match. * test(perplexity): drop unused mock imports flagged by ruff * fix: include model_access_groups when expanding all-team-models in get_team_models (#30622) * fix(fireworks_ai): return None for transcription in get_supported_openai_params Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is decommissioned. Without an explicit transcription branch, requests with request_type='transcription' fell through to the else and returned FireworksAIConfig chat-completion params. Return None instead to signal the provider does not support transcription. * fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting Without an operator gate, any authenticated caller could set include_fallback_errors=True, trigger a fallback, and read raw upstream exception messages from the x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event. Strip include_fallback_errors from request data in common_processing_pre_call_logic when expose_fallback_errors_to_caller is not set, so the router never builds the error list. Also gate _should_include_fallback_errors on the same setting as a secondary check for the streaming SSE injection path. * test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test The operator gate added in e7ff3e1 means include_fallback_errors is only honoured when general_settings.expose_fallback_errors_to_caller is True. Set that flag via monkeypatch in the test that exercises the emit path. * test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos test_convert_url called convert_url_to_base64 against a live picsum.photos URL and asserted nothing, so it added no real signal and broke CI whenever the host was unreachable (it was returning 522 and blocking this branch). Replace the live call with a mocked HTTP client and assert the produced base64 data URL, so the conversion path is exercised deterministically with no network dependency. This suite runs under VCR, which is why a transport level mock (respx) does not reliably intercept; mocking the client object itself is robust regardless. * fix(interactions): drop role from Interaction response to match Google spec Google removed the output-only role field from the Interaction schema (it now lives only on Turn), so the live OpenAPI compliance canary started failing with 'role' not in spec. Reconcile our generated types by removing role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse, stop stamping role=model in the responses-to-interactions transformation, and update the compliance and integration tests accordingly. Turn.role is kept since the spec still defines it. * fix: align all-team-models sentinel access * fix(router): forward include_fallback_errors through multi-hop fallbacks run_async_fallback received include_fallback_errors as an explicit named parameter, so it was bound out of **kwargs and never reached the nested async_function_with_fallbacks call. Multi-hop fallback chains (a fallback group that itself fails over) therefore stopped collecting fallback errors beyond the first hop when a caller opted in. Re-inject the flag into kwargs before the nested call so inner hops keep accumulating errors, which add_fallback_headers_to_response already merges across levels. * fix(router): stop fallback lookups from mutating the router fallbacks config get_fallback_model_group resolved a bare-string fallback by popping it out of the fallbacks list it was handed. That list is frequently the live router.fallbacks config, so a single lookup permanently removed the entry and the configured fallback stopped applying to later requests until restart. The pop also ran inside enumerate(), shifting indices and skipping an adjacent string fallback. Read the item instead of popping it, and add a regression test that fails on the old mutating behavior --------- Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * fix(sambanova): update pricing, deprecate retired models, and add missing models (#30016) * feat(bedrock): add amazon.titan-embed-g1-text-02 embedding model support - Add model to provider routing allowlist in embedding.py - Add request transformation using AmazonTitanG1Config - Add response transformation using AmazonTitanG1Config - Add pricing metadata to model_prices_and_context_window.json - Add unit tests for embedding and model info Fixes missing cost tracking reported in #29786 Related to VANDRANKI/litellm PR #29790 * style: fix syntax error, trailing whitespace and missing newline * style: apply black formatting to embedding.py * style: apply black formatting to test_bedrock_embedding.py * fix(sambanova): update pricing, fix context windows, add deprecation dates, and add missing models * fix(sambanova): sync model_prices_and_context_window_backup.json with primary * fix(sambanova): fix indentation on Meta-Llama-3.2-1B-Instruct deprecation_date * fix(bedrock): add amazon.titan-embed-g1-text-02 to unmapped model error message * style: apply black formatting to embedding.py * fix(sambanova): correct indentation on DeepSeek-V3.2 entry * fix(sambanova): replace gemma-3-12b-it with gemma-4-31B-it (verified pricing) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info (#30880) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info get_model_info rebuilt ModelInfo by copying a fixed allow-list of input/output_cost_per_token_above_<N>_tokens keys (128k/200k/272k/512k), so any other threshold a user registered was dropped before reaching _get_token_base_cost, which already reads an arbitrary threshold out of the key name. Custom tiers such as above_500k_tokens were silently ignored and billing fell back to the base per-token rate. Carry over any _above_<N>_tokens cost key present on the source cost-map entry that the fixed fields miss Fixes #30344 * test(cost): keep suite hermetic by popping the temp tiered-pricing model Wrap the regression body in try/finally so litellm.model_cost no longer leaks the litellm-test-non-standard-tier entry into later tests that iterate or reset the global cost map. Addresses Greptile review thread. * fix: resolve UP045 lint violations (Optional[X] -> X | None) Convert Optional[X] type annotations to X | None syntax across rerank transformations, spend tracking, and other modules to satisfy ruff strict gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: run black formatting on UP045-fixed files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused Optional imports after UP045 migration Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: black format cold_storage_handler.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): correct OSS staging branch name in guard-main-branch errors Co-authored-by: Cursor <cursoragent@cursor.com> * fix: strip trailing zeros from M/B spend formatter * fix: address focus and streaming edge cases * feat: add LAR-1 semantic routing strategy Optional router strategy that picks a deployment tier from request_kwargs.metadata.lar1 (confidence, evidence, time). Deployments are tagged with model_info.type (cloud-smart, cloud-fast, local, deep). Thresholds are configurable via routing_strategy_args. Includes 30 unit tests and an Ollama example config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mavvrik): advance metricsMarker on empty-content deliver When deliver() receives empty content (no spend data for a date), it now registers with Mavvrik and PATCHes the metricsMarker before returning instead of short-circuiting. Dates with zero spend no longer stall marker advancement, preventing unnecessary catch-up API calls on subsequent runs. * style: black format mavvrik_destination * fix: handle empty mavvrik exports and lar1 reset * test: add regression test for _reset_custom_routing_strategy * fix(test): mock async destination.deliver in mavvrik export window test * style: ruff format spend_management_endpoints after merge * fix(router): apply LAR-1 strategy atomically so invalid thresholds don't leave partial state apply_lar1_routing_strategy set router.routing_strategy to "lar1" before constructing LAR1RoutingStrategy, whose __init__ validates thresholds via _normalize_thresholds and raises on a misconfigured (out-of-order or out-of-range) set. On a live update_settings call with bad thresholds the router was left advertising routing_strategy="lar1" with no custom selector bound, while the previous strategy's selectors stayed registered. Build (and validate) the strategy before mutating any router state, so a threshold error leaves the router exactly as it was. Add a regression test that asserts a failed switch keeps the prior strategy intact. --------- Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com> Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: xbrxr03 <abrarhabib03@gmail.com> Co-authored-by: hayden <sktpghks138@gmail.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Wassim Badraoui <98709649+Wassbdr@users.noreply.github.com> Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> Co-authored-by: Neimar Avila <neimar.avila@gmail.com> Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> Co-authored-by: Jerry-Scintilla <jerrycaocao@126.com> Co-authored-by: AlexBGoode <me.at.forum@gmail.com> Co-authored-by: Carsten Boloz <cdboloz1@gmail.com> Co-authored-by: jesco <team@srswti.com> Co-authored-by: Praveen Ghuge <pghuge@digitalex.io> Co-authored-by: Jim Smith <j.h.smith@ieee.org> Co-authored-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Vedant Agarwal <43557509+Vedant-Agarwal@users.noreply.github.com> Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: bhumikadangayach <139267865+bhumikadangayach@users.noreply.github.com> Co-authored-by: Ewertonslv <ewertoncom297@gmail.com> Co-authored-by: carlsonchik <carlsonchik@users.noreply.github.com>	2026-06-26 09:17:44 -07:00
Yassin Kortam	29c254d3d3	fix(vertex): stop O(n^2) re-parse of accumulated Gemini stream JSON (#31297) ... handle_accumulated_json_chunk re-ran json.loads on the entire accumulated buffer after every fragment. For a streaming response fragmented across many chunks that is O(n^2) total work in a single GIL-holding C call, so a large enough Gemini response freezes the asyncio event loop for seconds, liveness probes time out, and the proxy pod gets killed and restarted. A complete Gemini stream value is a JSON object or array, so the buffer can only become parseable once its last non-whitespace byte can close one. Gate the json.loads attempt on that, which makes the common fragmented-response case parse roughly once instead of once per fragment. An 8MB payload drops from a 6.9s event-loop freeze to ~0.3s with identical parsed output. Resolves LIT-3503 Fixes #26181	2026-06-26 09:04:29 +03:00
Sameer Kankute	062d8ceeed	fix(vertex_ai): prevent stale Vertex bearer token causing /v1/messages 401 after token expiry (#31276) ... * fix(vertex_ai): prevent stale Vertex bearer token causing /v1/messages 401 after token expiry Router shallow-copies litellm_params so extra_headers is a shared reference. The chat/completions path was calling headers.update() on that shared dict, persisting the Vertex OAuth bearer. After ~1 h the token expired and /v1/messages kept reusing it (skipping refresh due to Authorization-already-present guard). - Build a new headers dict in the Claude partner-models completion path instead of mutating the shared extra_headers object. - Always call _ensure_access_token() in validate_anthropic_messages_environment regardless of an existing Authorization header; the token cache makes this cheap. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(vertex_ai): copy headers in validate_anthropic_messages_environment to prevent shared-dict mutation Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-25 21:01:02 -07:00
Sameer Kankute	7eacdd5258	chore: litellm oss staging 250626 (#31305) ... * fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes #30926) * style: format common_utils.py with black * fix(anthropic): extract api_base from litellm_params in batches/files validate_environment * fix(anthropic): scope Bearer key check to custom api_base endpoints * fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives The Anthropic streaming protocol emits `message_start.usage.output_tokens=1` as a placeholder cursor; the real cumulative output count only arrives in the final `message_delta` event. When a stream is cancelled before `message_delta` lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the `completion_tokens or token_counter(text=...)` fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed. Fix: track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content. Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct. Tests: - TestAnthropicCursorBug (6 cases) — pins the post-fix behavior - TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests still pass. * fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival Addresses both Greptile P2 threads on PR #30420: CLASS A — Anthropic-specific heuristic was applied globally ============================================================ The `completion_tokens == 1 and not saw_non_cursor_completion` reset lived in provider-neutral `streaming_chunk_builder_utils.py`. Any non-Anthropic provider that legitimately reports completion_tokens=1 in a single usage chunk (perfectly normal for short OpenAI / Bedrock / Vertex single-token replies with stream_options.include_usage=true) would have its value silently rewritten to 0 and re-billed via token_counter — producing a different number than what the provider actually charged. Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved from the first chunk's `_hidden_params` (the same field set by streaming_handler.py:722 on the live path). Unknown / missing provider is treated as non-Anthropic and skips the reset, so newer providers and custom plugins are also safe by default. CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies ============================================================ Previous condition was `usage_chunk_dict["completion_tokens"] > 1`, which never fires for an Anthropic stream where the model legitimately emits exactly one output token (e.g., "Yes."). Anthropic still sends message_start (output_tokens=1, the cursor) AND message_delta (output_tokens=1, the real value) — same value, but two distinct usage events. The old check couldn't tell that apart from a cancelled stream where only message_start landed. Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion` when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR (2) we've seen >=2 completion-bearing usage events (positive evidence that message_delta arrived). Cancelled cursor-only streams still have exactly one event and still hit the reset; cache chunks with completion_tokens=0 don't count toward the threshold. Tests ============================================================ - _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default "anthropic") so the gate is exercised by every existing test — none of them needed assertion changes besides the legitimate-single- token case, which now expects exactly 1 (was a fuzzy 0..3 range). - New: test_anthropic_cache_only_chunks_after_message_start_still_resets - New: test_non_anthropic_provider_completion_tokens_one_not_reset - New: test_unknown_provider_completion_tokens_one_not_reset 11/11 tests pass. * chore: add Co-authored-by trailer for attribution Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> * fix(anthropic): preserve messages cache usage * style(anthropic): format messages cache usage helper * fix(anthropic): accept integral float cache token counts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(anthropic): accept integral float cache token counts * test(anthropic): cover cache usage edge cases * fix(gemini): preserve thoughtSignature for server-side tool responses When Gemini API returns toolCall and toolResponse parts, they might have different thoughtSignatures. Previously, LiteLLM merged them into a single dict, overwriting the response's thoughtSignature with the call's. This fix extracts them separately and re-injects them correctly. TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * fix(gemini): address PR comments on thoughtSignature handling - Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature - Add missing assertions in existing tests - Add new unit tests for orphan-response signature handling TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * feat(mcp): include server alias and server_id in mcp_info response - Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint - Update rest_endpoints.py to surface alias from server config - Add test coverage in test_mcp_server.py and test_rest_endpoints.py Fixes #31015 * fix(proxy): reject non-finite spend via validate_finite_spend A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a shared finite-value guard, defined above the litellm.proxy.* imports to avoid the module-level cyclic-import warning. * fix(proxy): require admin for any /key/update spend, reject non-finite Gate the admin check on the presence of `spend` (not a value diff): the DB spend lags the live cross-pod counter, so an "unchanged" spend on the non-admin path let a key owner / team member overwrite the live counter below real usage. Also reject NaN/+-inf spend before the DB write. * fix(proxy): invalidate spend counter on /user/update spend change A direct spend change on /user/update wrote the DB row but left the warm cross-pod counter at the stale value, so enforcement kept reading the old spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB), and reject non-finite spend before the write. * fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244) The semantic cache's embedding model is a proxy Router alias whose AWS credentials (aws_role_name, aws_session_name) live only in the Router deployment's litellm_params. The sync embedding paths called litellm.embedding() directly, bypassing the Router, so they could neither resolve the alias nor assume the configured role; cross-account Bedrock semantic caching failed with "bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding during cache construction, while llm_router is still None. Fix A: make the sync paths mirror the already-correct async paths. A shared, dependency-injected helper (litellm/caching/_embedding_router.py) decides whether to route through llm_router.embedding(...) when the model is a Router deployment, else fall back to direct litellm.embedding(...). Redis and qdrant sync set_cache/get_cache now precompute the embedding and pass vector= to the backend, exactly as the async astore/acheck already do. Both async _get_async_embedding methods are unified onto the same helper and now forward the caller's full metadata instead of a hand-picked subset. Fix B (Redis only): defer redisvl index construction from __init__ into a lazy, memoized llmcache property, so the dimension-probe embedding fires on first cache use, after llm_router is wired. A failed build is not memoized, so a transient outage recovers on the next request. Known limitation: resolve_embedding_router gates on an exact model-name match (same as the shipped async path); wildcard/alias/team-public routes still fall back to direct embedding. Tracked as a follow-up. * fix(cache): harden embedding-router and shrink Any surface (review) Address review feedback on the semantic-cache aws-role fix (#28244): - resolve_embedding_router now skips deployment entries missing model_name instead of raising KeyError on a malformed model_list (Greptile P2); add a regression test that fails on the old direct-key access. - Replace the `**kwargs: Any` passthrough on the four cache _get_embedding / _get_async_embedding helpers with an explicit, typed `metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only ever consumed kwargs["metadata"], so this is behavior-preserving, makes the forwarded field obvious at the call site, and removes three bare-Any annotations (keeps the strict-rule ANN401 budget within ceiling). - Note in _build_llmcache that redisvl's dimension-probe embedding adds one extra billable embedding on the first cache request (Greptile P2). * fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist" Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash * test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression Adds the regression coverage codecov flagged on the two responses to completion bridge guard lines and the bedrock route-prefix helper. The handler tests drive both the sync and async fallback paths with litellm.completion and litellm.acompletion mocked, and assert the forwarded kwargs carry _skip_responses_api_bridge=True, so dropping either flag line fails the suite. The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids still do, exercising both branches of _model_has_route_prefix. Also aligns update_messages_with_model_file_ids model_id to Optional[str], matching its Responses API sibling, so the defensive model_info fallback no longer introduces a new reportArgumentType in completion(); the file-id lookup narrows model_id before the dict get * chore(ui): sync generated OpenAPI types for optional test_connection mode The test_model_connection mode body param default changed from chat to None so the mode is auto-detected from model capabilities, which makes the field optional in the proxy OpenAPI spec. Regenerate the committed schema so the dashboard types match: mode becomes optional and the description and default JSDoc follow the spec, keeping the Check UI API Types Sync gate green * refactor(bedrock): match all explicit route prefixes at path-segment boundary Migrates the remaining substring route checks to the existing _model_has_route_prefix helper so every explicit route token matches only as a leading path segment, consistent with get_bedrock_route and the mantle route. Covers _explicit_converse_route, _explicit_claude_platform_route, _explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route, _explicit_converse_like_route, _explicit_async_invoke_route and _explicit_openai_route. This also stops invoke/ from substring-matching async_invoke/. Route precedence and order are unchanged, and a note on the segment invariant is added to the helper docstring * test(bedrock): cover explicit route prefix segment matching Exercises all eight migrated _explicit_*_route helpers (converse, converse_like, invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each matches its token as a leading path segment and rejects the token glued to a preceding segment, so reverting any method to the old substring check fails the suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete improvement of the segment-boundary migration * test(proxy): assert negative spend is allowed (one-time grant use-case) Negative spend is intentionally permitted so admins can grant extra allowance for the current budget period only, without raising the recurring budget ceiling. Cover it explicitly in validate_finite_spend and via the /user/update invalidation test. * fix(google_genai): forward native generateContent top-level fields Google's native generateContent REST body carries safetySettings, toolConfig, cachedContent and labels at the top level as siblings of generationConfig. The proxy's :generateContent endpoint spread them into agenerate_content as loose kwargs and then dropped them, so callers had to wrap them in extra_body for them to take effect; safetySettings, for instance, was silently ignored The provider config now exposes the native top-level field names and setup_generate_content_call collects whichever are present, merging them into the outgoing request body through the existing extra_body merge so they reach Google verbatim. An explicit extra_body still wins on conflict. The sync generate_content_stream path now also forwards systemInstruction, matching the other three entry points Fixes #12671 Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK * fix(proxy): resolve env refs for DB-stored models * fix(proxy): restrict DB env ref resolution * fix(proxy): block team DB env ref resolution * fix(lint): resolve ANN401/UP045/C901 strict-gate violations - Replace Optional[X] with X | None (UP045) in 8 files - Replace Any return/param types with concrete types or object (ANN401) - Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix Users who pass a key already prefixed with "Bearer " get Authorization: Bearer. All other keys continue to use x-api-key, preserving backward compatibility with custom api_base endpoints that expect x-api-key rather than Authorization. Also consolidates get_auth_header to reuse _make_api_key_auth_header helper, eliminating the duplicated custom-endpoint routing logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base The backwards-compat change broke existing tests that verify the intentional Bearer-for-custom-base behavior (Fixes #30926). Restore original logic while keeping the _make_api_key_auth_header helper for code deduplication. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag Previously the auth-header switch from x-api-key to Authorization: Bearer applied unconditionally for non-sk-ant- keys on a custom api_base, silently breaking existing deployments that proxied to gateways expecting x-api-key. Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header, get_anthropic_headers, and get_auth_header. validate_environment reads it from litellm_params so callers can opt in per-model without any API surface change. Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981) DEL was the only Redis cache operation that skipped check_and_fix_namespace, so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM error on deployments with an ACL restricting DEL to the litellm:* pattern, and a silent no-op on all other deployments since the un-prefixed key was never stored. * style(anthropic): reformat common_utils.py with Black (--target-version py312) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve cache metadata and spend counters * style: apply ruff format to streaming_iterator.py * refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate Extract Anthropic message_start cursor reset into _reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter invalidation into _invalidate_user_spend_counter_if_changed, keeping both _calculate_usage_per_chunk and _update_single_user_helper under the max-complexity ceiling. Use builtin generics in the new signatures so no new UP006 violations are introduced. Behavior unchanged. --------- Co-authored-by: rupak-eng <rupakji99@gmail.com> Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com> Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com> Co-authored-by: Andrii Butko <booandrew23@gmail.com> Co-authored-by: Kent <kingdooo@gmail.com> Co-authored-by: kunal2002 <k.nayyar2002@gmail.com> Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com> Co-authored-by: jesco-absolut <team@srswti.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Matt Hill <mhill@dataminr.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-25 21:00:28 -07:00
Mateo Wang	6e3540856c	fix(vertex): preserve Gemini Embedding 2 usageMetadata for cost tracking (#31354) ... * fix(vertex): preserve Gemini Embedding 2 usageMetadata for cost tracking * style(vertex): apply ruff format to batch_embed_content_transformation * fix(vertex): bill files/ image refs in Gemini embedContent at per-image rate Resolved files/... references whose mime type is an image were not detected by _is_image_element, so image_count stayed 0 and generic_cost_per_token fell back to the text token rate instead of input_cost_per_image. Thread the resolved_files mapping into the usage builder so resolved image references are counted and billed per image. Also modernize the _flatten_input return annotation to satisfy the ruff UP006 strict gate. * fix(vertex): bill Gemini embedding audio per-second and stop video+audio double-billing Audio-only embedContent responses set audio_tokens, but generic_cost_per_token only charges audio via input_cost_per_audio_token. gemini-embedding-2 prices audio via input_cost_per_audio_per_second, so spend stayed at $0. Plumb a new audio_length_seconds field through PromptTokensDetailsWrapper, parse it in _parse_prompt_tokens_details, and bill it from _calculate_input_cost. The vertex embedding transformation derives audio_length_seconds from audio_tokens using the documented 32 tokens/sec Gemini rate. The 1-token text floor that protects video billing only fired when no other modality was billable, but audio presence flipped that flag, leaving text_tokens at zero for video+audio responses. generic_cost_per_token then rewrote text_tokens to prompt_tokens minus audio_tokens (the video token count), charging video tokens as text on top of the per-second video cost. The rewrite trigger is text_tokens == 0 and image_count == 0; align the floor with that trigger and ignore audio_tokens. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-25 19:53:16 -07:00
Mateo Wang	e0e920d80e	feat(mistral): support Mistral OCR 4 (mistral-ocr-4-0) (#31353) ... * feat(mistral): support Mistral OCR 4 (mistral-ocr-4-0) Add the mistral/mistral-ocr-4-0 model to the cost map and reprice mistral/mistral-ocr-latest, which now resolves to OCR 4 server-side, at $4 / 1000 pages. Add the include_blocks param so callers can request OCR 4's paragraph-level bounding boxes and typed content blocks. OCR 4's new per-page response fields (blocks, confidence_scores, tables, hyperlinks, header, footer) already pass through transform_ocr_response via the extra="allow" config on OCRPage; add a regression test pinning that behavior alongside cost and param coverage. * fix(mistral): revert unverified OCR 4 annotation_cost_per_page bump Mistral's published OCR 4 pricing lists $4/1000 pages for the API and no separate annotation rate; the $5/1000 figure is the distinct Document AI (Studio) tier. The earlier 0.003 -> 0.005 bump on annotation_cost_per_page had no cited source, and ocr_cost() never reads that field (it bills off ocr_cost_per_page), so the value is documentation-only. Revert annotation_cost_per_page to the existing 0.003 convention for both mistral-ocr-latest and mistral-ocr-4-0, keeping only the verified, tested ocr_cost_per_page: 0.004 change. * fix(mistral): set OCR 4 annotation_cost_per_page to verified $5/1000 rate Verified against Mistral's authoritative sources: the pricing page, the OCR 4 announcement, and the ocr-4-0 model card all list OCR 4 at $4/1000 pages for basic OCR and $5/1000 for annotated pages (Document AI). The $5/1000 figure is the annotated-pages rate, which is exactly what annotation_cost_per_page encodes, mirroring the original OCR entry's 0.001 basic / 0.003 annotated split. Restore annotation_cost_per_page to 0.005 for mistral-ocr-latest and mistral-ocr-4-0; the earlier revert to 0.003 was based on an incomplete reading that treated Document AI as a separate product. ocr_cost_per_page stays 0.004, which is the value billed by ocr_cost(). * fix(mistral-rust): include_blocks in Rust OCR supported params --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>	2026-06-25 16:42:37 -07:00
Mateo Wang	b7f28bd89f	feat(aiml): add openai/gpt-image-2 image model (#31323) ... * feat(aiml): add openai/gpt-image-2 image model Adds aiml/openai/gpt-image-2 to the cost map and teaches AimlImageGenerationConfig to route OpenAI-style image models through the upstream OpenAI request schema instead of the AI/ML flux schema. Without this, size, n, and response_format would be remapped to image_size/num_images/output_format, which the gpt-image-2 endpoint on api.aimlapi.com does not accept. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * chore(aiml): note gpt-image-2 flat-rate pricing basis; apply ruff format Documents in the cost-map notes that output_cost_per_image is AI/ML's published medium-quality rate, billed as a flat per-image price like the other aiml image entries. Reformats the touched files under the repo's ruff formatter (migrated from black in #31317). * fix(aiml): drop /v1/images/edits from gpt-image-2 supported_endpoints LiteLLM only implements an image generation transformer for AIML, so listing /v1/images/edits overclaimed support. Align with every other aiml image entry, which lists only /v1/images/generations. * style(aiml): format transformation.py at line-length 88 The repo formats litellm/ with ruff at line-length 88 (Makefile/CI call sites), while ruff.toml's global 120 only governs E501/import sorting. Reformat the transformer to 88 so make format-check / CI lint pass, and restore the test files to their original layout since tests/ is not part of the auto-formatted tree. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-06-25 16:41:43 -07:00
mubashir1osmani	56825926af	fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads (#31036) ... * fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker because the request body was buffered and multiplied 2-3x in size. The create-file path is now streaming end-to-end: transform_create_file_request returns a ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks (Content-Range, 308 between chunks) so the transformed payload is never held in full. The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of reading the whole body, and batch rate limiting counts tokens and models in a single streaming pass. Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is intentionally not read. Also removes the unreachable VertexAIFilesHandler create path and everything only it kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced. * fix(batches): return original JSONL on unparseable row to avoid silent batch truncation The streaming rewrite of replace_model_in_jsonl accumulated physical lines and skipped a row on JSONDecodeError to support multi-line objects, but a genuinely malformed or truncated row never completes: it poisons the buffer, swallows every following row, and the function still returned the partial rewrite (the rows before the bad one, already model-rewritten) as if the batch were complete. That turned the pre-rewrite behavior of returning the original file unchanged (so the provider rejects the bad batch loudly) into a silent partial submission. Restore the original-content fallback: when an unparseable remainder is left after the loop, return the original file_content (rewinding a consumed seekable source) instead of the truncated output. The multi-line happy path is unchanged. * test(batches): mock resumable GCS upload in vertex batch prediction test The vertex batch file-create path now streams to a GCS resumable session via _aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the existing test's post mock no longer intercepted the upload and a real request hit GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the resumable protocol itself is covered in test_vertex_ai_files_streaming.py. * fix(batches): resilient per-row token accounting; no hard-block on count failure The batch input-file pass iterated a generator whose json.loads raised on a malformed line; the outer except caught it and stopped the loop, so any body.model on rows after a bad line was never collected and the model allowlist check ran against a partial set. It also hard-blocked the batch with a 400 whenever token counting raised, a backwards-incompatible change from the prior swallow-and-proceed behavior that breaks legitimate rows the token counter cannot measure (e.g. some multimodal content). Iterate the JSONL line-by-line and account each row independently. A malformed line is skipped (its request cannot run upstream anyway) and a row the counter cannot measure falls back to a conservative size-based estimate. The loop never aborts, so the allowlist check always sees every parseable model, and the token total is never zeroed, so a crafted uncountable row still cannot evade the TPM limit, without hard-rejecting a legitimate batch. * perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types Three review follow-ups on the resumable batch upload: - _aresumable_chunked_upload pulled chunks from a synchronous generator that runs the per-row transform inline on the event loop thread, blocking other requests between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread. - _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly chunk-aligned upload finalizes on its last data chunk instead of an extra zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request. - valid_content_type now accepts the MIME types clients label .jsonl batch uploads with (text/plain, application/json, ndjson, ...), so such a batch file no longer silently bypasses the streaming path into the buffered media upload. * fix(vertex/files): keep legacy bucket_name as GCS bucket fallback The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present * style: sort imports in llm_http_handler to satisfy I001 budget --------- Co-authored-by: Yuneng Jiang <yuneng@berri.ai>	2026-06-24 13:19:57 -07:00
Sameer Kankute	8bca05d311	fix(anthropic): sanitize tool_use ids on native /v1/messages path (#31094)	2026-06-24 07:57:46 -07:00
Krrish Dholakia	d0706c17fe	fix(anthropic): drop unsupported speed param with drop_params (#31152) ... * fix(anthropic): drop unsupported speed param with drop_params Anthropic fast mode (speed) is Opus 4.6/4.7/4.8 on the direct API only. Strip speed when the model map lacks supports_speed and drop_params is set, for both chat completions and /v1/messages passthrough. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): allow supports_speed in model map schema The new supports_speed flag on Opus entries must pass JSON schema validation in test_aaamodel_prices_and_context_window_json_is_valid. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(review): raise on unsupported speed without drop_params Passthrough /v1/messages now raises UnsupportedParamsError when speed is unsupported and drop_params is false. Emit drop warning from map_openai_params when speed is silently skipped. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): gate speed param by routed provider, not just model id Vertex, Azure, and Bedrock reuse the shared Anthropic transform and strip their provider prefix first, so a bare `claude-opus-4-8` resolved to the direct-API model-map entry (`supports_speed: true`) and forwarded `speed` upstream, producing the same 400 that drop_params is meant to prevent. Gate fast mode on `custom_llm_provider == "anthropic"` so it stays on the direct Anthropic API across both the chat completions and `/v1/messages` passthrough paths, and collapse the duplicated drop/raise logic in map_openai_params into the shared `_maybe_drop_speed_param` helper. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-23 22:22:49 -07:00
Shivam Rawat	8dfe702e5b	fix(bedrock-mantle): honor api_base for VPC endpoint routing on bedrock/mantle/... (#31141) ... The bedrock/mantle chat and messages paths hardcoded the public Mantle host and ignored api_base, so private VPC/VPCE/GovCloud endpoints could not be used. Route URL construction through a shared helper that prefers api_base and aws_bedrock_runtime_endpoint before falling back to the regional public host. Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-23 17:31:59 -07:00
Mateo Wang	f12c9bec48	fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119) ... request_timeout was shadowed by router_settings.timeout: Router stored a single slot via `self.timeout = timeout or litellm.request_timeout`, so when a router timeout was set the configured request_timeout was never used. Provider calls with no per-model timeout (Bedrock especially) then fell back to the hardcoded 600s httpx client default. Mirrors PR #25701 and completes it on top of the CompletionTimeout work already on this branch. - Router: add an independent self.request_timeout and prefer it over router_settings.timeout in both _get_non_stream_timeout and _get_stream_timeout - http_handler: cached default clients now fall back to request_timeout instead of a hardcoded 600s - Replace the brittle `== 6000` default-detection heuristic with a single get_configured_request_timeout() resolver backed by an explicit request_timeout_explicitly_set sentinel (set from REQUEST_TIMEOUT env and litellm_settings), keeping the value-differs fallback for SDK assignment. This also fixes an explicit request_timeout of 6000 being coerced to 600 - CompletionTimeout no longer second-guesses the package default; the caller passes the explicitly-configured value or None Regression for LIT-2369.	2026-06-23 14:22:54 -07:00
tin-berri	9c014716ec	fix(search): block server credential leak to caller-supplied api_base (#30682) ... Search providers resolved the server-configured API key (e.g. get_secret_str("SERPER_API_KEY")) in validate_environment whenever the caller omitted api_key, while get_complete_url independently honored a caller-supplied api_base. A caller who passes their own api_base and no api_key therefore made the proxy send the operator's provider key to a host they control; POST /search_tools/test_connection forwards request-body api_base/api_key straight into asearch, so any authenticated user could exfiltrate the server's search credentials. Add a shared host-aware fallback in BaseSearchConfig.resolve_server_api_key that only applies a server-managed secret when the caller-supplied api_base is absent or resolves to a trusted host (the provider default or the operator's own *_API_BASE env override); otherwise it refuses and asks for an explicit api_key. The guard only triggers when a server secret actually exists, so keyless and self-hosted providers (searxng, you.com free tier) keep working. Every provider that carries a server secret is migrated to the helper; dataforseo reuses the same guard for its login:password basic-auth credentials. This changes behavior for callers that previously passed a per-request api_base while relying on a server-configured key: they must now pass an explicit api_key, or the operator must configure the base via the provider's *_API_BASE env var (which stays trusted).	2026-06-23 13:01:21 -07:00
Mateo Wang	1be957da17	fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint (#31053) ... * fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint * fix(cloudflare): guard missing account id and migrate legacy /ai/run base Centralize the OpenAI-compatible api_base default in get_complete_url so it is built in one place instead of being duplicated in main.py. When neither api_base nor CLOUDFLARE_ACCOUNT_ID is set the call now fails fast with a clear error rather than sending a request to a URL containing the literal 'None'. An api_base still pinned to the legacy Workers AI '/ai/run' path is rewritten to the '/ai/v1' OpenAI-compatible endpoint with a deprecation warning, so users who hardcoded the previous default migrate gracefully instead of hitting a silently broken '/ai/run/chat/completions' URL. * fix(cloudflare): treat empty api_base as unset when resolving URL * fix(cloudflare): treat empty CLOUDFLARE_ACCOUNT_ID as unset An empty or whitespace-only CLOUDFLARE_ACCOUNT_ID slipped past the None guard and built .../accounts//ai/v1, producing the same confusing 404 the PR set out to prevent. Normalize the secret with normalize_nonempty_secret_str so blank values raise the explicit missing-account-id error instead.	2026-06-23 12:09:11 -07:00
Krrish Dholakia	c8a9618afd	feat: add opensandbox sandbox provider (#31024) ... * feat: add opensandbox sandbox provider * fix: harden opensandbox sandbox startup * fix: address opensandbox review feedback * fix: address opensandbox sandbox review feedback * fix: address sandbox parser nits * fix(ci): clear opensandbox gates * fix(review): require opensandbox api base * chore(ci): rerun pass-through check	2026-06-23 09:05:13 -07:00
Sameer Kankute	e73cbfb026	fix(realtime): post-tool-call function_response id omission (#30446)	2026-06-23 08:40:52 -07:00
Sameer Kankute	80c5a84871	chore: litellm oss staging (#30968) ... * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens * test: scope local cost map env var with monkeypatch to avoid test pollution * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764) * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold _mask_value did partial reveal by showing the first visible_prefix and last visible_suffix characters, but for a value whose length was at or below visible_prefix + visible_suffix (8 by default) it returned the value verbatim. A value of exactly 8 chars fell through the length guard and computed masked_length == 0, reconstructing the original string with no mask characters; anything shorter hit the early return. Either way short credentials were emitted in plaintext. mask_dict routes real secrets through this path, so an 8-char-or-shorter redis password, api key, or token could be written to logs and the UI unmasked. The sibling helper mask_sensitive_keys already guards this case; _mask_value now does the same by fully masking any value at or below the threshold. * fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers Fully masking short values is the right default for secret masking, but CooldownCache reuses the masker purely to truncate exception messages to the first 50 characters, and it relies on short messages being returned readable. Masking those blanked out short exception text and broke its tests. Add a mask_short_values flag (default True, secure) and have CooldownCache pass False so it keeps the truncation behavior, while every secret-masking caller still gets short values fully masked. * fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview MCPDebug uses the masker to preview auth tokens in debug headers and documents that values of 10 chars or fewer are shown unchanged so token types stay distinguishable. Pass mask_short_values=False so that diagnostic behavior is preserved while secret maskers keep masking short values. * fix(mcp_debug): mask short auth values in debug headers instead of echoing them Earlier this masker opted out of short-value masking to keep a token preview, but that echoes short authorization and token values verbatim in debug response headers, which is the same leak this change is meant to close. Auth material should never be emitted in full, so mask short values here too; the first/last character preview still applies to longer tokens. Only CooldownCache keeps the opt-out, since it truncates exception text rather than masking secrets. * test(mcp_debug): assert masked short value preserves length * refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917) Fireworks AI deprecated audio inference on 2026-06-10 (https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation). Live API testing confirms the endpoint is already non-functional: a valid Fireworks API key receives HTTP 401 "Unauthorized" from api.fireworks.ai/inference/v1/audio/transcriptions for every request, regardless of payload. The audio-prod.api.fireworks.ai host referenced in the test suite returns 401 for every path; the entire host is decommissioned. Remove the dead FireworksAIAudioTranscriptionConfig class and every reference to it across the codebase: - Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line config class that inherited from OpenAIWhisperAudioTranscriptionConfig) - Remove the Fireworks branch from ProviderConfigManager.get_provider_audio_transcription_config() in litellm/utils.py; update the stale comment in get_optional_params_transcription that referenced fireworks ai - Remove the FireworksAIAudioTranscriptionConfig entries from LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in litellm/_lazy_imports_registry.py - Remove the TYPE_CHECKING re-export in litellm/__init__.py - Remove the transcription branch in the fireworks_ai case of get_supported_openai_params() in litellm/litellm_core_utils/get_supported_openai_params.py - Remove the whisper-v3 and whisper-v3-turbo entries from model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json (both had mode: audio_transcription and zero-cost pricing) - Remove the TestFireworksAIAudioTranscription test class and its imports from tests/llm_translation/test_fireworks_ai_translation.py No other provider is affected. The openai_compatible_providers list, FireworksAIMixin, and the OpenAI Whisper transcription handler all stay because they are shared with other Fireworks endpoints and other providers. The provider_endpoints_support.json registry already had audio_transcriptions set to false for fireworks_ai. * feat: add darkbloom provider (#30876) * feat: add darkbloom provider * fix: document darkbloom provider endpoints * fix: address darkbloom review feedback * fix: update darkbloom tool metadata * fix: fail fast for non-Postgres database URLs (#30883) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging * Validate DIRECT_URL alongside DATABASE_URL startup guards * fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946) * fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608) * test(bedrock): mid-stream server errors trigger streaming fallback (#24608) * style(bedrock): black-format stream-error helper (#24608) * fix(mcp): re-land native tool preservation with typed annotations (#30645) * fix(mcp): preserve native tools in semantic filter hook with typed annotations * fix(mcp): tighten _is_mcp_tool Chat Completions shape check * fix(sambanova): return embeddings supported params instead of dropping them (#30937) * fix(router): send fallback metadata when streaming (#30914) When a streaming request triggers a fallback, there was previously no way to know it happened. This commit addresses this in a few ways: 1. The response now correctly populates the fallback headers (`x-litellm-attempted-fallbacks`) so callers know a fallback happened. 2. The correct model ID is passed in the streaming chunks. 3. A streaming chunk with the fallback error can be optionally sent back to the client (opt-in) by passing `include_fallback_errors: true` in the request. The format of the fallback errors while streaming is intentionally OpenAI compatible to not break existing libraries that parse these events. It was tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not delieved unexpectedly to callers by default. * fix(mistral): drop output-only reasoning fields from input messages (#30884) LiteLLM attaches reasoning_content and thinking_blocks to assistant responses. Replaying those assistant turns verbatim forwarded the fields back to Mistral, whose input schema forbids unknown keys, so the whole request failed with a 422 extra_forbidden and reasoning models became unusable across multiple turns. Strip both fields from assistant messages before the request is built, in a spot that runs ahead of the image/file branch so it applies on every path. Fixes #30835 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652) * fix(perplexity): bill search queries at the per-request price, not 1/1000 The fallback cost calculator divided search_context_cost_per_query by 1000, but that field stores the per-request price in USD: sonar is {low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published $5/$8/$12 per 1,000 requests expressed per request. The gemini cost calculator reads the same field per request with no division (its docstring calls it "the per-request cost"). The division understated search cost by 1000x on every Perplexity call that falls back to manual calculation (i.e. when the API does not return a pre-computed usage.cost). Use the value directly. Update the tests that had encoded the /1000 factor in their expectations, and drop an unused import flagged by ruff in the touched test file. * test(perplexity): update integration test search-cost expectations to per-request The integration tests still encoded the old /1000 search-cost factor, so they failed once the fallback calculator was corrected to bill search_context_cost_per_query per request. Update the four expected-cost computations (and the high-volume dollar-value comments) to match. * test(perplexity): drop unused mock imports flagged by ruff * fix: include model_access_groups when expanding all-team-models in get_team_models (#30622) * fix(fireworks_ai): return None for transcription in get_supported_openai_params Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is decommissioned. Without an explicit transcription branch, requests with request_type='transcription' fell through to the else and returned FireworksAIConfig chat-completion params. Return None instead to signal the provider does not support transcription. * fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting Without an operator gate, any authenticated caller could set include_fallback_errors=True, trigger a fallback, and read raw upstream exception messages from the x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event. Strip include_fallback_errors from request data in common_processing_pre_call_logic when expose_fallback_errors_to_caller is not set, so the router never builds the error list. Also gate _should_include_fallback_errors on the same setting as a secondary check for the streaming SSE injection path. * test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test The operator gate added in e7ff3e1 means include_fallback_errors is only honoured when general_settings.expose_fallback_errors_to_caller is True. Set that flag via monkeypatch in the test that exercises the emit path. * test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos test_convert_url called convert_url_to_base64 against a live picsum.photos URL and asserted nothing, so it added no real signal and broke CI whenever the host was unreachable (it was returning 522 and blocking this branch). Replace the live call with a mocked HTTP client and assert the produced base64 data URL, so the conversion path is exercised deterministically with no network dependency. This suite runs under VCR, which is why a transport level mock (respx) does not reliably intercept; mocking the client object itself is robust regardless. * fix(interactions): drop role from Interaction response to match Google spec Google removed the output-only role field from the Interaction schema (it now lives only on Turn), so the live OpenAPI compliance canary started failing with 'role' not in spec. Reconcile our generated types by removing role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse, stop stamping role=model in the responses-to-interactions transformation, and update the compliance and integration tests accordingly. Turn.role is kept since the spec still defines it. * fix: align all-team-models sentinel access * fix(router): forward include_fallback_errors through multi-hop fallbacks run_async_fallback received include_fallback_errors as an explicit named parameter, so it was bound out of **kwargs and never reached the nested async_function_with_fallbacks call. Multi-hop fallback chains (a fallback group that itself fails over) therefore stopped collecting fallback errors beyond the first hop when a caller opted in. Re-inject the flag into kwargs before the nested call so inner hops keep accumulating errors, which add_fallback_headers_to_response already merges across levels. --------- Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-23 07:31:44 -07:00
yucheng-berri	4ef7d0815b	fix(bedrock): only expand config-sourced AWS credential references (#30867) ... AWS auth parameters in the Bedrock and SageMaker path could be expanded against the process environment when credentials were built. Config-sourced references are already expanded at load time, so restrict expansion to that path: a reference still present at request time is treated as caller-supplied input and is left as-is, and the web-identity helper rejects environment-variable references before resolving the token. Also rework the ambient AWS_* fallback as a single pass that pairs each value with its own env-var name, fixing a latent index misalignment that left AWS_EXTERNAL_ID unresolved. Adds regression tests covering the resolution behavior.	2026-06-22 15:28:43 -07:00
yucheng-berri	1667b8f740	fix: reject model_list in proxy body and gate advisor client credentials (#30585) ... * fix: validate proxy request body and nested fields Ensure caller-supplied request fields cannot override server-side deployment configuration, and apply request-body validation consistently to nested structures. Adjusts router kwarg handling and client-side credential handling for base-url overrides * test: cover router strip ordering and advisor clientside credential gate * fix: clear deployment credentials on client base-url override When a request overrides api_base/base_url, recompute the deployment's litellm_params (clearing the deployment's own api_key) and drop the cached client built for the original endpoint, so the deployment credential is not reused for the client-supplied endpoint. Adds regression tests that assert the credentials actually forwarded to litellm.completion/acompletion. * fix(proxy): require api_key alongside api_base override A request that overrides api_base/base_url but supplies no api_key still left the proxy carrying a server credential: once the override clears the banned-param opt-in, the provider re-resolves a key from the environment (api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in main.py) and forwards it to the caller-controlled URL. Popping the deployment api_key only changed which server key leaked. Gate is_request_body_safe so a permitted api_base/base_url override must also carry a non-empty caller api_key; reject otherwise. The env resolution in main.py is left as the provider boundary. * fix(proxy): extend request-body banlist with five additional credential and session targeting fields Yuneng's review found five deployment-owned request-body params still missing from the denylist and the router strip set. Each lets a caller reach the operator's provider credentials or retarget the outbound request: aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region retarget the OCI request, litellm_credential_name selects any server-loaded credential by name with no ownership check, and runtimeSessionId resumes a Bedrock AgentCore runtime session (AWS does not enforce session-to-user mapping, so this is a cross-tenant session-resume vector). Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to _DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and SDK direct calls are unaffected: the banlist gates the request body only, and the router strip drops caller kwargs, never deployment["litellm_params"]. * test: rename arbitrary canary values in security tests to neutral placeholders * fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS (litellm_embedding_config, extra_body) for the banned-param check but not for the api_key co-presence check, so a base override smuggled into one of those nested dicts cleared the client-side-credentials opt-in without a paired api_key and let the provider re-resolve a server credential from the environment. Run _check_base_override_has_api_key on each nested config dict too, so the requirement applies wherever a base override is permitted. P1-B: the deployment-owned credential strip in the Router runs unconditionally on every _completion/_acompletion, which is security-correct but silently drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a single warning (key names only, never values) when the strip removes a non-empty value, so the backwards-incompatible behavior is visible without gating the strip on a context flag that does not exist. * fix(proxy): apply api_key co-presence to tool-entry base override is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and tools[]); the previous commit extended the api_key co-presence rule to root and nested config dicts but not to tool entries. With allow_client_side_credentials enabled, a tool entry carrying api_base/base_url and no paired api_key cleared the gate, letting a provider interceptor fall back to a server-side credential for a caller-controlled URL. Add the same _check_base_override_has_api_key call to each tool dict and its nested function dict, mirroring the symmetry already applied to the nested config keys. The rule is unchanged: api_key must live in the same dict as the base override it accompanies. * test(proxy/auth): require paired api_key under extra_body opt-in * fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running * fix(advisor): narrow proxy-import guard to ImportError-family * fix(router): gate api_key clear on base override behind litellm.proxy_is_running * test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture * style: use built-in generics in PR-added type annotations * revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate * revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes * style: black-format advisor orchestration test	2026-06-22 11:34:20 -07:00
Mateo Wang	9f97111edd	feat(fireworks_ai): sync chat completions endpoint with full API surface (#30885) ... * feat(fireworks_ai): sync chat completions endpoint with full API surface Add 23 missing request parameters to get_supported_openai_params(): seed, top_logprobs, min_p, typical_p, repetition_penalty, mirostat_target, mirostat_lr, logit_bias, echo, echo_last, ignore_eos, prompt_cache_key, prompt_cache_isolation_key, raw_output, perf_metrics_in_response, return_token_ids, safe_tokenization, service_tier, metadata, speculation, prediction, stream_options, sampling_mask. Also add reasoning_history gated on supports_reasoning. Fix prompt_truncate_length to prompt_truncate_len to match the actual API parameter name. The old name was never in DEFAULT_CHAT_COMPLETION_PARAM_VALUES, so it always went to extra_body and was rejected by Fireworks; it never actually worked. Normalize reasoning_effort boolean values to strings: True becomes "medium", False becomes "none". The Fireworks OpenAPI schema documents these as accepted types, but the server rejects non-string values with HTTP 400 in practice. Integers pass through as-is since the server is expected to validate them. Auto-inject stream_options.include_usage=true when stream=true and the user has not explicitly set stream_options. Without this, Fireworks returns null usage in all streaming chunks, which is inconsistent with the non-streaming behavior where usage is always present. If the user explicitly sets include_usage=false, it is preserved. Capture Fireworks-specific response fields in transform_response(): perf_metrics, prompt_token_ids, raw_output, and token_ids are now extracted from the response and stored in response._hidden_params (fireworks_perf_metrics, fireworks_prompt_token_ids, fireworks_raw_outputs, fireworks_token_ids) so they are accessible to logging, the proxy, and downstream consumers when the corresponding request parameters are enabled. Remove deprecated document inlining logic. Document inlining was deprecated on 2025-06-30 (https://docs.fireworks.ai/updates/changelog#-document-inlining-deprecation). This removes _add_transform_inline_image_block(), the file-to-image_url migration in _transform_messages_helper(), and the disable_add_transform_inline_image_block lookup. Current models that support image input do so natively as VLMs. cache_control, provider_specific_fields, and thinking_blocks stripping is retained. Update get_provider_info() to look up supports_vision and supports_pdf_input from the model cost map instead of hardcoding both to True (which was based on the now-deprecated document inlining). supports_prompt_caching remains True. API docs: https://docs.fireworks.ai/api-reference/post-chatcompletions Reasoning guide: https://docs.fireworks.ai/guides/reasoning Prompt caching: https://docs.fireworks.ai/guides/prompt-caching * fix fireworks chat api surface gaps * Scope Fireworks thinking param to reasoning models * style: fix black formatting * fix(test): update minimax-m3 expected_vision to True * test: cover non-dict content branch in transform_messages_helper * fix(fireworks_ai): remove metadata from supported params to prevent internal metadata disclosure * test(fireworks_ai): replace stale document-inlining capability test The CircleCI-only litellm_utils_tests suite still asserted the old behavior where document inlining made every Fireworks model report supports_pdf_input and supports_vision as True. That premise was removed in this change, so the test now reflects cost-map-driven capabilities: unmapped models no longer advertise vision/PDF support while mapped VLMs like minimax-m3 still do. * test(fireworks_ai): add end-to-end regression for native OpenAI params The existing coverage for the newly supported OpenAI-native params asserted list membership in get_supported_openai_params or called map_openai_params with a hand-built dict, both of which bypass the get_optional_params gate (DEFAULT_CHAT_COMPLETION_PARAM_VALUES). That gate is what previously raised UnsupportedParamsError for seed, top_logprobs, logit_bias, prompt_cache_key, service_tier and prediction when drop_params=False. Assert the full path so a revert of the supported-params additions fails the test instead of passing a shallow membership check. * test(fireworks_ai): fix test isolation in vision/inlining tests Use monkeypatch in test_fireworks_ai_vision_capability_from_cost_map so the LITELLM_LOCAL_MODEL_COST_MAP env var and litellm.model_cost are restored after the test instead of leaking global state into the rest of the process. Switch the document-inlining integration tests off deepseek-v3p1, whose supports_vision is null in the cost map, onto minimax-m3 which is explicitly supports_vision:true. The pass-through assertions no longer depend on a model incidentally not being marked non-vision. * fix(fireworks_ai): gate image rejection on exact vision capability The image_url rejection read supports_vision via _get_model_cost_capability, which falls back to hyphen-boundary substring matching when no exact cost-map entry exists. A custom or fine-tuned model id that merely contains a known non-vision model's short name (e.g. an id ending in -glm-5p2) inherited that entry's supports_vision:false and hard-failed valid image_url blocks on a vision-capable deployment. Split the exact candidate-key lookup into _get_model_cost_capability_exact and use it for the hard rejection so a fuzzy match can never block images; the substring fallback stays a soft signal for capability reporting. Also rewrites the fallback as a comprehension + max instead of an accumulating loop. * feat(fireworks_ai): surface response fields on streaming responses The Fireworks-specific response fields (perf_metrics, prompt_token_ids, per-choice raw_output and token_ids) were only captured into _hidden_params in transform_response, which runs for non-streaming completions; streaming chat went through the default OpenAI chunk handler and dropped them. Add a FireworksAIChatCompletionStreamingHandler that the provider now returns from get_model_response_iterator. It reuses one extraction helper with transform_response and attaches the fields to each streamed chunk's provider_specific_fields, which is the channel litellm preserves when it rebuilds streamed chunks (per-chunk _hidden_params is not carried through). Per-choice token_ids/raw_output ride the content chunks; response-level perf_metrics/prompt_token_ids ride the final usage chunk. Covered by an end-to-end streaming test through litellm.completion(stream=True). --------- Co-authored-by: Ahmad Shahzad <ahmad@shahzad.dev> Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com>	2026-06-20 19:49:07 -07:00
Shivam Rawat	c7efa77de3	fix(watsonx): wrap string embedding input in array for WatsonX API (#30897) ... * fix(watsonx): wrap string embedding input in array for WatsonX API WatsonX text/embeddings expects inputs as []string; OpenAI clients often send a single string. Co-authored-by: Cursor <cursoragent@cursor.com> * style(watsonx): format watsonx embed transformation for black Co-authored-by: Cursor <cursoragent@cursor.com> * fix(watsonx): avoid UP006 in embed transformation strict lint gate Use list[str] and branch-based input normalization instead of List and cast so the watsonx embedding change does not add strict ruff UP006 violations. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-20 15:50:44 -07:00
Sameer Kankute	4c25b7a13d	chore: litellm oss staging (#30745) ... * fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708) OpenAI GPT-5 models require max_completion_tokens >= 16. Health checks were using 5 (proxy/health_check.py) and 10 (health_check_helpers.py), causing failures on GPT-5 models. Fixes #23836 * fix: increase health check max_tokens from 5 to 16 (#23836) (#26610) GPT-5 models enforce a minimum of 16 for max_output_tokens. The current default of 5 still causes health checks to fail for these models. Bump the non-wildcard default to 16 — the smallest value that satisfies all known provider minimums while keeping health checks lightweight. Also tightens the wildcard test assertion from a weak disjunctive check to strict key-absence. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696) * fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema. * fix: remove async keyword from test. * fix: make Bedrock Mantle Responses routing data-driven per model (#30700) * Make Bedrock Mantle Responses routing data-driven per model Route Bedrock Mantle models to the native Responses API based on each model's price-map capability signal instead of a hardcoded model-name heuristic, and derive the OpenAI-compatible base path segment per model. Responses dispatch now selects the native config when the model advertises responses support (/v1/responses in supported_endpoints, or mode=responses), both overridable via register_model and proxy model_info. This enables native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing chat-completions emulation. Capability is per-model, so gpt-oss-120b routes natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss substring. The wire path is a separate concern, driven by the existing use_openai_responses_path flag rather than a model-name match: gpt-5.x and gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat config now derives its base from the same flag, fixing gemma-4 chat-completions requests that previously went to /v1 instead of /openai/v1. Cost maps: add supported_endpoints to the gpt-oss entries (responses for the non-safeguard variants, chat-only for safeguard) and supported_endpoints + use_openai_responses_path to all three gemma-4 entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: move capability helper into bedrock_mantle package Move the Responses capability check out of utils.py into litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses, alongside its companion wire-path helper mantle_base_segment. Both are now pure functions of (model, model_cost): the price-map mode/supported_endpoints read replaces the get_model_info call, so the rules are unit-testable without patching global state and the Bedrock Mantle package is self-contained. Use str | None instead of Optional[str] on the new signatures to satisfy the ruff UP045 strict-rule gate. Add direct unit tests for both helpers. Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b now legitimately supports Responses, so it can no longer be the "None after restore" vehicle; use the chat-only safeguard variant, which isolates the register/restore effect from the model's own capability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653) The tiered cost calculator resolved a tier's per-token cost with `tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or` short-circuits on any falsy value, a tier that legitimately prices a component at 0.0 (e.g. a free-cache-read tier with cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated as missing and silently billed at the full fallback rate (input_cost_per_token / output_cost_per_token). The flat-pricing path in the same module already handles this correctly with an `is None` guard. Resolve tier costs through a small helper that mirrors it, so 0.0 is honored at both the in-range and overflow sites. No shipped model currently has a 0.0 tier cost, so this is a latent defect; the fix makes the tiered path consistent with the flat path and prevents over-charging the first time such a tier appears. Adds unit tests covering the in-range and overflow paths, and drops an unused import flagged by ruff in the touched test file. * feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507) * fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618) In the messages->chat/completions bridge, translate_anthropic_tools_to_openai merged every non-mapped tool key into the function parameters dict. The Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object' -> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type). Exclude 'type' from the passthrough. Fixes #30557. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * feat(bedrock): support file content retrieval for batch output files (#30595) Implements transform_file_content_request and transform_file_content_response in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch files. The request transform resolves the file id (direct s3:// URI or base64 unified id) to its S3 object, validates bucket and key prefix against the server-configured bucket, and SigV4-signs an S3 GetObject using the same credential and region resolution as the existing upload path. The credential and region params are validated into a typed model at the boundary, so the only untyped values left are the botocore signing primitives. Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries s3_bucket_name (previously dropped when building deployment credentials) and the managed-files hook passes the deployment credential snapshot when routing afile_content, so unified-id content retrieval works with per-model bucket config instead of only the AWS_S3_BUCKET_NAME env var. Preserves managed-file access control: the proxy file-content endpoint now rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the owner/team check that only runs for unified ids and let a caller read another tenant's batch output by its object key. Managed outputs are reachable only through their unified file id. The afile_content "not found" error now reports the caller's unified id rather than the resolved internal S3 URI. Fixes #16186, #15563 * fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646) * fix(oci): map Cohere tool array/object params to lowercase builtins OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare "List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema arrays. MLflow {{trace}} judges trip this: their tools (get_root_span, get_span) take an attributes_to_fetch array. The lowercase builtins list/dict are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but both are lowercased for consistency). Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest). Adds a unit regression on the transformed parameterDefinitions plus a gated integration test exercising an array-param tool end to end. * fix(oci): make Cohere agentic tool-calling continuation work Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges drive once a tool has been executed and its result is fed back. Request side: litellm pulled the last user message into the top-level `message` and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that ("cannot specify message if the last entry in chat history contains tool results"), and an empty message alone is rejected too ("message must be at least 1 token long or tool results must be specified"). OCI carries the current turn's results in a dedicated top-level `toolResults` field. The Cohere transform now sends an empty message, keeps the user turn in chatHistory, and puts the results in `toolResults`, matching the langchain-oracle reference. Tool results are no longer represented as chatHistory entries. Response side: tool-grounded answers come back with citations carrying `documentIds` (camelCase) and no `document_ids`, which made the required `CohereCitation.document_ids` field fail validation and sink the whole response parse. Those citations are never surfaced, so the field (and CohereSearchQuery's generation_id) is now optional. Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest), single and multi-round tool loops. Adds unit regressions on the transformed request shape and on citation parsing, plus gated integration tests for the continuation. * feat: integrate Repelloai Argus guardrail (#30673) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * refactor(guardrails/repello): clean up typing and remove lint-any workarounds - Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout - Use dict[str, object] instead of bare dict in all signatures - Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly - Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel - Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks - Use TypeAdapter.validate_json() instead of response.json() + manual dict construction - Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any - Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check - Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType - Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel] * fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning - Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate - Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks * refactor: modifications for lint check * feat: add Pinstripes as an OpenAI-compatible provider (#30567) * feat: add Pinstripes as an OpenAI-compatible provider Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.) with per-token pricing and no subscriptions. Changes: - `litellm/llms/openai_like/providers.json`: register pinstripes with base_url, api_key_env, and max_completion_tokens→max_tokens mapping - `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders - `litellm/constants.py`: add to openai_compatible_providers and openai_compatible_endpoints lists - `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect provider when api_base is "https://pinstripes.io/v1" - `provider_endpoints_support.json`: document supported endpoints - `tests/`: 7 unit tests covering provider registration, resolution, URL auto-detection, api_base override, and Router config Usage: import litellm response = litellm.completion( model="pinstripes/ps/glm-4.5-air", messages=[{"role": "user", "content": "Hello"}], api_key=os.environ["PINSTRIPES_API_KEY"], ) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): resolve Greptile P1 review comments - Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works - Set responses: false in provider_endpoints_support.json — not actually wired up - Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): add api_base_env and correct responses capability - Add api_base_env: PINSTRIPES_API_BASE to providers.json - Set responses: false in provider_endpoints_support.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): wire up Responses API — add supported_endpoints Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so JSONProviderRegistry.supports_responses_api returns true correctly, matching what provider_endpoints_support.json advertises. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(pinstripes): enable embeddings endpoint Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings. Add /v1/embeddings to supported_endpoints and set embeddings: true. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json Matches the file's existing convention. Flagged by Greptile review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): set a2a: false — A2A protocol not implemented All comparable JSON-configured providers (tensormesh, parasail, empiriolabs, libertai, neosantara) have a2a: false. Pinstripes does not implement the Google A2A protocol, so this should be false to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(rag): attach existing OpenAI file ids (#30628) * fix(rag): attach existing OpenAI file ids * chore: use modern typing in rag ingest fix * chore: retrigger ci * fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341) cache_control_injection_points was only consumed by the chat/completions prompt-management hook; on the native Anthropic /v1/messages path it was forwarded unused, so deployment-level cache injection was silently dropped (cache_creation_input_tokens stayed 0 for Anthropic-native clients). Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject cache_control at block level for system / tools / message locations (the only forms /v1/messages accepts), wire it into the native anthropic_messages handler, and pop the param so it does not leak upstream as an unknown field. A {location: message, role: system} config is redirected to the top-level system prompt so the same YAML works on both endpoints. Injection respects Anthropic's 4-block cache_control limit shared across system, tools, and messages: client-supplied markers count toward the cap and are never overwritten, a slot is reserved per Bedrock tool_config point, and injection stops once the budget is exhausted. Locations this path cannot represent (tool_config) are forwarded downstream instead of being silently consumed, mirroring get_chat_completion_prompt's remaining_points pass-through. Built on litellm_internal_staging. Refs BerriAI/litellm#30293 * fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522) * fix(proxy): release budget reservation on cancel when no chunk was delivered The pre-call budget reservation increments the cross-pod spend counter by a request's worst-case cost, then reconciles it on success (cost callback) or error (failure hook). A client disconnect or timeout cancels the request and surfaces as CancelledError / GeneratorExit, which neither path catches, so the reservation leaks. Under a retry storm the leaked holds accumulate, pin the counter above real spend, and return spurious 429 "Budget has been exceeded" to keys whose spend is far below budget; the counter only recovers when its TTL lapses, so the failure is intermittent and self-healing. Release the reservation in async_streaming_data_generator (which the Anthropic and Google SSE generators delegate to) on the (CancelledError, GeneratorExit) path, alongside the existing max_parallel_requests release. release_budget_ reservation_on_cancel runs under asyncio.shield so it completes despite the in-progress cancellation, is guarded by the reservation's finalized flag, and swallows a failing release so it cannot replace the in-flight cancellation. The refund is gated on whether a chunk reached the client. The flag is set immediately before the yield, after the slow-path hook await: an async generator suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk sees it True (keep the hold), while a cancellation during the slow-path await leaves it False (refund, nothing sent). A non-streaming cancellation delivers nothing and a completed non-streaming response is reconciled by the success callback, so neither needs a release here. Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): reconcile a cancelled reservation to input cost, not zero A streaming request cancelled before the first chunk previously reconciled its reservation to zero and finalized it. But by the time the generator is consuming the response the provider call was already dispatched, so the input tokens were billed even though no chunk reached the client, and the success/failure cost callbacks are skipped on cancellation. Refunding to zero let a caller send an expensive request and abort pre-token to dodge the input charge. Compute the request's input-token cost at reservation time and reconcile the cancelled reservation to it instead of zero. The worst-case output portion of the reservation is still released (so a legitimate mid-flight cancellation no longer pins the counter and 429s the key), while the input the provider already processed is charged. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(caching): encode object name in GCS cache GET path (#30378) GCS cache reads always missed when gcs_path was set. The GET methods interpolated the object name directly into the URL path, while the GCS JSON API requires it to be URL-encoded (a "/" must be sent as %2F). With gcs_path configured the object name is "<prefix>/<sha256>", so the raw slash produced a malformed object path and GCS returned 404. httpx does not raise on 4xx, so the status_code == 200 check fell through and get/async_get returned None, silently missing on every read. Without gcs_path the key has no slash, which is why this went unnoticed. Wrap the object name with urllib.parse.quote(..., safe="") in get_cache and async_get_cache. Apply the same encoding to the name= query parameter in set_cache and async_set_cache so the key written matches the key read back. Adds regression tests asserting the GET path and SET query are encoded (%2F) when gcs_path is set, for both sync and async paths; these fail on the unpatched code. Fixes #30377 * chore: add soniox stt-async-v5 model (#30672) * fix(proxy): include model group aliases in v1 model info (#30626) * Include model group aliases in v1 model info * Fix model info alias implementation * removed extra blank line * chore: rerun CI * fix(lint): remove redundant noqa directive in proxy_cli.py * fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme * Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme" This reverts commit 52c7a07777. * Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)" This reverts commit c9e8a177bd. * Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)" This reverts commit 85828da695. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045) * Revert "feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)" This reverts commit f530b2237c. * Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)" This reverts commit 4f58bd0df5. * Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)" This reverts commit 50f34e0b15. * Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)" This reverts commit 0544eed6ea. * fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite * fix(key management): restore exact /key/list user_id & key_alias matching by default (#30593) Before substring search was added (commit 33bd570d5e), /key/list matched user_id and key_alias exactly. That change made admin-authenticated calls substring-match by default, breaking the prior contract: a caller passing an exact user_id as an access filter (e.g. an integration scoping to one user with an admin key) then received other users' keys -- user_id="alice" also returned "alice2", "alice-test", etc. This is a cross-user key disclosure. Make substring matching opt-in via a new admin-only substring_matching=true query param; default to exact, restoring the prior behavior. The dashboard search box (keyListCall) passes the flag so partial search still works. Non-admins remain exact and scoped to their own keys. Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default guard; adds list_keys unit coverage for the opt-in gate. --------- Co-authored-by: perseus <51974392+tcconnally@users.noreply.github.com> Co-authored-by: Hannah Smith <64043506+hannahmadison@users.noreply.github.com> Co-authored-by: Charlie Patterson <Pattersoncharlesl@gmail.com> Co-authored-by: Matthew Lapointe <mlapointe@alpha-sense.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Nitish Agarwal <1592163+nitishagar@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com> Co-authored-by: AD Mohanraj <admohanraj@gmail.com> Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Lavish Bansal <lavish.bansal619@gmail.com> Co-authored-by: max-amos <gruffulom@gmail.com> Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: NK <93352237+Nithish-Yenaganti@users.noreply.github.com> Co-authored-by: 安妮的心动录 <74543653+anneheartrecord@users.noreply.github.com> Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Burak Ömür <burak.omur.1998@gmail.com> Co-authored-by: Dan Lemon <daniel.lemon@amazee.io> Co-authored-by: Vanika Dangi <166420943+vanika02@users.noreply.github.com> Co-authored-by: Jay Gowdy <130084966+jgowdy-godaddy@users.noreply.github.com>	2026-06-18 13:55:35 -07:00