13 commits

Author	SHA1	Message	Date
Sameer Kankute	133da06aa3	chore: litellm oss staging (#31185) ... * fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped The Total Tokens Over Time and Total Requests Over Time AreaCharts on the Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow once totals pass the hundred-million mark — leading digits of labels like "100.00M" / "4500.00M" got clipped against the chart edge. The requests chart was worse: it formatted with toLocaleString(), so billion-scale request counts produced "1,000,000,000" (13 chars) and overflowed immediately. Fix in two places so neither alone has to carry the whole margin: - activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and switch the requests chart to the shared valueFormatter so it uses the same compact k/M/B suffixes as the tokens chart. - value_formatters.tsx: add a >= 1e9 branch to valueFormatter / valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping every formatted label at most 7 chars. Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com> * Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * docs(readme): add Deploy on AWS/GCP with Terraform section Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): add 1-click deploy buttons for AWS + GCP GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): move AWS + GCP deploy buttons next to Render button * docs(readme): unify deploy button sizes and badge styles * docs(readme): bump deploy button height to 48 to match Render/Railway * docs(readme): bump AWS/GCP badge height to compensate for SVG padding * docs(readme): bump AWS/GCP badge height to 72 * docs(readme): bump AWS/GCP badge height to 84 * fix(readme): make deploy buttons same height (48px) https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc * docs(readme): flag GCP project ID substitution in image_registry * docs(readme): equalize deploy button heights and fix Cloud Shell button font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface. * docs(readme): collapse Railway deploy anchor to a single line The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline. * Add Claude Fable 5 cost map entries as a data-only hotfix Backports only the model map changes from #30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano Three bugs in model_prices_and_context_window.json: 1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K max output, but the values were set as max_input=128000, max_tokens=272000. This caused token limit errors when sending prompts over 128K tokens to GPT-5 Pro. 2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was 272000, but GPT-5.4 Mini shares the same 1,050,000 token context window as GPT-5.4. This was inconsistent with the azure/ variants which already correctly had 1,050,000. 3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini, max_input_tokens was 272000 instead of 1,050,000. Source: OpenAI model documentation and contextwindows.dev which aggregates official context window sizes. Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini should be 400K; their 272K values are correct per OpenAI docs) * fix: also correct max_output_tokens for gpt-5-pro (272000→128000) Per reviewer feedback, max_output_tokens was left at 272000 while max_tokens was corrected to 128000, causing an internal inconsistency. Both should be 128000 per OpenAI docs. * fix(cost): price gpt-image generated output tokens as image tokens (#31147) The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return usage with no output token breakdown — litellm's `ImageUsage` has no `output_tokens_details` field — so generated-image OUTPUT tokens were priced at the text rate (`output_cost_per_token`) instead of the image rate (`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x undercount on the dominant cost component (image output is ~74% of spend). This also affects azure gpt-image, which shares this calculator. The OpenAI gpt-image cost calculator re-implemented usage handling instead of reusing `calculate_image_response_cost_from_usage`, the shared helper that azure_ai/gemini/vertex_ai already use. That helper classifies generated output tokens as image tokens when the provider does not itemize output, and splits text/image when it does. Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage` (pre-transformed chat Usage objects are still costed directly). Adds a regression test for the no-breakdown ImageUsage case (gpt-image-2). * fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098) A bare application-inference-profile ARN passed as bedrock/arn:... fell through to the invoke route, which cannot derive a provider from the opaque profile id and raised 'Unknown provider=None'. The converse route needs no provider, so detect these ARNs in get_bedrock_route and route them to converse, matching the behavior of the already-documented bedrock/converse/arn:... workaround. Explicit invoke/ prefixes still win, and they remain a dead end for these ARNs by design (no provider derivable). System-defined inference-profile ARNs that embed a known model, and other opaque ARN types (provisioned-model, imported-model, custom-model-deployment) that are frequently invoke-only, are deliberately left on their current routes; tests guard both boundaries. * fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060) _add_tool_choice_required_message appended the "select a tool" prompt to the caller's messages list in place, so transform_request corrupted the caller's conversation history and appended a duplicate prompt on every retry. Build and return a new list instead so the call stays idempotent. Adds a regression test asserting the input messages list is unchanged across repeated transform_request calls. Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> * fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996) gpt-4o-transcribe and compatible ASR backends return a diarized_json response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8. TranscriptionUsageDurationObject typed seconds as int, so parsing the response raised a pydantic ValidationError (int_from_float). That error surfaces as an APIConnectionError which the router treats as retryable, so it keeps re-calling the upstream (200 every time) until the upstream rate-limits and returns 429 to the caller. OpenAI specs this field as a float (see openai SDK UsageDuration.seconds), so widen seconds to float. With the parse succeeding there is no exception left to retry, which removes the loop. Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> * fix(deepseek): drop non-function tools before chat completions call (#30910) * fix(deepseek): drop non-function tools before chat completions call DeepSeek's /chat/completions only accepts tools of type "function". Requests bridged from /v1/responses can carry responses-API-native tool types, for example a Codex CLI tool typed "namespace", which DeepSeek rejects with "unknown variant 'namespace', expected 'function'" so the whole request fails (issue #30722). Filter unsupported tool types in the DeepSeek request transform so the function tools still go through; when nothing callable remains, also drop the now-dangling tool_choice and parallel_tool_calls Fixes #30722 * test(deepseek): cover async tool filtering and document tool_choice assumption Add an async_transform_request regression test so the sync and async tool filtering paths cannot silently diverge, and document in _drop_unsupported_tools that only non-function tools are dropped, so a function-named tool_choice always references a surviving tool * feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840) * feat(ui): surface team budget on key overview when key has no own budget (#30801) * feat(ui): surface team budget on key overview when key has no own budget * fix(ui): replace IIFE with derived variable and use find() for team budget display * fix(anthropic): emit replayable streaming thinking blocks (#31022) * feat(proxy): read cold-storage prompts back in the logs detail view (#30364) * feat(proxy): read cold-storage prompts back in the logs detail view When a deployment offloads prompts and responses to cold storage instead of Postgres, the spend-log row holds only "{}" placeholders plus a metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed nothing. The detail endpoint only read the placeholder columns and never fetched the object back. Resolve the payload per row based on actual content, not a config flag: if Postgres has content, return it; otherwise read the exact stored object key and fetch from the configured cold storage backend through ColdStorageHandler. Reading the persisted key is a single GET. The key embeds a microsecond timestamp that cannot be reconstructed from the millisecond-precision startTime column, and listing the day's prefix to match on request_id would be too expensive for this per-open path. Also teach the detail drawer's pretty-view parser to accept a bare messages array. The cold storage payload carries the prompt as a top-level messages list with no proxy_server_request, so without this the output rendered while the input stayed blank. ColdStorageHandler gains an optional injected logger so the resolver can be unit tested without monkeypatching. Postgres-stored prompts are unaffected: the fast path returns the existing columns and the request-body object still renders the same way. * Update litellm/proxy/spend_tracking/spend_management_endpoints.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure Add unit tests for ColdStorageHandler (injected logger, graceful None when no logger is configured, and resolution of a configured logger from the callback registry) and a regression test asserting a cold storage backend exception degrades to the Postgres values instead of surfacing a 500. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068) * fix(mavvrik): advance metricsMarker after upload + fix scheduler startup Two bugs fixed: 1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a successful GCS upload, so metricsMarker stayed at 0 and every daily run re-exported the same dates in an infinite catch-up loop. Fix: add _update_metrics_marker(date_epoch) called at the end of deliver() after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS file is already committed). A 410 raises consistent with the rest of the destination. 2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call has triggered lazy instantiation of MavvrikFocusLogger, so it found no logger instance and silently skipped registering the daily export job. Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call _init_custom_logger_compatible_class to force instantiation before the APScheduler job is registered. * fix(mavvrik): catch up from earliest window when metricsMarker=0 When the connector is freshly registered, metricsMarker=0 parses to None. The catch-up block was guarded by `if last_ingested and ...` which skipped it entirely for None, so only yesterday was exported instead of the full _MAX_CATCHUP_DAYS window. Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup). The existing > 7 day warning only fires for non-None markers that are old. * fix(mavvrik): use now as end_time for yesterday's export window LiteLLM_DailyUserSpend rows for a given date get their updated_at bumped by the spend flush job throughout the next morning. The core database query filters on updated_at, so capping end_time at midnight (yesterday + 1 day) missed any spend rows flushed after midnight. Fix: pass now (cron fire time) as end_time for the daily "yesterday" window so all fully-settled rows are captured regardless of when the flush job ran. Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per row in the exported FOCUS CSV. * fix(mavvrik): also use now as end_time for catch-up windows * fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class Calling it with only logging_integration raised TypeError at proxy startup because internal_usage_cache and llm_router have no defaults. Also fix test name to reflect the actual status code (5xx not 4xx) used in the mock. * ci: retrigger CI run * feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757) * Add optional `instruction` passthrough to the rerank API vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction` field (folded into the model's chat_template_kwargs and consumed by the chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently dropped it: RerankRequest / OptionalRerankParams had no such field, so the outgoing body was rebuilt without it. Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(), get_optional_rerank_params, and the hosted_vllm transformation into the request body, only when non-None. When callers omit it, model_dump(exclude_none) drops the field and the outgoing request is byte-for-byte unchanged — fully backward-compatible. (DeepInfra already forwards `instruction` via non_default_params; this formalizes the field in the shared types.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: thread `instruction` as a typed param + cover rerank_utils Per PR review (greptile P2 + codecov): - Make `instruction` a typed, named argument on the rerank provider interface instead of recovering it from the opaque `non_default_params` blob. Adds `instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params` and every provider override, and forwards it explicitly from `get_optional_rerank_params`. hosted_vllm now reads the named param directly. It is still also surfaced in `non_default_params` so providers that read it there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction` as a named param rather than leaving it in **kwargs. - Add get_optional_rerank_params unit tests (present + absent) to cover the previously-uncovered threading line flagged by codecov. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix: scan rerank `instruction` through request guardrails The rerank guardrail translation (CohereRerankHandler.process_input_messages) only scanned `query`, so the newly added `instruction` field reached the backend model unscanned. Since instruction-aware rerankers (hosted vLLM / Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller could place content there to bypass configured rerank request guardrails. Generalize the handler to scan every user-controlled text field (`query` and `instruction`) in one apply_guardrail call and write each sanitized value back by index. Query-only requests are unchanged (single-element list at index 0); non-string fields are left untouched. Adds tests covering instruction scanning, PII masking write-back, and the non-string case. Addresses the Veria AI security review on PR #30757. * test: narrow Optional results before len() to satisfy basedpyright budget The lint gate (basedpyright delta-vs-base budget) flagged one new reportArgumentType: len(result.results) where results is List[RerankResponseResult] | None. Assert results is not None first to narrow the type before len()/indexing. * fix: read rerank `instruction` from kwargs to satisfy basedpyright budget The basedpyright delta-vs-base gate flagged one new reportArgumentType: the Router forwards rerank calls via an untyped `**kwargs` unpack (`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a typed named param on the public `rerank`/`arerank` entrypoints made pyright check that key against `str | None`, adding an error at router.py with no real safety gain. Read `instruction` from kwargs in `rerank` instead. It remains fully typed where it matters - threaded as a typed argument through `get_optional_rerank_params` and each provider's `map_cohere_rerank_params` (the original Greptile P2 ask). Whole-repo reportArgumentType is back to the base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(github_copilot): synthesize empty choices at the provider seam (#30929) Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with choices=[], either carrying Anthropic-native content blocks or, for the max_tokens=1 probe Claude Code sends, no content at all. github_copilot is dispatched through the OpenAI SDK handler, which calls convert_to_model_response_object directly and never invokes GithubCopilotConfig.transform_response, so the empty-choices guard there surfaced as a 500 Instead of synthesizing choices inside the shared convert_to_model_response_object (which would silently turn empty choices into a fabricated success for every provider), add a no-op transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig overrides it to synthesize choices from Anthropic-native content, reusing its existing parsing, and the OpenAI SDK handler routes its parsed response through the hook before generic conversion. The core utility keeps treating empty choices as an error for all other providers Fixes: https://github.com/BerriAI/litellm/issues/30927 Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> * fix(router): stop fallback lookups from mutating the router fallbacks config (#30624) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens * test: scope local cost map env var with monkeypatch to avoid test pollution * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764) * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold _mask_value did partial reveal by showing the first visible_prefix and last visible_suffix characters, but for a value whose length was at or below visible_prefix + visible_suffix (8 by default) it returned the value verbatim. A value of exactly 8 chars fell through the length guard and computed masked_length == 0, reconstructing the original string with no mask characters; anything shorter hit the early return. Either way short credentials were emitted in plaintext. mask_dict routes real secrets through this path, so an 8-char-or-shorter redis password, api key, or token could be written to logs and the UI unmasked. The sibling helper mask_sensitive_keys already guards this case; _mask_value now does the same by fully masking any value at or below the threshold. * fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers Fully masking short values is the right default for secret masking, but CooldownCache reuses the masker purely to truncate exception messages to the first 50 characters, and it relies on short messages being returned readable. Masking those blanked out short exception text and broke its tests. Add a mask_short_values flag (default True, secure) and have CooldownCache pass False so it keeps the truncation behavior, while every secret-masking caller still gets short values fully masked. * fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview MCPDebug uses the masker to preview auth tokens in debug headers and documents that values of 10 chars or fewer are shown unchanged so token types stay distinguishable. Pass mask_short_values=False so that diagnostic behavior is preserved while secret maskers keep masking short values. * fix(mcp_debug): mask short auth values in debug headers instead of echoing them Earlier this masker opted out of short-value masking to keep a token preview, but that echoes short authorization and token values verbatim in debug response headers, which is the same leak this change is meant to close. Auth material should never be emitted in full, so mask short values here too; the first/last character preview still applies to longer tokens. Only CooldownCache keeps the opt-out, since it truncates exception text rather than masking secrets. * test(mcp_debug): assert masked short value preserves length * refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917) Fireworks AI deprecated audio inference on 2026-06-10 (https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation). Live API testing confirms the endpoint is already non-functional: a valid Fireworks API key receives HTTP 401 "Unauthorized" from api.fireworks.ai/inference/v1/audio/transcriptions for every request, regardless of payload. The audio-prod.api.fireworks.ai host referenced in the test suite returns 401 for every path; the entire host is decommissioned. Remove the dead FireworksAIAudioTranscriptionConfig class and every reference to it across the codebase: - Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line config class that inherited from OpenAIWhisperAudioTranscriptionConfig) - Remove the Fireworks branch from ProviderConfigManager.get_provider_audio_transcription_config() in litellm/utils.py; update the stale comment in get_optional_params_transcription that referenced fireworks ai - Remove the FireworksAIAudioTranscriptionConfig entries from LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in litellm/_lazy_imports_registry.py - Remove the TYPE_CHECKING re-export in litellm/__init__.py - Remove the transcription branch in the fireworks_ai case of get_supported_openai_params() in litellm/litellm_core_utils/get_supported_openai_params.py - Remove the whisper-v3 and whisper-v3-turbo entries from model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json (both had mode: audio_transcription and zero-cost pricing) - Remove the TestFireworksAIAudioTranscription test class and its imports from tests/llm_translation/test_fireworks_ai_translation.py No other provider is affected. The openai_compatible_providers list, FireworksAIMixin, and the OpenAI Whisper transcription handler all stay because they are shared with other Fireworks endpoints and other providers. The provider_endpoints_support.json registry already had audio_transcriptions set to false for fireworks_ai. * feat: add darkbloom provider (#30876) * feat: add darkbloom provider * fix: document darkbloom provider endpoints * fix: address darkbloom review feedback * fix: update darkbloom tool metadata * fix: fail fast for non-Postgres database URLs (#30883) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging * Validate DIRECT_URL alongside DATABASE_URL startup guards * fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946) * fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608) * test(bedrock): mid-stream server errors trigger streaming fallback (#24608) * style(bedrock): black-format stream-error helper (#24608) * fix(mcp): re-land native tool preservation with typed annotations (#30645) * fix(mcp): preserve native tools in semantic filter hook with typed annotations * fix(mcp): tighten _is_mcp_tool Chat Completions shape check * fix(sambanova): return embeddings supported params instead of dropping them (#30937) * fix(router): send fallback metadata when streaming (#30914) When a streaming request triggers a fallback, there was previously no way to know it happened. This commit addresses this in a few ways: 1. The response now correctly populates the fallback headers (`x-litellm-attempted-fallbacks`) so callers know a fallback happened. 2. The correct model ID is passed in the streaming chunks. 3. A streaming chunk with the fallback error can be optionally sent back to the client (opt-in) by passing `include_fallback_errors: true` in the request. The format of the fallback errors while streaming is intentionally OpenAI compatible to not break existing libraries that parse these events. It was tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not delieved unexpectedly to callers by default. * fix(mistral): drop output-only reasoning fields from input messages (#30884) LiteLLM attaches reasoning_content and thinking_blocks to assistant responses. Replaying those assistant turns verbatim forwarded the fields back to Mistral, whose input schema forbids unknown keys, so the whole request failed with a 422 extra_forbidden and reasoning models became unusable across multiple turns. Strip both fields from assistant messages before the request is built, in a spot that runs ahead of the image/file branch so it applies on every path. Fixes #30835 Co-authored-by: Cursor <cursoragent@cursor.com> * fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652) * fix(perplexity): bill search queries at the per-request price, not 1/1000 The fallback cost calculator divided search_context_cost_per_query by 1000, but that field stores the per-request price in USD: sonar is {low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published $5/$8/$12 per 1,000 requests expressed per request. The gemini cost calculator reads the same field per request with no division (its docstring calls it "the per-request cost"). The division understated search cost by 1000x on every Perplexity call that falls back to manual calculation (i.e. when the API does not return a pre-computed usage.cost). Use the value directly. Update the tests that had encoded the /1000 factor in their expectations, and drop an unused import flagged by ruff in the touched test file. * test(perplexity): update integration test search-cost expectations to per-request The integration tests still encoded the old /1000 search-cost factor, so they failed once the fallback calculator was corrected to bill search_context_cost_per_query per request. Update the four expected-cost computations (and the high-volume dollar-value comments) to match. * test(perplexity): drop unused mock imports flagged by ruff * fix: include model_access_groups when expanding all-team-models in get_team_models (#30622) * fix(fireworks_ai): return None for transcription in get_supported_openai_params Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is decommissioned. Without an explicit transcription branch, requests with request_type='transcription' fell through to the else and returned FireworksAIConfig chat-completion params. Return None instead to signal the provider does not support transcription. * fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting Without an operator gate, any authenticated caller could set include_fallback_errors=True, trigger a fallback, and read raw upstream exception messages from the x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event. Strip include_fallback_errors from request data in common_processing_pre_call_logic when expose_fallback_errors_to_caller is not set, so the router never builds the error list. Also gate _should_include_fallback_errors on the same setting as a secondary check for the streaming SSE injection path. * test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test The operator gate added in e7ff3e1 means include_fallback_errors is only honoured when general_settings.expose_fallback_errors_to_caller is True. Set that flag via monkeypatch in the test that exercises the emit path. * test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos test_convert_url called convert_url_to_base64 against a live picsum.photos URL and asserted nothing, so it added no real signal and broke CI whenever the host was unreachable (it was returning 522 and blocking this branch). Replace the live call with a mocked HTTP client and assert the produced base64 data URL, so the conversion path is exercised deterministically with no network dependency. This suite runs under VCR, which is why a transport level mock (respx) does not reliably intercept; mocking the client object itself is robust regardless. * fix(interactions): drop role from Interaction response to match Google spec Google removed the output-only role field from the Interaction schema (it now lives only on Turn), so the live OpenAPI compliance canary started failing with 'role' not in spec. Reconcile our generated types by removing role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse, stop stamping role=model in the responses-to-interactions transformation, and update the compliance and integration tests accordingly. Turn.role is kept since the spec still defines it. * fix: align all-team-models sentinel access * fix(router): forward include_fallback_errors through multi-hop fallbacks run_async_fallback received include_fallback_errors as an explicit named parameter, so it was bound out of **kwargs and never reached the nested async_function_with_fallbacks call. Multi-hop fallback chains (a fallback group that itself fails over) therefore stopped collecting fallback errors beyond the first hop when a caller opted in. Re-inject the flag into kwargs before the nested call so inner hops keep accumulating errors, which add_fallback_headers_to_response already merges across levels. * fix(router): stop fallback lookups from mutating the router fallbacks config get_fallback_model_group resolved a bare-string fallback by popping it out of the fallbacks list it was handed. That list is frequently the live router.fallbacks config, so a single lookup permanently removed the entry and the configured fallback stopped applying to later requests until restart. The pop also ran inside enumerate(), shifting indices and skipping an adjacent string fallback. Read the item instead of popping it, and add a regression test that fails on the old mutating behavior --------- Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * fix(sambanova): update pricing, deprecate retired models, and add missing models (#30016) * feat(bedrock): add amazon.titan-embed-g1-text-02 embedding model support - Add model to provider routing allowlist in embedding.py - Add request transformation using AmazonTitanG1Config - Add response transformation using AmazonTitanG1Config - Add pricing metadata to model_prices_and_context_window.json - Add unit tests for embedding and model info Fixes missing cost tracking reported in #29786 Related to VANDRANKI/litellm PR #29790 * style: fix syntax error, trailing whitespace and missing newline * style: apply black formatting to embedding.py * style: apply black formatting to test_bedrock_embedding.py * fix(sambanova): update pricing, fix context windows, add deprecation dates, and add missing models * fix(sambanova): sync model_prices_and_context_window_backup.json with primary * fix(sambanova): fix indentation on Meta-Llama-3.2-1B-Instruct deprecation_date * fix(bedrock): add amazon.titan-embed-g1-text-02 to unmapped model error message * style: apply black formatting to embedding.py * fix(sambanova): correct indentation on DeepSeek-V3.2 entry * fix(sambanova): replace gemma-3-12b-it with gemma-4-31B-it (verified pricing) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info (#30880) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info get_model_info rebuilt ModelInfo by copying a fixed allow-list of input/output_cost_per_token_above_<N>_tokens keys (128k/200k/272k/512k), so any other threshold a user registered was dropped before reaching _get_token_base_cost, which already reads an arbitrary threshold out of the key name. Custom tiers such as above_500k_tokens were silently ignored and billing fell back to the base per-token rate. Carry over any _above_<N>_tokens cost key present on the source cost-map entry that the fixed fields miss Fixes #30344 * test(cost): keep suite hermetic by popping the temp tiered-pricing model Wrap the regression body in try/finally so litellm.model_cost no longer leaks the litellm-test-non-standard-tier entry into later tests that iterate or reset the global cost map. Addresses Greptile review thread. * fix: resolve UP045 lint violations (Optional[X] -> X | None) Convert Optional[X] type annotations to X | None syntax across rerank transformations, spend tracking, and other modules to satisfy ruff strict gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: run black formatting on UP045-fixed files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove unused Optional imports after UP045 migration Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: black format cold_storage_handler.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ci): correct OSS staging branch name in guard-main-branch errors Co-authored-by: Cursor <cursoragent@cursor.com> * fix: strip trailing zeros from M/B spend formatter * fix: address focus and streaming edge cases * feat: add LAR-1 semantic routing strategy Optional router strategy that picks a deployment tier from request_kwargs.metadata.lar1 (confidence, evidence, time). Deployments are tagged with model_info.type (cloud-smart, cloud-fast, local, deep). Thresholds are configurable via routing_strategy_args. Includes 30 unit tests and an Ollama example config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mavvrik): advance metricsMarker on empty-content deliver When deliver() receives empty content (no spend data for a date), it now registers with Mavvrik and PATCHes the metricsMarker before returning instead of short-circuiting. Dates with zero spend no longer stall marker advancement, preventing unnecessary catch-up API calls on subsequent runs. * style: black format mavvrik_destination * fix: handle empty mavvrik exports and lar1 reset * test: add regression test for _reset_custom_routing_strategy * fix(test): mock async destination.deliver in mavvrik export window test * style: ruff format spend_management_endpoints after merge * fix(router): apply LAR-1 strategy atomically so invalid thresholds don't leave partial state apply_lar1_routing_strategy set router.routing_strategy to "lar1" before constructing LAR1RoutingStrategy, whose __init__ validates thresholds via _normalize_thresholds and raises on a misconfigured (out-of-order or out-of-range) set. On a live update_settings call with bad thresholds the router was left advertising routing_strategy="lar1" with no custom selector bound, while the previous strategy's selectors stayed registered. Build (and validate) the strategy before mutating any router state, so a threshold error leaves the router exactly as it was. Add a regression test that asserts a failed switch keeps the prior strategy intact. --------- Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com> Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: xbrxr03 <abrarhabib03@gmail.com> Co-authored-by: hayden <sktpghks138@gmail.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Wassim Badraoui <98709649+Wassbdr@users.noreply.github.com> Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com> Co-authored-by: Neimar Avila <neimar.avila@gmail.com> Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> Co-authored-by: Jerry-Scintilla <jerrycaocao@126.com> Co-authored-by: AlexBGoode <me.at.forum@gmail.com> Co-authored-by: Carsten Boloz <cdboloz1@gmail.com> Co-authored-by: jesco <team@srswti.com> Co-authored-by: Praveen Ghuge <pghuge@digitalex.io> Co-authored-by: Jim Smith <j.h.smith@ieee.org> Co-authored-by: David J. M. Karlsen <david@davidkarlsen.com> Co-authored-by: Vedant Agarwal <43557509+Vedant-Agarwal@users.noreply.github.com> Co-authored-by: Srivatsa Kamballa <skamb10@uic.edu> Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn <hoangson091104@gmail.com> Co-authored-by: Tal Marian <tal.marian@island.io> Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: bhumikadangayach <139267865+bhumikadangayach@users.noreply.github.com> Co-authored-by: Ewertonslv <ewertoncom297@gmail.com> Co-authored-by: carlsonchik <carlsonchik@users.noreply.github.com>	2026-06-26 09:17:44 -07:00
Sameer Kankute	cfcdf8714a	feat: litellm oss 110626 (#30202) ... * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) (#29775) * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) Adds first-class support for the gpt-realtime-whisper streaming speech-to-text model, which uses the Realtime transcription session API rather than the file-based /audio/transcriptions path. Model registration: registers gpt-realtime-whisper and azure/gpt-realtime-whisper with audio-duration pricing (input_cost_per_second = 0.017/60, matching the published $0.017/minute input audio rate). REST endpoint: implements POST /v1/realtime/transcription_sessions (plus /realtime and /openai/v1 aliases) to mint an ephemeral transcription session for the WebRTC flow. Adds request/response types, OpenAI and Azure URL builders, a shared base handler (refactored from the client_secrets handler), the acreate_realtime_transcription_session SDK function, and route registration. The proxy encrypts the ephemeral key returned under client_secret.value and records the session type in the token so the follow-up /realtime/calls replays type=transcription rather than type=realtime. WebSocket: forwards intent=transcription through to the Azure handler (OpenAI already received it) with URL-encoding, so gpt-realtime-whisper opens a transcription session. Transcription-only sessions no longer trigger an erroneous response.create. Cost tracking: transcription sessions emit no response.done events; their usage arrives on conversation.item.input_audio_transcription.completed as {type: duration, seconds}. That usage is captured out-of-band (usage only, no transcript duplication) and billed by input_cost_per_second, with a token-billed fallback for token-priced transcription models. Adds tests for pricing math, URL builders, request/response types, the proxy route and SDK function, WebSocket intent forwarding, transcription-session streaming behavior, and the /realtime/calls session-type replay. * Address PR review: URL-encode all Azure WS query params; forward query_params through provider_config branch * Address PR review: session_type validation, model auth fix, cost perf, billing fallback, detail/docs cleanup * Improve test coverage: detection from backend, error paths, unknown usage type, resolved_model None * Backport realtime transcription websocket fixes * Enforce authorized realtime transcription model * Enforce realtime transcription model access * Enforce realtime resolved model scopes * Enforce WebRTC transcription model scope * Lazy evaluate debug log in pass-through endpoint (#30177) * Pass through debug lazy logging * fix(proxy): convert remaining eager pass-through debug logs to lazy formatting * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint (#30157) * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint The Parallel Search API moved from /v1beta/search (processor: base/pro, parallel-beta header) to /v1/search (mode: turbo/basic/advanced, no beta header). Request fields moved too: max_results, source_policy, and excerpt settings are now nested under advanced_settings, and source_policy uses include_domains/exclude_domains. The v1 response returns publish_date per result, which now maps to SearchResult.date instead of being hardcoded to None. The legacy processor param is mapped to the equivalent mode so existing callers keep working. * fix(parallel_ai): default mode to basic and simplify param handling The v1 API defaults to advanced mode when mode is omitted, while v1beta defaulted to the base processor. Without an explicit default, callers who pass no mode would be silently upgraded to a tier costing 2.25x more while litellm's cost map reports the basic-tier price. Sending mode=basic preserves the v1beta default and keeps cost tracking accurate. Also replaces the handled_params set with pop-as-consumed param handling so mapped params no longer need to be tracked in two places, and extends the tests to pin the default mode, processor=base mapping, mode-over-processor precedence, and top-level v1 param passthrough. * fix(parallel_ai): avoid double /v1 when api_base is already versioned A PARALLEL_AI_API_BASE like https://api.parallel.ai/v1 previously produced .../v1/v1/search. Strip a trailing /v1 before appending the search path and cover the api_base variants with a parametrized test. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(focus): add Mavvrik destination for FOCUS export (#29935) * fix: preserve responses streaming flag (#30189) * fix: preserve responses streaming flag * test: cover async responses streaming flag * fix(spend/daily-activity): stable offset pagination via id tiebreaker (#30164) (#30167) date alone is not a unique sort key for LiteLLM_DailyUserSpend or LiteLLM_DailyTeamSpend (many rows per date: api_key x model x model_group x provider x endpoint). Offset pagination over a non-unique sort landed on arbitrary boundaries, so a client paging through all results and summing per-page metrics (the Usage dashboard) got non-deterministic totals - sometimes inflated, sometimes deflated, different at different page_size values. Adding the row's UUID id (present on both tables) as a secondary sort gives every page a stable cursor. order=[{date desc}, {id asc}]. Fixes #30164 * fix(oci): inject a default maxTokens so omitted max_tokens doesn't truncate responses (#30018) * fix(oci): inject default maxTokens so omitted max_tokens doesn't truncate OCI GenAI applies a tiny server-side maxTokens default (~20 tokens) when the request omits it, so any call that doesn't send max_tokens comes back cut off mid-string with finishReason "length". MLflow judges never send max_tokens, so their JSON responses arrived as unterminated strings and json.loads failed in MLflow's gateway adapter. When no maxTokens/maxCompletionTokens target is set, inject DEFAULT_OCI_CHAT_MAX_TOKENS (env-overridable, defaults 4096), mirroring the Anthropic config's default-max-tokens behaviour. An explicit max_tokens still wins, and reasoning models still route to maxCompletionTokens. Used a fixed default rather than the catalog max_output_tokens because the catalog value is unreliable for some models (grok-4 reports max_output_tokens equal to its context window, not a real output cap, which would risk 400s). Adds TestOCIDefaultMaxTokens covering Cohere and generic injection, the explicit-override case, and the reasoning maxCompletionTokens branch. * test(oci): e2e regression that omitted max_tokens isn't truncated Real-proxy integration test asserting a chat completion that omits max_tokens completes with finish_reason "stop" instead of being cut off at OCI's ~20-token server default. Fails before the maxTokens-default injection (finish_reason "length", ~19 tokens), passes after. * test(oci): update cohere default-params test for injected maxTokens test_cohere_default_parameters asserted no maxTokens was injected, encoding the old behaviour where OCI's ~20-token server default truncated responses. Now that transform_request injects DEFAULT_OCI_CHAT_MAX_TOKENS, assert maxTokens equals that default while the other params (topK/topP/frequencyPenalty) stay pass-through with no hardcoded default. * fix(oci): make DEFAULT_OCI_CHAT_MAX_TOKENS a plain constant Drop the os.getenv override. The env knob was not requested and introducing a new env var forced a cross-repo dependency on litellm-docs (test_env_keys.py validates every referenced env var against the docs table there). A plain 4096 constant keeps the PR self-contained; callers who want a different limit pass max_tokens explicitly per request. * fix(oci): route all OpenAI commercial models to maxCompletionTokens OCI serves OpenAI models (gpt-4.1, gpt-5.1 through 5.5, o-series) that the litellm catalog doesn't track, so the supports_reasoning lookup returned False for them and the provider sent maxTokens, which the reasoning families reject with HTTP 400. With the injected default maxTokens this broke every request to those models, not just ones with an explicit max_tokens. Route the whole openai.* vendor prefix to maxCompletionTokens since OpenAI accepts max_completion_tokens on every chat model; the openai.gpt-oss-* open weights are served by OCI's own stack and keep maxTokens. Verified live against gpt-5.2, gpt-5, gpt-4o, gpt-4.1, gpt-oss-120b, llama-3.3, command-a and grok-3-mini * test(oci): hoist transformation imports and drop unused ones Makes the generic-chat test file ruff-clean: the per-test local imports of OCIChatConfig/OCIVendors shadowed the module-level import (F811) and left it unused (F401), and json plus three OCI type imports were never referenced * fix(oci): translate response_format json_schema to OCI's accepted shape (#29691) * fix(oci): translate response_format json_schema to OCI's accepted shape OCI GenAI rejected every json_schema response_format with HTTP 400 "Please pass in correct format of request", which broke structured-output callers such as MLflow LLM judges (they always send a json_schema). The provider forwarded OpenAI's raw json_schema body unchanged. For GENERIC models OCI's ResponseJsonSchema accepts only name/description/schema/isStrict, so OpenAI's `strict` key (and any other extra) 400s the request; the key must be renamed to isStrict and the body whitelisted. For Cohere models there is no JSON_SCHEMA type at all; the schema has to ride on JSON_OBJECT as {"type": "JSON_OBJECT", "schema": ...}. Cohere type values must also be the canonical uppercase TEXT/JSON_OBJECT. _normalize_response_format now branches by vendor and emits the exact shape each one accepts (verified live against OCI GenAI for Cohere, Meta, Gemini and Grok). Drops the unused, incorrect Cohere response-format pydantic models. Two existing tests asserted the broken behavior (lowercase type, raw jsonSchema on Cohere); they are rewritten to assert the corrected shape, and generic/Cohere json_schema regression tests are added. * fix(oci): raise early on json_schema response_format with no body A GENERIC model request with {"type": "json_schema"} and no json_schema object fell through to the JSON_OBJECT branch and emitted a bodyless {"type": "JSON_SCHEMA"}, which OCI rejects with an opaque HTTP 400. Raise a descriptive 400 at translation time instead. Cohere is unaffected since it always maps to JSON_OBJECT. * test(oci): gateway integration test for response_format json_schema Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): accept default n=1 on Cohere instead of hard-failing (#29705) * fix(oci): accept default n=1 on Cohere instead of hard-failing Cohere on OCI has no numGenerations field, so n was mapped to False and map_openai_params raised "param `n` is not supported on OCI" whenever a client sent n. But n=1 (and None) is the OpenAI default single-generation request, which every OCI model produces anyway, so standard clients that always send n=1 (such as the MLflow gateway) were rejected with a 500. Drop n=1/None silently for Cohere; only n>1 is genuinely unsupported and still raises (or drops under drop_params). Generic models are unaffected and keep numGenerations, including n>1. * docs(oci): explain why n is not advertised for Cohere despite tolerating n=1 * test(oci): gateway integration test for Cohere default n=1 Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): drop max_retries instead of hard-failing on OCI (#29727) max_retries is a litellm-level control param (litellm applies retries itself), not a generation param OCI accepts. The provider mapped it to False and raised "param `max_retries` is not supported on OCI" whenever it was present. The litellm proxy injects max_retries on every request, so any OCI call through the proxy 500'd unless drop_params was set. Drop max_retries silently in map_openai_params. Adds a unit test (Cohere and generic) and a gateway integration test that a plain request succeeds through a proxy without drop_params. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): rehydrate metadata JSONB text on ui_view_spend_logs (#29682) Fixes #29674. `/spend/logs/ui` raw-SQL path returns the JSONB metadata column as a string — prisma's query_raw skips the ORM-layer hydration. The UI reads metadata.status / metadata.error_information as object fields, so provider-failure rows look like successes. Fix: json.loads the metadata field right after query_raw, fall back to {} on malformed JSON. 3 existing error-code/error-message tests called json.loads on response.data[0]["metadata"] — they were leaning on the bug. Updated to read the dict directly. Plus 2 new regression tests (failure metadata roundtrip + invalid-json fallback). Reverting the fix makes both new tests fail with AssertionError: metadata should be dict, got <class 'str'>. * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) (#30020) * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) * fix: refund max_parallel_requests on disconnect from outer streaming generators The cancellation refund previously lived in async_post_call_streaming_iterator_hook, but that hook is nested inside the outer streaming generators and a nested async generator only receives GeneratorExit on garbage collection (non-deterministic). With only the v3 limiter enabled, /chat/completions also bypasses the hook entirely (needs_iterator_wrap() is false). Move the release into async_data_generator and async_streaming_data_generator, the generators Starlette closes on client disconnect, so the refund fires deterministically on every streaming route. Warn when no event loop is running, and document the window TTL refresh on the decrement * fix(mcp): propagate model into model_call_details for passthrough tool calls (#30122) * fix(mcp): propagate model into model_call_details for passthrough tool calls The @client decorator on call_mcp_tool creates the logging object via function_setup without a model kwarg, so model_call_details["model"] starts as None. execute_mcp_tool only set logging_obj.model as an instance attribute, which the spend-log writer never reads (it reads kwargs["model"] from model_call_details). MCP passthrough tools/call rows therefore persisted with model="" while list_tools rows showed "MCP: list_tools", degrading the Logs UI display and bucketing all MCP tool spend under an empty model in DailyUserSpend. Propagate the model into model_call_details alongside the existing attribute assignment so the StandardLoggingPayload and SpendLogs writer pick it up. Covers the /mcp passthrough, REST /mcp-rest/tools/call, and orchestrated paths (the latter already passed model into function_setup, so this is a no-op there). * test(mcp): trim regression test docstring * fix(mcp): surface upstream challenges for delegated OAuth (#30124) * fix(mcp): surface upstream challenges for delegated OAuth * docs(mcp): clarify delegated upstream auth comments * perf(benchmarks): add CPU timing metrics to streaming benchmark (#29980) * Add CPU timing metrics to streaming benchmark * Fix spacing around timing sample dataclass * fix(gemini): don't emit empty choices on metadata-only stream chunks (#29167) web_search + reasoning makes Gemini stream mid-chunks that carry only grounding/thought metadata — no content part, no finishReason. _process_candidates skips content-less candidates and the existing fallback only ran when finishReason was set, so choices stayed empty and the downstream streaming handler raised IndexError on choices[0]. Emit an empty-delta choice for content-less chunks regardless of finishReason. Fixes #28884 * fix(key): allow /key/update to clear budget_limits with [] or null (#30085) * Fix /key/update rejecting budget_limits clear requests with HTTP 400 Sending budget_limits: [] or null to /key/update returned HTTP 400, so once a key had budget windows the last one could never be removed. prepare_key_update_data only json.dumps'd budget_limits when the value was truthy, so [] and None passed through raw to the Prisma Json? column; jsonify_object only serializes dicts, and prisma-client-py has no DbNull sentinel for Json? writes, so Prisma rejected both shapes. Serialize the clear case explicitly as the JSON literal null, matching how memory_endpoints encodes metadata for the same column type. Truthy values keep the existing reset_at window initialization path. Fixes #30067. * Require admin access for budget_limits changes on /key/update Clearing budget_limits via [] or null is a budget mutation, but _validate_update_key_data only counted max_budget and spend as budget changes before deciding whether to skip _check_key_admin_access. A non-admin key owner or a team member with /key/update could therefore remove a key's per-window spend caps without admin authorization. Treat any explicit budget_limits value in the request (set, change, or clear) as a budget change so it gates through the same admin check as max_budget. model_fields_set is used because an explicit null is indistinguishable from an omitted field by value alone. * fix(proxy): persist guardrail info in spend logs for /v1/responses (#30092) Pre-call guardrail blocks on /v1/responses wrote guardrail_information as null in LiteLLM_SpendLogs because _handle_logging_proxy_only_error splits request_data by LoggedLiteLLMParams keys and litellm_metadata, where the Responses API stores request metadata including standard_logging_guardrail_information, was not among them. It fell into optional_params, so merge_litellm_metadata never saw it. Add litellm_metadata to LoggedLiteLLMParams so it routes into litellm_params the same way metadata does on the chat completions path Fixes #28971. * fix(proxy): handle non-standard SSE frames in Anthropic passthrough logging (#26000) Some third-party Anthropic-compatible providers emit non-standard SSE frames (OpenAI-style [DONE] sentinels, non-JSON keep-alive lines) in streaming responses. These caused json.JSONDecodeError in _build_complete_streaming_response, breaking the passthrough logging pipeline so the request was never logged or billed. Skip whole-line 'data: [DONE]' sentinels and catch JSONDecodeError per event. Matching the full line (not a substring) keeps a valid chunk whose text payload contains '[DONE]' from being dropped. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(newrelic): Add New Relic extension (#26989) * initial New Relic integration. * Minor fixes for basic observability. * Implemented basic support for the success path. Generates New Relic custom events needed by the AI Monitorin interface. * Supportability metric is sent on first request. * Emit supportability metric every hour instead of once a day. * Add the start/end times to the messages before sending them so that the start time and end time reflect the correct time and both are not set to 'now'. * Make use of `turn_off_message_logging` configuration that is available by default from CustomLogger. * Enabling New Relic agent to be wired when docker container starts if an environment variable is set. * If we cannot find trace information, send the AI events without the trace ID attached. * Use a fake trace_id if we cannot find one. * Implementing a configuration so that users can use litellm configuration to disable sending LLM messages to New Relic. There is a second method to do this via New Relic env var. * Mised file. * Cleaning up logic to turn off recording content via either the LiteLLM configuration or an env var. * Removing debugging. Fixed logic / comments around how often to send supportability metric. * Initial version of public doc for New Relic. * Use a proper name for the doc file. * Updating newrelic.md document. * Updating LiteLLM documentation for New Relic extension. * Moving New Relic imports into the methods to support unit tests. * Adding unit tests for the New Relic extension. * Updating linting and the unit tests that are not running in the CI environment. * Address reviewer feedback on New Relic integration. - Fix _record_error_metric to use app.record_custom_metric() instead of module-level newrelic.agent.record_custom_metric() so the call works outside of an active transaction context - Remove unreachable except ImportError block in _get_trace_context - Update stale "23 hours" comment to "27 hours" (matches 97200s threshold) - Remove commented-out debug code from _process_success - Fix docs typo: NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STOREDA -> NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STORED - Update TestRecordErrorMetric to verify app.record_custom_metric call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Reformating for the linter. * Addressing additional automated feedback. - Removed a legacy comment about the New Relic header - Reordered imports in one file - Switched another file to use the import at the top of the file instead of inline when used - Added unit tests for untested methods that were identified * Addressing new feedback. - Proper handling of time to floats. Created a util method and updated code to use it. - added the missing guard to ensure the app is enabled * Addressing feedback. - When an error occurs, still check if the periodic supportability metric should be emitted - Added a check to ensure the extension is ready in the error handler to match _process_success * Updating the NR event timestamps to more accurately reflect when the messages were generated. * Addressing feedback for potential better practice. * Addressing feedback on accessing default values. Added tests for most of these cases. * Adding a new catch exception block based on feedback. * Addressing feedback about a potential issue around a timestamp for the supportability metric. * Addressing minor feedback on length of generated, fallback traceId. * Addressing feedback. - A few more cases were found where the dictionary access might not return the correct value. - Handling cases where `traceparent` is not lower cased * Addressed feedback where the newrelic options might not apply correctly. * Addressing some feedback. * Addressing feedback. * Validating testing / formatting for our changes. * Updating linting, adding tests, defining data type for UI. * Configuration for the logging callback definition. * Adding a newrelic image for the UI to use. * Putting the New Relic callback in proper alphabetic order. * Copying the logo to a committed output directory so it shows up in a locally built container. * Adding missing definition of new env vars that were causing a build failure. * Addressing automated feedback from greptile. * Adding a few more unit tests to increase the code coverage just a bit more. * Additional unit tests to push coverage to almost 90%. * Adding a custom newrelic docker image build process. This removes the need to add the newrelic agent to the core litellm container or dependencies. * Clarifying message when the New Relic agent is not installed and someone is trying to use the newrelic extension. Either use the proper image when using docker, or install the agent manually when running from source. * Ensuring pip is available to install the New Relic agent. * Updating the definition and handling of traceId (no spanId). Clarifying behavior of env vars vs UI configuration for the newrelic extension. * Removing entries from the New Relic logger configuraiton UI as these values must be set as part of running the image. * Removing a stale doc file that has moved to the litellm-docs repo. Cleanup of Dockerfile to remove a LABEL that was incorrect. * Updating container image name to be the best guess for the new name. * Addressing feedback from greptile. - Added a comment around token_count=0 - Updated the boolean parser to allow a wider set of options which matches existing patterns in other parts of LiteLLM. * Removing option for a separate New Relic container image. The agreement is to handle this in the New Relic integration docs. * Updating error message when New Relic agent is not available. * Wiring in the test message from the LiteLLM callback UX. * Missed saving one of the file conflicts. * Fixed a lint error I introduced. Somehow, I dropped another string and now added it back. * Adding newrelic to the schema definition. * Added an admin check on the call before sending test message as mentioned by the AI code review. * Updating to use should_redact_message_logging(kwargs) as part of the logic to determine if message content should be sent to New Relic or not. This still uses the `record_content` property as well, but both have to be true in order for content to be included. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add Azure AI Foundry DeepSeek V3.1 and V4 Pro/Flash global pricing to cost map (#30134) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(logging): translate Responses bridge result to ModelResponse for spend logs (#28985) PR #29394 fixed the AnthropicResponse.model_validate crash for the streaming anthropic_messages -> OpenAI Responses bridge by unwrapping terminal events and returning the inner ResponsesAPIResponse. The spend_logs row lands and usage/cost are correct, but the row's response field stores the Responses API shape (output[...].content[...].text). The proxy UI Logs tab reads response.choices[0].message via parseMessages in prettyMessagesUtils.ts with no fallback for the Responses shape, so the OutputCard renders "No response data available" for every cross-routed call. The same shape mismatch affects every downstream consumer of spend_logs that assumes the canonical chat-completion shape This change keeps the unwrap from #29394 but routes the resulting ResponsesAPIResponse (and the bare-response non-streaming path) through LiteLLMResponsesTransformationHandler.transform_response, which is the same conversion already used by the chat-completion Responses bridge. Spend_logs now stores a ModelResponse with choices[0].message.content, so the UI and other consumers see the assistant text. On a translation failure (eg. empty output on an incomplete response) the handler falls back to a minimal ModelResponse carrying model and usage so the row still lands rather than being dropped as a Non-Blocking error Also corrects a stale comment in the Responses adapter that implied the call type was reclassified to acompletion; the code preserves anthropic_messages and the success handler translates back to ModelResponse for the row Fixes #28595 * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions (#30024) * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions The `/v1/messages` -> `/v1/chat/completions` streaming adapter (`AnthropicStreamWrapper`) silently dropped the first non-empty delta of every content block that started via a *transition* (e.g. text -> tool_use -> text, text -> thinking). When an upstream chunk both triggers a new content block (its type differs from the active block) and carries that block's first delta, the wrapper emitted `content_block_stop` -> `content_block_start` and then only re-queued the trigger chunk when it was an `input_json_delta` (bundled tool args). The synthesized `content_block_start` always carries an empty body, so the first `text_delta` / `thinking_delta` was lost — the client output started from the second token (e.g. "Hi, how can I help you?" rendered as ", how can I help you?", or text resuming after a tool call lost its first sentence). This is especially visible with Claude Code-style clients that consume Anthropic Messages streaming events strictly. Fix: re-queue the trigger chunk's translated delta whenever it carries non-empty content (text/thinking/signature/tool args), via a shared `_trigger_delta_has_content` helper used by both the sync and async paths. Empty trigger deltas are still suppressed so no spurious empty `content_block_delta` is introduced. Fixes #30014 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(anthropic-adapter): cover all _trigger_delta_has_content branches Add a direct parametrized unit test for the re-emit predicate so every delta type (text/input_json/thinking/signature), the empty-payload guards, and the malformed/non-delta cases are exercised independently of upstream chunk translation. Raises patch coverage for the new helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat: add opt-in healthy_only filter to GET /v1/models (#30130) * feat: add opt-in healthy_only filter to GET /v1/models Adds an opt-in `healthy_only=true` query parameter to GET /v1/models and GET /models that hides models whose backing deployments are all marked unhealthy by background health checks. - Add Router.async_get_fully_unhealthy_model_names(), mirroring the semantics of get_fully_blocked_model_names(): a model is hidden only when every backing deployment is unhealthy and the health state is not stale (fail open otherwise). - Reuses the existing DeploymentHealthCache populated by _run_background_health_check(), so no new health state is introduced. - No-op when allowed_fails_policy is set, mirroring _async_filter_health_check_unhealthy_deployments semantics. - team_public_model_name aliases are aggregated alongside model_name. - Hiding is presentation-only; default behavior is unchanged. Fixes #30128 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: address Greptile review notes - Note team-alias asymmetry vs get_fully_blocked_model_names - Debug-log when healthy_only is set but no health state is available Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Dedupe team soft budget alerts by team_id instead of token (#30097) _team_soft_budget_check sends type="soft_budget" alerts with event_group=TEAM, but SoftBudgetAlert.get_id always returned the request token. The alert cache key was therefore scoped per virtual key, so every active key in a team over its soft budget fired its own alert within budget_alert_ttl. Branch on event_group so team-level alerts dedupe by team_id, matching TeamBudgetAlert, while key and project level alerts keep per-token dedupe. Fixes #27398. * feat(bedrock guardrails): support contextual grounding qualifiers (request-side) (#30057) * test: add failing tests for Bedrock contextual grounding (request-side) Drive the request-side of Bedrock contextual grounding: callers tag message content blocks as grounding_source/query, the post_call hook assembles an ApplyGuardrail(OUTPUT) call carrying source + query + response(guard_content), and the bedrock converse transform must render the tags as prompt text instead of silently dropping them. Non-grounding payloads must stay byte-identical. * feat(bedrock guardrails): support contextual grounding qualifiers Bedrock contextual grounding scores a model response against a reference source and the user query, expressed via a per-content-block `qualifiers` array on ApplyGuardrail. The guardrail hook previously sent plain text only, so grounding could not be driven through it even though the response-side contextualGroundingPolicy parsing already existed. Callers now tag message content blocks `{"type":"grounding_source"}` / `{"type":"query"}` (mirroring the existing `guarded_text` marker). On the generate path the bedrock converse transform renders them as plain text; at post_call the hook harvests them from the request and assembles one ApplyGuardrail(OUTPUT) call carrying grounding_source + query + the response (as guard_content). Requests without these tags produce a byte-identical payload, so existing behaviour is unchanged. * Feat(guardrail): Adding support for custom Ovalix guardrail (#21887) * Feat(guardrail): Adding support for custom Ovalix guardrail * Internal CR comments fixes * greptileai comments fixes * fix conflict * fixes * fix sha256 * clarify Ovalix actor-id hash is for normalization, not PII protection * fix(github_copilot): normalize per-event item_id in /responses streaming (#30072) GitHub Copilot's native /v1/responses stream assigns a different item_id to every event of a single output item (output_item.added, the part.added / delta / done events, and output_item.done). Spec-strict clients like the Vercel AI SDK key streaming parts by item_id and abort with "reasoning part <id> not found" / "text part <id> not found" when a delta references an unregistered id. Override transform_streaming_response in GithubCopilotResponsesAPIConfig to anchor every event of an output item to the id from its output_item.added. Copilot accepts that id paired with the final encrypted_content on the next turn, so multi-turn replay is unaffected. Fixes #30071 * feat: add /model/block and /model/unblock endpoints (#30125) * feat: add /model/block and /model/unblock endpoints Add dedicated proxy-admin POST /model/block and /model/unblock endpoints over the existing blocked flag on LiteLLM_ProxyModelTable, mirroring the /key/block and /key/unblock pattern. Calling a model whose deployments are all blocked now returns a clear 403 "Model is blocked" instead of a generic no-deployment error, including direct-dispatch route types (e.g. eval) via a pre-route guard. Includes audit-log entries for block/unblock and unit tests. Closes #29742 Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * chore: regenerate dashboard API types for model block/unblock endpoints Regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts from the proxy OpenAPI spec (npm run gen:api) so it includes the new endpoints. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: widen router block-helper param type and add direct unit tests Type the _are_all_deployments_blocked deployments parameter to match its callers (DeploymentTypedDict) so mypy passes, and add tests/test_litellm/test_router_block_helpers.py with direct unit tests for the three block helper methods so router_code_coverage recognizes them. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: restore type-ignore on messages arg after black reflow Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * refactor: raise model-block 403 in proxy layer, not SDK Router Keep the SDK Router's documented behavior for blocked deployments (filtered -> "no healthy deployment") and move the 403 PermissionDeniedError into the proxy layer (route_llm_request), where model blocking is an admin concept. This avoids a backwards-incompatible 403 for SDK users who set blocked=True on their own deployments, per maintainer review. Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: add week unit support to get_next_standardized_reset_time (#30100) * fix: add week unit support to get_next_standardized_reset_time The function handled d/h/m/s/mo units but silently fell through to the default next-midnight branch for the w (week) unit. This was inconsistent: _extract_from_regex already accepted w in its character class, and duration_in_seconds already returned value * 604800 for it. Add the missing elif unit == 'w' branch that delegates to _handle_day_reset with value * 7, which reuses the existing Monday- alignment logic for 1w and the generic N-day-from-midnight path for larger multiples. Add test_week_based_resets covering 1w from a Wednesday (expects next Monday) and 2w from a Monday (expects 14 days forward at midnight). Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * test: exercise relative week semantics with non-Monday base dates + add docstring Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * fix: black formatting and remove undocumented MAVVRIK_FOCUS_FREQUENCY env var * fix: black formatting with correct version and sync schema.d.ts for healthy_only param * fix: resolve mypy errors and add transcription_sessions to JSON schema endpoint enum * fix: restore MAVVRIK_FOCUS_FREQUENCY guard and exclude it from docs key scan * fix: address Greptile P2 comments - move constant, use UTC datetime, skip redundant team lookup * revert: restore original team lookup logic in can_key_call_resolved_model --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: nina-hu <nina.huuu@gmail.com> Co-authored-by: Sahith Jagarlamudi <104647530+s-jag@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com> Co-authored-by: alex107ivanov <30668368+alex107ivanov@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com> Co-authored-by: Teo Xian Zhong Augustine <35527068+auggie246@users.noreply.github.com> Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: Saksham Maggo <122939011+SakshamMaggo@users.noreply.github.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Kelvin <leikaiwei@outlook.com> Co-authored-by: Josh Bonczkowski <josh.bonczkowski@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: M. Dennis Turp <mdturp@pm.me> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Piotr Minkina <piotrminkina@users.noreply.github.com> Co-authored-by: Martín Alcalá Rubí <martin@tryolabs.com> Co-authored-by: T. Kobayashi <13004314+nix-tkobayashi@users.noreply.github.com> Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: Shalom <shalom@ovalix.io> Co-authored-by: codgician <15964984+codgician@users.noreply.github.com> Co-authored-by: FugoP <kim@pomsora.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-06-11 22:30:26 -07:00
milan-berri	411bd3da5b	feat(vantage): include organization metadata in FOCUS Tags export (#28184) ... * feat(vantage): include organization metadata in FOCUS Tags export Join LiteLLM_OrganizationTable when building Vantage/FOCUS export rows so organization_id and organization_alias appear in Tags for org-level filtering. Co-authored-by: Cursor <cursoragent@cursor.com> * test(focus): include api_requests in organization Tags tests FocusTransformer now requires api_requests after staging merge; add the column to test fixtures so integrations CI can run the Tags assertions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>	2026-06-09 02:59:21 +03:00
Sameer Kankute	32c88ca74f	Litellm oss staging 080626 (#29932) ... * feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (fixes #29665) (#29788) * feat(responses): add default no-op sign_request to BaseResponsesAPIConfig * feat(responses): call sign_request after body is final, send signed bytes when signed * feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path) * test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency * feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer * docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route * test(responses): cover fake-stream signing order and mantle bearer arg/env precedence * fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance * fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked * fix(bedrock_mantle): sign the compact Responses route too, not just create * fix(github-copilot): route per-model on /v1/responses based on model info (#29747) * feat(focus): add GCS destination for FOCUS export (#29751) * test: add failing tests for FocusGCSDestination * feat: add FocusGCSDestination reusing GCSBucketBase auth * feat: register FocusGCSDestination in factory; export from __init__ * fix(focus): preserve GCS_PATH_SERVICE_ACCOUNT when service_account_json not in config * style: apply Black formatting to gcs_destination and tests * style: apply Black formatting to factory.py * fix(bedrock): omit empty additionalModelRequestFields and system from Converse API payload (#29565) Amazon Nova Pro (and other strict Bedrock models) return 400 Malformed input request when additionalModelRequestFields: {} or system: [] are present in the payload. Both fields are optional in CommonRequestObject (total=False) and must be omitted rather than sent as empty structures. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking (#29730) * fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible Azure OpenAI resources created via the newer "Azure AI Foundry" / Cognitive Services pathway live on `*.cognitiveservices.azure.com` subdomains, not the older `openai.azure.com`. Both are valid Azure OpenAI surfaces in production today. The OpenAI pass-through cost-tracking handler hard-codes only the older hostname in five places (four `is_openai_*_route` methods on OpenAIPassthroughLoggingHandler, plus is_openai_route on PassThroughEndpointLogging). As a result, calls from newer Azure deployments are silently classified as "not an OpenAI route", the dispatch into the cost-tracking handler is skipped, and tokens/cost never get extracted into LiteLLM_SpendLogs — the row gets written with prompt_tokens=0, completion_tokens=0, spend=0, model='unknown'. Reproduced 2026-06-04 against a real Azure OpenAI deployment on `*.cognitiveservices.azure.com` proxied through LiteLLM v1.88.0. Fix: factor the hostname check into a single helper `_is_openai_compatible_host` listing all three recognized surfaces (api.openai.com, openai.azure.com, cognitiveservices.azure.com), and have all five call sites delegate to it. Purely additive — never weakens recognition for the originally-supported hostnames. Adds a test `test_is_openai_route_recognizes_cognitiveservices_azure_com` that exercises all four `is_openai_*_route` static methods against `*.cognitiveservices.azure.com` URLs (positive cases per route + a small cross-route negative to confirm route-specific path matching still works on the new hostname). Out of scope for this PR (separate followup): - `openai_passthrough_handler` calls chat/completions `transform_response` on Responses API payloads (`output:` not `choices:`), which throws inside the dispatch and drops the SpendLogs row entirely. Recognized + tracked separately. * ci: trigger fresh run Empty commit to re-run checks. The previous auth-and-jwt failure was a transient HuggingFace Hub 429 rate-limit hitting tokenizer downloads in tests/proxy_unit_tests/test_custom_tokenizer_bug.py — unrelated to this PR's scope (hostname recognition in pass-through cost tracking). No code change. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(responses): preserve forced-function tool_choice name in Responses to Chat transform (#29812) The Responses API forces a specific function with a top-level name ({"type": "function", "name": "X"}), but _transform_tool_choice only handled the nested Chat Completions shape and fell through to returning "required" for the flat form, silently dropping the function name and degrading a forced function call to force-any-tool. Map the flat Responses shape to the nested Chat shape, keeping the "required" fallback when no name is present. * Preserve x-anthropic-billing-header system blocks for first-party Anthropic (#29584) * Preserve x-anthropic-billing-header system blocks for first-party Anthropic PR #20951 strips system blocks beginning with "x-anthropic-billing-header:" for every Anthropic target. That block is how the first-party Anthropic API recognizes Claude Code subscription (OAuth) traffic, so dropping it makes requests that carry only that block, such as the auto-mode tool-safety classifier, fail with a misleading 429 rate_limit_error; normal turns still work because they also carry the "You are Claude Code" identity block. Gate the strip behind should_strip_billing_metadata(), defaulting to False on the first-party AnthropicConfig and AnthropicMessagesConfig so the block is kept, and overridden to True on the providers that reach these transforms and reject the block (Bedrock platform, Vertex, Azure for the chat path; Minimax, Azure, DeepSeek for the messages path). Behavior for those providers is unchanged. * Strip billing header on Bedrock invoke and Vertex messages pass-through Two more subclasses reach the gated strip but inherited keep-by-default. AmazonAnthropicClaudeConfig (Bedrock invoke) calls AnthropicConfig.transform_request, which calls translate_system_message, and VertexAIPartnerModelsAnthropicMessagesConfig (Vertex messages pass-through) calls super().transform_anthropic_messages_request. Override should_strip_billing_metadata() to True on both. Add a parametrized test asserting the flag for every first-party base (False) and provider subclass (True), covering all overrides, plus a translate_system_message regression test for the Bedrock invoke path. * fix(cache): log hashed cache keys (#29890) * fix(ui): save routing groups as list (#29889) * Revert "fix(ui): save routing groups as list (#29889)" (#29928) This reverts commit 9b1f78ffa7. * feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider (#29842) * feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider Registers parasail in the openai_like JSON provider loader with both /v1/chat/completions and /v1/responses support. Parasail's Responses API rejects store:true and any request that omits store, so the loader gains a force_store_false special_handling flag; the parasail entry sets it and the generated Responses config overrides store=false on every call. This keeps callers from hitting "State storage not supported" and matches what Parasail's docs require. Adds the PARASAIL enum value, listing under openai_compatible_providers, provider documentation at docs/my-website/docs/providers/parasail.md, and a focused unit test file under tests/test_litellm/llms/parasail/ that covers JSON registration, chat URL construction, Responses URL construction with PARASAIL_API_BASE override, and the force_store_false regression in both the caller-sent-store=true and caller-omitted cases. * fix(parasail): register in provider_endpoints_support, drop in-repo docs Greptile review feedback. The provider doc belongs in the litellm-docs repo, not this one's docs/my-website tree; removing it here. Adds the parasail entry to provider_endpoints_support.json so the check_provider_folders_documented.py CI check passes (chat_completions and responses true; others false). * fix: normalize Anthropic passthrough server tool usage (#29827) * test(anthropic): cover server_tool_use dict cost tracking * fix: normalize Anthropic server tool usage (cherry picked from commit 982f726bed) * fix: keep server tool usage subscriptable (cherry picked from commit 70280b9b27) --------- Co-authored-by: Genmin <joey@joeyroth.com> * fix(proxy): fix typo generic_role_mappoings -> generic_role_mappings in ui_sso.py (#29753) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(proxy): add disable_budget_reservation general setting (#27639) (#29493) * feat(proxy): add disable_budget_reservation general setting (#27639) * feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639) * docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639) * ci: re-trigger flaky docker build (prisma generate ECONNRESET) * fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639) * feat(gemini_tts): adding support to Gemini TTS languageCode parameters (#29623) * Adding support to Gemini TTS Language Code parameters * Mapping Gemini TTS languageCode param in Docstring * Use snake_case for language_code input keyMapping Gemini TTS languageCode param in Docstring * Restoring files modified under enterprise/litellm_enterprise due to lint/formatting checks --------- Co-authored-by: João Garrido <joaogarrido@google.com> * feat(guardrails): capture user and model metadata in CrowdStrike AIDR (#29517) * fix(proxy): require OpenAI path segment for shared Azure Cognitive Services domains Address Greptile review: the `*.cognitiveservices.azure.com` / `*.openai.azure.com` domains are shared by every Azure Cognitive Service (Speech, Vision, Language, ...), so a hostname-only substring match misclassified non-OpenAI Azure traffic as OpenAI routes. - Replace the substring host test with suffix matching (rejects look-alike domains like cognitiveservices.azure.com.attacker.example). - Add `_is_openai_compatible_url` that requires an OpenAI-style path marker (`/openai/` or `/v1/`) on the shared Azure domains, and use it in PassThroughEndpointLogging.is_openai_route (previously hostname-only). - Add negative tests for Azure Speech/Vision paths and look-alike domains. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: support Responses input in Redis semantic cache (#29581) * fix: support responses input in redis semantic cache * test: cover redis semantic prompt extraction * test: handle blank redis semantic text fallbacks * chore: remove async cache dead statement * test: cover redis semantic cache miss paths * fix: filter sensitive cache lookup kwargs * chore: rerun ci after huggingface rate limit * chore(ui): regenerate dashboard API types (npm run gen:api) Sync src/lib/http/schema.d.ts with the proxy OpenAPI spec: adds the disable_budget_reservation general-settings field and picks up the RateLimitError docstring reindent. Fixes the gen:api CI drift check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(bedrock): assert empty additionalModelRequestFields is omitted The Converse transformer now drops an empty additionalModelRequestFields block instead of sending it as `{}`. Update test_bedrock_top_k_param so models without top_k support (llama3) assert the key is absent rather than equal to an empty dict. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: codgician <15964984+codgician@users.noreply.github.com> Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com> Co-authored-by: Roi <roytev@gmail.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Liam Scott <liam@uilliam.com> Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com> Co-authored-by: Ceder Dens <cederdens@gmail.com> Co-authored-by: 冯基魁 <56265583+fengjikui@users.noreply.github.com> Co-authored-by: Kai Huang <kaihuang724@gmail.com> Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com> Co-authored-by: Genmin <joey@joeyroth.com> Co-authored-by: Arnav Bhilwariya <arnavbhilwariya0408@gmail.com> Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com> Co-authored-by: João Garrido <48538534+johngarrido@users.noreply.github.com> Co-authored-by: João Garrido <joaogarrido@google.com> Co-authored-by: Kenan Yildirim <kenan@kenany.me> Co-authored-by: Dávid Balatoni <balcsida@gmail.com>	2026-06-08 13:49:52 -07:00
Sameer Kankute	5fd27141cf	Litellm OSS Staging 010626 (#29422)	2026-06-01 21:42:51 -07:00
Ishaan Jaffer	e8461b5b97	style: run black formatter on files from main merge	2026-04-17 13:02:59 -07:00
yuneng-jiang	fbad073a1f	[Fix] Use cached async HTTP client in Vantage destination ... Replace per-request `httpx.AsyncClient` with `get_async_httpx_client` to avoid the +500ms latency penalty from creating new clients per request. Updates tests to mock the cached client factory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 12:23:40 -07:00
yuneng-jiang	6abdf5adde	[Fix] Responses bridge variable mismatch and outdated CI tests ... Fix genuine regression in responses_api_bridge_check where the second call assigned to `model_info` instead of `responses_api_model_info`, preventing gpt-5.4 + tools + reasoning_effort from routing to the Responses API bridge. Also update outdated tests: - Vantage tests: match "csv" file key and use supported column names - Anthropic caching test: add "type": "custom" to expected tool payload - Claude Agent SDK test: remove non-deterministic LLM content assertion Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-14 12:12:35 -07:00
Harshit28j	24d4e5bc60	Deregister VantageLogger on delete and add Decimal cast test ... - DELETE /vantage/delete now removes the in-memory VantageLogger from litellm.callbacks via remove_callbacks_by_type, preventing the scheduler from continuing to fire exports with stale credentials - Add test_should_cast_decimal_columns_to_float covering the Decimal→Float64 cast in FocusCsvSerializer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 19:28:27 +05:30
Harshit28j	ce052d07be	Fix test helper hour=23 bug and add row-count batching test ... - Use timedelta(hours=1) instead of replace(hour=hour+1) in _window() to avoid ValueError when hour=23 - Add test_should_batch_by_row_count covering the >10K rows batching path (previously only the 2 MB size-limit path was tested) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-03-11 18:12:27 +05:30
Harshit28j	69a94a873c	Add Vantage integration for FOCUS CSV export ... Adds a pluggable Vantage destination to the existing FOCUS export pipeline, enabling LiteLLM to export spend data in FOCUS format directly to Vantage's cost-import API. Supports automatic hourly exports via scheduled background job, with admin API endpoints for manual control and configuration. Includes CSV serializer, batching for 10K row / 2MB API limits, and enriched Tags JSON with team/user/key metadata for Vantage Token Allocation feature. - Add CSV serializer (FocusCsvSerializer) for FOCUS data - Add Vantage API destination with automatic batching - Add VantageLogger that wraps FocusLogger with Vantage defaults - Add proxy endpoints: /vantage/{init,settings,export,dry-run,delete} - Register "vantage" callback in logger registry and literal type - Wire up background job in proxy_server.py startup - Populate Tags column with JSON metadata (team_id, user_id, user_email, etc.) - Add 14 unit tests covering serializer, destination, and factory All tests pass (23 focus tests total, no regressions). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>	2026-03-11 13:51:36 +05:30
Yuta Saito	48bc5ccb4f	fix: test	2026-01-09 11:23:47 +09:00
Yuta Saito	271ee0959b	test: focus	2026-01-08 16:53:09 +09:00