From 133da06aa3b855ad09f231abd0cdc24bbe0b4067 Mon Sep 17 00:00:00 2001 From: Sameer Kankute Date: Fri, 26 Jun 2026 21:47:44 +0530 Subject: [PATCH] chore: litellm oss staging (#31185) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped The Total Tokens Over Time and Total Requests Over Time AreaCharts on the Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow once totals pass the hundred-million mark — leading digits of labels like "100.00M" / "4500.00M" got clipped against the chart edge. The requests chart was worse: it formatted with toLocaleString(), so billion-scale request counts produced "1,000,000,000" (13 chars) and overflowed immediately. Fix in two places so neither alone has to carry the whole margin: - activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and switch the requests chart to the shared valueFormatter so it uses the same compact k/M/B suffixes as the tokens chart. - value_formatters.tsx: add a >= 1e9 branch to valueFormatter / valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping every formatted label at most 7 chars. Co-Authored-By: Claude Opus 4 (1M context) * Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * docs(readme): add Deploy on AWS/GCP with Terraform section Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 * docs(readme): add 1-click deploy buttons for AWS + GCP GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 * docs(readme): move AWS + GCP deploy buttons next to Render button * docs(readme): unify deploy button sizes and badge styles * docs(readme): bump deploy button height to 48 to match Render/Railway * docs(readme): bump AWS/GCP badge height to compensate for SVG padding * docs(readme): bump AWS/GCP badge height to 72 * docs(readme): bump AWS/GCP badge height to 84 * fix(readme): make deploy buttons same height (48px) https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc * docs(readme): flag GCP project ID substitution in image_registry * docs(readme): equalize deploy button heights and fix Cloud Shell button font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface. * docs(readme): collapse Railway deploy anchor to a single line The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline. * Add Claude Fable 5 cost map entries as a data-only hotfix Backports only the model map changes from #30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano Three bugs in model_prices_and_context_window.json: 1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K max output, but the values were set as max_input=128000, max_tokens=272000. This caused token limit errors when sending prompts over 128K tokens to GPT-5 Pro. 2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was 272000, but GPT-5.4 Mini shares the same 1,050,000 token context window as GPT-5.4. This was inconsistent with the azure/ variants which already correctly had 1,050,000. 3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini, max_input_tokens was 272000 instead of 1,050,000. Source: OpenAI model documentation and contextwindows.dev which aggregates official context window sizes. Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini should be 400K; their 272K values are correct per OpenAI docs) * fix: also correct max_output_tokens for gpt-5-pro (272000→128000) Per reviewer feedback, max_output_tokens was left at 272000 while max_tokens was corrected to 128000, causing an internal inconsistency. Both should be 128000 per OpenAI docs. * fix(cost): price gpt-image generated output tokens as image tokens (#31147) The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return usage with no output token breakdown — litellm's `ImageUsage` has no `output_tokens_details` field — so generated-image OUTPUT tokens were priced at the text rate (`output_cost_per_token`) instead of the image rate (`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x undercount on the dominant cost component (image output is ~74% of spend). This also affects azure gpt-image, which shares this calculator. The OpenAI gpt-image cost calculator re-implemented usage handling instead of reusing `calculate_image_response_cost_from_usage`, the shared helper that azure_ai/gemini/vertex_ai already use. That helper classifies generated output tokens as image tokens when the provider does not itemize output, and splits text/image when it does. Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage` (pre-transformed chat Usage objects are still costed directly). Adds a regression test for the no-breakdown ImageUsage case (gpt-image-2). * fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098) A bare application-inference-profile ARN passed as bedrock/arn:... fell through to the invoke route, which cannot derive a provider from the opaque profile id and raised 'Unknown provider=None'. The converse route needs no provider, so detect these ARNs in get_bedrock_route and route them to converse, matching the behavior of the already-documented bedrock/converse/arn:... workaround. Explicit invoke/ prefixes still win, and they remain a dead end for these ARNs by design (no provider derivable). System-defined inference-profile ARNs that embed a known model, and other opaque ARN types (provisioned-model, imported-model, custom-model-deployment) that are frequently invoke-only, are deliberately left on their current routes; tests guard both boundaries. * fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060) _add_tool_choice_required_message appended the "select a tool" prompt to the caller's messages list in place, so transform_request corrupted the caller's conversation history and appended a duplicate prompt on every retry. Build and return a new list instead so the call stays idempotent. Adds a regression test asserting the input messages list is unchanged across repeated transform_request calls. Co-authored-by: Wassbdr * fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996) gpt-4o-transcribe and compatible ASR backends return a diarized_json response with usage={"type": "duration", "seconds": }, e.g. 295.8. TranscriptionUsageDurationObject typed seconds as int, so parsing the response raised a pydantic ValidationError (int_from_float). That error surfaces as an APIConnectionError which the router treats as retryable, so it keeps re-calling the upstream (200 every time) until the upstream rate-limits and returns 429 to the caller. OpenAI specs this field as a float (see openai SDK UsageDuration.seconds), so widen seconds to float. With the parse succeeding there is no exception left to retry, which removes the loop. Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> * fix(deepseek): drop non-function tools before chat completions call (#30910) * fix(deepseek): drop non-function tools before chat completions call DeepSeek's /chat/completions only accepts tools of type "function". Requests bridged from /v1/responses can carry responses-API-native tool types, for example a Codex CLI tool typed "namespace", which DeepSeek rejects with "unknown variant 'namespace', expected 'function'" so the whole request fails (issue #30722). Filter unsupported tool types in the DeepSeek request transform so the function tools still go through; when nothing callable remains, also drop the now-dangling tool_choice and parallel_tool_calls Fixes #30722 * test(deepseek): cover async tool filtering and document tool_choice assumption Add an async_transform_request regression test so the sync and async tool filtering paths cannot silently diverge, and document in _drop_unsupported_tools that only non-function tools are dropped, so a function-named tool_choice always references a surviving tool * feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840) * feat(ui): surface team budget on key overview when key has no own budget (#30801) * feat(ui): surface team budget on key overview when key has no own budget * fix(ui): replace IIFE with derived variable and use find() for team budget display * fix(anthropic): emit replayable streaming thinking blocks (#31022) * feat(proxy): read cold-storage prompts back in the logs detail view (#30364) * feat(proxy): read cold-storage prompts back in the logs detail view When a deployment offloads prompts and responses to cold storage instead of Postgres, the spend-log row holds only "{}" placeholders plus a metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed nothing. The detail endpoint only read the placeholder columns and never fetched the object back. Resolve the payload per row based on actual content, not a config flag: if Postgres has content, return it; otherwise read the exact stored object key and fetch from the configured cold storage backend through ColdStorageHandler. Reading the persisted key is a single GET. The key embeds a microsecond timestamp that cannot be reconstructed from the millisecond-precision startTime column, and listing the day's prefix to match on request_id would be too expensive for this per-open path. Also teach the detail drawer's pretty-view parser to accept a bare messages array. The cold storage payload carries the prompt as a top-level messages list with no proxy_server_request, so without this the output rendered while the input stayed blank. ColdStorageHandler gains an optional injected logger so the resolver can be unit tested without monkeypatching. Postgres-stored prompts are unaffected: the fast path returns the existing columns and the request-body object still renders the same way. * Update litellm/proxy/spend_tracking/spend_management_endpoints.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure Add unit tests for ColdStorageHandler (injected logger, graceful None when no logger is configured, and resolution of a configured logger from the callback registry) and a regression test asserting a cold storage backend exception degrades to the Postgres values instead of surfacing a 500. --------- Co-authored-by: Bytechoreographer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068) * fix(mavvrik): advance metricsMarker after upload + fix scheduler startup Two bugs fixed: 1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a successful GCS upload, so metricsMarker stayed at 0 and every daily run re-exported the same dates in an infinite catch-up loop. Fix: add _update_metrics_marker(date_epoch) called at the end of deliver() after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS file is already committed). A 410 raises consistent with the rest of the destination. 2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call has triggered lazy instantiation of MavvrikFocusLogger, so it found no logger instance and silently skipped registering the daily export job. Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call _init_custom_logger_compatible_class to force instantiation before the APScheduler job is registered. * fix(mavvrik): catch up from earliest window when metricsMarker=0 When the connector is freshly registered, metricsMarker=0 parses to None. The catch-up block was guarded by `if last_ingested and ...` which skipped it entirely for None, so only yesterday was exported instead of the full _MAX_CATCHUP_DAYS window. Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup). The existing > 7 day warning only fires for non-None markers that are old. * fix(mavvrik): use now as end_time for yesterday's export window LiteLLM_DailyUserSpend rows for a given date get their updated_at bumped by the spend flush job throughout the next morning. The core database query filters on updated_at, so capping end_time at midnight (yesterday + 1 day) missed any spend rows flushed after midnight. Fix: pass now (cron fire time) as end_time for the daily "yesterday" window so all fully-settled rows are captured regardless of when the flush job ran. Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per row in the exported FOCUS CSV. * fix(mavvrik): also use now as end_time for catch-up windows * fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class Calling it with only logging_integration raised TypeError at proxy startup because internal_usage_cache and llm_router have no defaults. Also fix test name to reflect the actual status code (5xx not 4xx) used in the mock. * ci: retrigger CI run * feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757) * Add optional `instruction` passthrough to the rerank API vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction` field (folded into the model's chat_template_kwargs and consumed by the chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently dropped it: RerankRequest / OptionalRerankParams had no such field, so the outgoing body was rebuilt without it. Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(), get_optional_rerank_params, and the hosted_vllm transformation into the request body, only when non-None. When callers omit it, model_dump(exclude_none) drops the field and the outgoing request is byte-for-byte unchanged — fully backward-compatible. (DeepInfra already forwards `instruction` via non_default_params; this formalizes the field in the shared types.) Co-Authored-By: Claude Opus 4.8 (1M context) * Address review: thread `instruction` as a typed param + cover rerank_utils Per PR review (greptile P2 + codecov): - Make `instruction` a typed, named argument on the rerank provider interface instead of recovering it from the opaque `non_default_params` blob. Adds `instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params` and every provider override, and forwards it explicitly from `get_optional_rerank_params`. hosted_vllm now reads the named param directly. It is still also surfaced in `non_default_params` so providers that read it there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction` as a named param rather than leaving it in **kwargs. - Add get_optional_rerank_params unit tests (present + absent) to cover the previously-uncovered threading line flagged by codecov. Co-Authored-By: Claude Opus 4.8 (1M context) * fix: scan rerank `instruction` through request guardrails The rerank guardrail translation (CohereRerankHandler.process_input_messages) only scanned `query`, so the newly added `instruction` field reached the backend model unscanned. Since instruction-aware rerankers (hosted vLLM / Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller could place content there to bypass configured rerank request guardrails. Generalize the handler to scan every user-controlled text field (`query` and `instruction`) in one apply_guardrail call and write each sanitized value back by index. Query-only requests are unchanged (single-element list at index 0); non-string fields are left untouched. Adds tests covering instruction scanning, PII masking write-back, and the non-string case. Addresses the Veria AI security review on PR #30757. * test: narrow Optional results before len() to satisfy basedpyright budget The lint gate (basedpyright delta-vs-base budget) flagged one new reportArgumentType: len(result.results) where results is List[RerankResponseResult] | None. Assert results is not None first to narrow the type before len()/indexing. * fix: read rerank `instruction` from kwargs to satisfy basedpyright budget The basedpyright delta-vs-base gate flagged one new reportArgumentType: the Router forwards rerank calls via an untyped `**kwargs` unpack (`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a typed named param on the public `rerank`/`arerank` entrypoints made pyright check that key against `str | None`, adding an error at router.py with no real safety gain. Read `instruction` from kwargs in `rerank` instead. It remains fully typed where it matters - threaded as a typed argument through `get_optional_rerank_params` and each provider's `map_cohere_rerank_params` (the original Greptile P2 ask). Whole-repo reportArgumentType is back to the base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean. --------- Co-authored-by: Claude Opus 4.8 (1M context) * fix(github_copilot): synthesize empty choices at the provider seam (#30929) Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with choices=[], either carrying Anthropic-native content blocks or, for the max_tokens=1 probe Claude Code sends, no content at all. github_copilot is dispatched through the OpenAI SDK handler, which calls convert_to_model_response_object directly and never invokes GithubCopilotConfig.transform_response, so the empty-choices guard there surfaced as a 500 Instead of synthesizing choices inside the shared convert_to_model_response_object (which would silently turn empty choices into a fabricated success for every provider), add a no-op transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig overrides it to synthesize choices from Anthropic-native content, reusing its existing parsing, and the OpenAI SDK handler routes its parsed response through the hook before generic conversion. The core utility keeps treating empty choices as an error for all other providers Fixes: https://github.com/BerriAI/litellm/issues/30927 Signed-off-by: David J. M. Karlsen * fix(router): stop fallback lookups from mutating the router fallbacks config (#30624) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693) * fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens * test: scope local cost map env var with monkeypatch to avoid test pollution * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764) * fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold _mask_value did partial reveal by showing the first visible_prefix and last visible_suffix characters, but for a value whose length was at or below visible_prefix + visible_suffix (8 by default) it returned the value verbatim. A value of exactly 8 chars fell through the length guard and computed masked_length == 0, reconstructing the original string with no mask characters; anything shorter hit the early return. Either way short credentials were emitted in plaintext. mask_dict routes real secrets through this path, so an 8-char-or-shorter redis password, api key, or token could be written to logs and the UI unmasked. The sibling helper mask_sensitive_keys already guards this case; _mask_value now does the same by fully masking any value at or below the threshold. * fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers Fully masking short values is the right default for secret masking, but CooldownCache reuses the masker purely to truncate exception messages to the first 50 characters, and it relies on short messages being returned readable. Masking those blanked out short exception text and broke its tests. Add a mask_short_values flag (default True, secure) and have CooldownCache pass False so it keeps the truncation behavior, while every secret-masking caller still gets short values fully masked. * fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview MCPDebug uses the masker to preview auth tokens in debug headers and documents that values of 10 chars or fewer are shown unchanged so token types stay distinguishable. Pass mask_short_values=False so that diagnostic behavior is preserved while secret maskers keep masking short values. * fix(mcp_debug): mask short auth values in debug headers instead of echoing them Earlier this masker opted out of short-value masking to keep a token preview, but that echoes short authorization and token values verbatim in debug response headers, which is the same leak this change is meant to close. Auth material should never be emitted in full, so mask short values here too; the first/last character preview still applies to longer tokens. Only CooldownCache keeps the opt-out, since it truncates exception text rather than masking secrets. * test(mcp_debug): assert masked short value preserves length * refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917) Fireworks AI deprecated audio inference on 2026-06-10 (https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation). Live API testing confirms the endpoint is already non-functional: a valid Fireworks API key receives HTTP 401 "Unauthorized" from api.fireworks.ai/inference/v1/audio/transcriptions for every request, regardless of payload. The audio-prod.api.fireworks.ai host referenced in the test suite returns 401 for every path; the entire host is decommissioned. Remove the dead FireworksAIAudioTranscriptionConfig class and every reference to it across the codebase: - Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line config class that inherited from OpenAIWhisperAudioTranscriptionConfig) - Remove the Fireworks branch from ProviderConfigManager.get_provider_audio_transcription_config() in litellm/utils.py; update the stale comment in get_optional_params_transcription that referenced fireworks ai - Remove the FireworksAIAudioTranscriptionConfig entries from LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in litellm/_lazy_imports_registry.py - Remove the TYPE_CHECKING re-export in litellm/__init__.py - Remove the transcription branch in the fireworks_ai case of get_supported_openai_params() in litellm/litellm_core_utils/get_supported_openai_params.py - Remove the whisper-v3 and whisper-v3-turbo entries from model_prices_and_context_window.json and litellm/model_prices_and_context_window_backup.json (both had mode: audio_transcription and zero-cost pricing) - Remove the TestFireworksAIAudioTranscription test class and its imports from tests/llm_translation/test_fireworks_ai_translation.py No other provider is affected. The openai_compatible_providers list, FireworksAIMixin, and the OpenAI Whisper transcription handler all stay because they are shared with other Fireworks endpoints and other providers. The provider_endpoints_support.json registry already had audio_transcriptions set to false for fireworks_ai. * feat: add darkbloom provider (#30876) * feat: add darkbloom provider * fix: document darkbloom provider endpoints * fix: address darkbloom review feedback * fix: update darkbloom tool metadata * fix: fail fast for non-Postgres database URLs (#30883) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging * Validate DIRECT_URL alongside DATABASE_URL startup guards * fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946) * fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608) * test(bedrock): mid-stream server errors trigger streaming fallback (#24608) * style(bedrock): black-format stream-error helper (#24608) * fix(mcp): re-land native tool preservation with typed annotations (#30645) * fix(mcp): preserve native tools in semantic filter hook with typed annotations * fix(mcp): tighten _is_mcp_tool Chat Completions shape check * fix(sambanova): return embeddings supported params instead of dropping them (#30937) * fix(router): send fallback metadata when streaming (#30914) When a streaming request triggers a fallback, there was previously no way to know it happened. This commit addresses this in a few ways: 1. The response now correctly populates the fallback headers (`x-litellm-attempted-fallbacks`) so callers know a fallback happened. 2. The correct model ID is passed in the streaming chunks. 3. A streaming chunk with the fallback error can be optionally sent back to the client (opt-in) by passing `include_fallback_errors: true` in the request. The format of the fallback errors while streaming is intentionally OpenAI compatible to not break existing libraries that parse these events. It was tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not delieved unexpectedly to callers by default. * fix(mistral): drop output-only reasoning fields from input messages (#30884) LiteLLM attaches reasoning_content and thinking_blocks to assistant responses. Replaying those assistant turns verbatim forwarded the fields back to Mistral, whose input schema forbids unknown keys, so the whole request failed with a 422 extra_forbidden and reasoning models became unusable across multiple turns. Strip both fields from assistant messages before the request is built, in a spot that runs ahead of the image/file branch so it applies on every path. Fixes #30835 Co-authored-by: Cursor * fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652) * fix(perplexity): bill search queries at the per-request price, not 1/1000 The fallback cost calculator divided search_context_cost_per_query by 1000, but that field stores the per-request price in USD: sonar is {low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published $5/$8/$12 per 1,000 requests expressed per request. The gemini cost calculator reads the same field per request with no division (its docstring calls it "the per-request cost"). The division understated search cost by 1000x on every Perplexity call that falls back to manual calculation (i.e. when the API does not return a pre-computed usage.cost). Use the value directly. Update the tests that had encoded the /1000 factor in their expectations, and drop an unused import flagged by ruff in the touched test file. * test(perplexity): update integration test search-cost expectations to per-request The integration tests still encoded the old /1000 search-cost factor, so they failed once the fallback calculator was corrected to bill search_context_cost_per_query per request. Update the four expected-cost computations (and the high-volume dollar-value comments) to match. * test(perplexity): drop unused mock imports flagged by ruff * fix: include model_access_groups when expanding all-team-models in get_team_models (#30622) * fix(fireworks_ai): return None for transcription in get_supported_openai_params Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is decommissioned. Without an explicit transcription branch, requests with request_type='transcription' fell through to the else and returned FireworksAIConfig chat-completion params. Return None instead to signal the provider does not support transcription. * fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting Without an operator gate, any authenticated caller could set include_fallback_errors=True, trigger a fallback, and read raw upstream exception messages from the x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event. Strip include_fallback_errors from request data in common_processing_pre_call_logic when expose_fallback_errors_to_caller is not set, so the router never builds the error list. Also gate _should_include_fallback_errors on the same setting as a secondary check for the streaming SSE injection path. * test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test The operator gate added in e7ff3e1 means include_fallback_errors is only honoured when general_settings.expose_fallback_errors_to_caller is True. Set that flag via monkeypatch in the test that exercises the emit path. * test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos test_convert_url called convert_url_to_base64 against a live picsum.photos URL and asserted nothing, so it added no real signal and broke CI whenever the host was unreachable (it was returning 522 and blocking this branch). Replace the live call with a mocked HTTP client and assert the produced base64 data URL, so the conversion path is exercised deterministically with no network dependency. This suite runs under VCR, which is why a transport level mock (respx) does not reliably intercept; mocking the client object itself is robust regardless. * fix(interactions): drop role from Interaction response to match Google spec Google removed the output-only role field from the Interaction schema (it now lives only on Turn), so the live OpenAPI compliance canary started failing with 'role' not in spec. Reconcile our generated types by removing role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse, stop stamping role=model in the responses-to-interactions transformation, and update the compliance and integration tests accordingly. Turn.role is kept since the spec still defines it. * fix: align all-team-models sentinel access * fix(router): forward include_fallback_errors through multi-hop fallbacks run_async_fallback received include_fallback_errors as an explicit named parameter, so it was bound out of **kwargs and never reached the nested async_function_with_fallbacks call. Multi-hop fallback chains (a fallback group that itself fails over) therefore stopped collecting fallback errors beyond the first hop when a caller opted in. Re-inject the flag into kwargs before the nested call so inner hops keep accumulating errors, which add_fallback_headers_to_response already merges across levels. * fix(router): stop fallback lookups from mutating the router fallbacks config get_fallback_model_group resolved a bare-string fallback by popping it out of the fallbacks list it was handed. That list is frequently the live router.fallbacks config, so a single lookup permanently removed the entry and the configured fallback stopped applying to later requests until restart. The pop also ran inside enumerate(), shifting indices and skipping an adjacent string fallback. Read the item instead of popping it, and add a regression test that fails on the old mutating behavior --------- Co-authored-by: Srivatsa Kamballa Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn Co-authored-by: Tal Marian Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: Sameer Kankute Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * fix(sambanova): update pricing, deprecate retired models, and add missing models (#30016) * feat(bedrock): add amazon.titan-embed-g1-text-02 embedding model support - Add model to provider routing allowlist in embedding.py - Add request transformation using AmazonTitanG1Config - Add response transformation using AmazonTitanG1Config - Add pricing metadata to model_prices_and_context_window.json - Add unit tests for embedding and model info Fixes missing cost tracking reported in #29786 Related to VANDRANKI/litellm PR #29790 * style: fix syntax error, trailing whitespace and missing newline * style: apply black formatting to embedding.py * style: apply black formatting to test_bedrock_embedding.py * fix(sambanova): update pricing, fix context windows, add deprecation dates, and add missing models * fix(sambanova): sync model_prices_and_context_window_backup.json with primary * fix(sambanova): fix indentation on Meta-Llama-3.2-1B-Instruct deprecation_date * fix(bedrock): add amazon.titan-embed-g1-text-02 to unmapped model error message * style: apply black formatting to embedding.py * fix(sambanova): correct indentation on DeepSeek-V3.2 entry * fix(sambanova): replace gemma-3-12b-it with gemma-4-31B-it (verified pricing) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info (#30880) * fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info get_model_info rebuilt ModelInfo by copying a fixed allow-list of input/output_cost_per_token_above__tokens keys (128k/200k/272k/512k), so any other threshold a user registered was dropped before reaching _get_token_base_cost, which already reads an arbitrary threshold out of the key name. Custom tiers such as above_500k_tokens were silently ignored and billing fell back to the base per-token rate. Carry over any _above__tokens cost key present on the source cost-map entry that the fixed fields miss Fixes #30344 * test(cost): keep suite hermetic by popping the temp tiered-pricing model Wrap the regression body in try/finally so litellm.model_cost no longer leaks the litellm-test-non-standard-tier entry into later tests that iterate or reset the global cost map. Addresses Greptile review thread. * fix: resolve UP045 lint violations (Optional[X] -> X | None) Convert Optional[X] type annotations to X | None syntax across rerank transformations, spend tracking, and other modules to satisfy ruff strict gate. Co-Authored-By: Claude Sonnet 4.6 * fix: run black formatting on UP045-fixed files Co-Authored-By: Claude Sonnet 4.6 * fix: remove unused Optional imports after UP045 migration Co-Authored-By: Claude Sonnet 4.6 * fix: black format cold_storage_handler.py Co-Authored-By: Claude Sonnet 4.6 * fix(ci): correct OSS staging branch name in guard-main-branch errors Co-authored-by: Cursor * fix: strip trailing zeros from M/B spend formatter * fix: address focus and streaming edge cases * feat: add LAR-1 semantic routing strategy Optional router strategy that picks a deployment tier from request_kwargs.metadata.lar1 (confidence, evidence, time). Deployments are tagged with model_info.type (cloud-smart, cloud-fast, local, deep). Thresholds are configurable via routing_strategy_args. Includes 30 unit tests and an Ollama example config. Co-authored-by: Cursor * fix(mavvrik): advance metricsMarker on empty-content deliver When deliver() receives empty content (no spend data for a date), it now registers with Mavvrik and PATCHes the metricsMarker before returning instead of short-circuiting. Dates with zero spend no longer stall marker advancement, preventing unnecessary catch-up API calls on subsequent runs. * style: black format mavvrik_destination * fix: handle empty mavvrik exports and lar1 reset * test: add regression test for _reset_custom_routing_strategy * fix(test): mock async destination.deliver in mavvrik export window test * style: ruff format spend_management_endpoints after merge * fix(router): apply LAR-1 strategy atomically so invalid thresholds don't leave partial state apply_lar1_routing_strategy set router.routing_strategy to "lar1" before constructing LAR1RoutingStrategy, whose __init__ validates thresholds via _normalize_thresholds and raises on a misconfigured (out-of-order or out-of-range) set. On a live update_settings call with bad thresholds the router was left advertising routing_strategy="lar1" with no custom selector bound, while the previous strategy's selectors stayed registered. Build (and validate) the strategy before mutating any router state, so a threshold error leaves the router exactly as it was. Add a regression test that asserts a failed switch keeps the prior strategy intact. --------- Signed-off-by: David J. M. Karlsen Co-authored-by: Bytechoreographer Co-authored-by: Claude Opus 4 (1M context) Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: shin-berri Co-authored-by: yuneng-jiang Co-authored-by: Yassin Kortam Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Krrish Dholakia Co-authored-by: xbrxr03 Co-authored-by: hayden Co-authored-by: Kent <72616338+kingdoooo@users.noreply.github.com> Co-authored-by: Wassim Badraoui <98709649+Wassbdr@users.noreply.github.com> Co-authored-by: Wassbdr Co-authored-by: Neimar Avila Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com> Co-authored-by: Jerry-Scintilla Co-authored-by: AlexBGoode Co-authored-by: Carsten Boloz Co-authored-by: jesco Co-authored-by: Praveen Ghuge Co-authored-by: Jim Smith Co-authored-by: David J. M. Karlsen Co-authored-by: Vedant Agarwal <43557509+Vedant-Agarwal@users.noreply.github.com> Co-authored-by: Srivatsa Kamballa Co-authored-by: Ahmad Shahzad <107808273+shzdehmd@users.noreply.github.com> Co-authored-by: Jeremy Chapeau <113923302+jychp@users.noreply.github.com> Co-authored-by: KRISH SONI <67964054+krishvsoni@users.noreply.github.com> Co-authored-by: Ayush Shekhar <106994833+ayushh0110@users.noreply.github.com> Co-authored-by: dav nguyxn Co-authored-by: Tal Marian Co-authored-by: Hemant K <51333870+hemant1026@users.noreply.github.com> Co-authored-by: Cursor Co-authored-by: Yash Raj Pandey <55940078+devYRPauli@users.noreply.github.com> Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: bhumikadangayach <139267865+bhumikadangayach@users.noreply.github.com> Co-authored-by: Ewertonslv Co-authored-by: carlsonchik --- examples/lar1_ollama_config.yaml | 45 + .../focus/destinations/mavvrik_destination.py | 66 +- .../mavvrik_focus/mavvrik_focus_logger.py | 73 +- litellm/llms/anthropic/chat/handler.py | 124 +-- litellm/llms/base_llm/chat/transformation.py | 11 + .../llms/base_llm/rerank/transformation.py | 33 +- litellm/llms/bedrock/common_utils.py | 12 + litellm/llms/bedrock/embed/embedding.py | 10 + .../rerank/guardrail_translation/handler.py | 76 +- litellm/llms/cohere/rerank/transformation.py | 29 +- .../llms/cohere/rerank_v2/transformation.py | 23 +- .../llms/dashscope/rerank/transformation.py | 35 +- .../llms/deepinfra/rerank/transformation.py | 27 +- litellm/llms/deepseek/chat/transformation.py | 82 ++ .../fireworks_ai/rerank/transformation.py | 31 +- .../github_copilot/chat/transformation.py | 175 ++-- .../llms/hosted_vllm/rerank/transformation.py | 57 +- .../llms/huggingface/rerank/transformation.py | 41 +- litellm/llms/jina_ai/rerank/transformation.py | 35 +- litellm/llms/moonshot/chat/transformation.py | 10 +- .../llms/nvidia_nim/rerank/transformation.py | 29 +- .../image_generation/cost_calculator.py | 69 +- litellm/llms/openai/openai.py | 10 +- .../llms/vertex_ai/rerank/transformation.py | 27 +- litellm/llms/voyage/rerank/transformation.py | 35 +- litellm/llms/watsonx/rerank/transformation.py | 35 +- ...odel_prices_and_context_window_backup.json | 987 +++++++++++++----- .../spend_tracking/cold_storage_handler.py | 49 +- .../spend_management_endpoints.py | 240 +++-- litellm/rerank_api/main.py | 44 +- litellm/rerank_api/rerank_utils.py | 23 +- litellm/router.py | 57 +- litellm/router_strategy/lar1_routing.py | 192 ++++ .../router_utils/fallback_event_handlers.py | 2 +- litellm/types/lar1.py | 39 + litellm/types/rerank.py | 5 + litellm/types/utils.py | 2 +- litellm/utils.py | 12 +- model_prices_and_context_window.json | 123 ++- .../chat/test_deepseek_chat_transformation.py | 21 + .../llm_translation/test_bedrock_embedding.py | 15 + .../test_convert_dict_to_chat_completion.py | 20 +- .../test_router_custom_routing.py | 25 + .../focus/test_mavvrik_destination.py | 160 ++- .../test_mavvrik_focus_logger.py | 67 ++ .../llm_cost_calc/test_llm_cost_calc_utils.py | 38 + .../chat/test_anthropic_chat_handler.py | 144 +++ .../llms/bedrock/test_bedrock_common_utils.py | 62 ++ .../rerank/test_rerank_guardrail_handler.py | 70 +- .../llms/deepseek/chat/__init__.py | 0 .../chat/test_deepseek_chat_transformation.py | 103 ++ .../test_github_copilot_transformation.py | 104 ++ .../test_hosted_vllm_rerank_transformation.py | 79 ++ .../test_moonshot_chat_transformation.py | 28 + .../test_transcription_duration_hidden.py | 47 +- .../test_spend_management_endpoints.py | 295 ++++++ .../router_strategy/test_lar1_routing.py | 460 ++++++++ .../test_fallback_event_handlers.py | 18 +- .../test_gpt_image_cost_calculator.py | 88 ++ .../UsagePage/utils/value_formatters.tsx | 14 +- .../VirtualKeysPage/VirtualKeysTable.tsx | 11 +- .../src/components/activity_metrics.tsx | 4 +- .../key_info_view.budget_display.test.tsx | 79 +- .../components/templates/key_info_view.tsx | 16 +- .../PrettyMessagesView.test.tsx | 11 + .../LogDetailsDrawer/prettyMessagesUtils.ts | 24 +- 66 files changed, 4049 insertions(+), 929 deletions(-) create mode 100644 examples/lar1_ollama_config.yaml create mode 100644 litellm/router_strategy/lar1_routing.py create mode 100644 litellm/types/lar1.py create mode 100644 tests/test_litellm/integrations/mavvrik_focus/test_mavvrik_focus_logger.py create mode 100644 tests/test_litellm/llms/deepseek/chat/__init__.py create mode 100644 tests/test_litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py create mode 100644 tests/test_litellm/router_strategy/test_lar1_routing.py diff --git a/examples/lar1_ollama_config.yaml b/examples/lar1_ollama_config.yaml new file mode 100644 index 00000000000..998cbf169b3 --- /dev/null +++ b/examples/lar1_ollama_config.yaml @@ -0,0 +1,45 @@ +model_list: + - model_name: agent-router + litellm_params: + model: ollama/qwen3.5:9b + api_base: http://127.0.0.1:11434 + model_info: + id: cloud-smart + type: cloud-smart + + - model_name: agent-router + litellm_params: + model: ollama/phi4-mini:latest + api_base: http://127.0.0.1:11434 + model_info: + id: cloud-fast + type: cloud-fast + + - model_name: agent-router + litellm_params: + model: ollama/llama3.2:3b + api_base: http://127.0.0.1:11434 + model_info: + id: local + type: local + + - model_name: agent-router + litellm_params: + model: ollama/lfm2.5-thinking:latest + api_base: http://127.0.0.1:11434 + model_info: + id: deep + type: deep + +router_settings: + routing_strategy: lar1 + routing_strategy_args: + confidence_threshold_low: 0.3 + confidence_threshold_medium: 0.5 + confidence_threshold_high: 0.7 + +general_settings: + master_key: sk-lar1-demo + +litellm_settings: + set_verbose: true diff --git a/litellm/integrations/focus/destinations/mavvrik_destination.py b/litellm/integrations/focus/destinations/mavvrik_destination.py index 1e3c98b9a70..81944ba69e2 100644 --- a/litellm/integrations/focus/destinations/mavvrik_destination.py +++ b/litellm/integrations/focus/destinations/mavvrik_destination.py @@ -3,6 +3,7 @@ Flow: 1. GET /metrics/agent/ai/{connection_id}/upload-url → GCS signed URL 2. PUT with CSV content + 3. PATCH /metrics/agent/ai/{connection_id} → advance metricsMarker """ from __future__ import annotations @@ -33,8 +34,7 @@ def _validate_api_endpoint(api_endpoint: str) -> None: hostname = (urlparse(api_endpoint).hostname or "").lower() if not any(hostname.endswith(suffix) for suffix in _MAVVRIK_ALLOWED_SUFFIXES): raise ValueError( - "MAVVRIK_API_ENDPOINT host must be a Mavvrik domain " - "(e.g. https://api.mavvrik.dev/)" + "MAVVRIK_API_ENDPOINT host must be a Mavvrik domain (e.g. https://api.mavvrik.dev/)" ) @@ -50,8 +50,7 @@ def _validate_gcs_url(url: str, label: str) -> None: or hostname.endswith(".storage.googleapis.com") ): raise ValueError( - f"Mavvrik FOCUS destination: {label} must be a GCS endpoint " - f"(storage.googleapis.com), got '{hostname}'" + f"Mavvrik FOCUS destination: {label} must be a GCS endpoint (storage.googleapis.com), got '{hostname}'" ) @@ -127,8 +126,6 @@ class FocusMavvrikDestination(FocusDestination): timeout=30.0, ) if resp.status_code == 410: - # Connector has been disconnected in Mavvrik — reset flag so next - # delivery attempt re-registers after it becomes active again. self._registered = False raise RuntimeError( "Mavvrik FOCUS destination: connector is disconnected (410). " @@ -136,8 +133,7 @@ class FocusMavvrikDestination(FocusDestination): ) if resp.status_code >= 400: raise RuntimeError( - f"Mavvrik FOCUS destination: register failed " - f"({resp.status_code}): {resp.text[:200]}" + f"Mavvrik FOCUS destination: register failed ({resp.status_code}): {resp.text[:200]}" ) self._registered = True metrics_marker = resp.json().get("metricsMarker", 0) @@ -159,8 +155,7 @@ class FocusMavvrikDestination(FocusDestination): ) if resp.status_code >= 400: raise RuntimeError( - f"Mavvrik FOCUS destination: failed to get signed URL " - f"({resp.status_code}): {resp.text[:200]}" + f"Mavvrik FOCUS destination: failed to get signed URL ({resp.status_code}): {resp.text[:200]}" ) signed_url = resp.json().get("url") if not signed_url: @@ -205,8 +200,7 @@ class FocusMavvrikDestination(FocusDestination): ) if init_resp.status_code not in (200, 201): raise RuntimeError( - f"Mavvrik FOCUS destination: GCS session init failed " - f"({init_resp.status_code}): {init_resp.text[:400]}" + f"Mavvrik FOCUS destination: GCS session init failed ({init_resp.status_code}): {init_resp.text[:400]}" ) session_uri = init_resp.headers.get("Location") @@ -217,8 +211,7 @@ class FocusMavvrikDestination(FocusDestination): _validate_gcs_url(session_uri, "session URI") verbose_logger.debug( - "Mavvrik FOCUS destination: GCS session started, uploading %d gzip bytes " - "in %d chunk(s)", + "Mavvrik FOCUS destination: GCS session started, uploading %d gzip bytes in %d chunk(s)", total, max(1, -(-total // _GCS_CHUNK_SIZE)), # ceiling division ) @@ -273,14 +266,33 @@ class FocusMavvrikDestination(FocusDestination): pass raise + async def _update_metrics_marker(self, date_epoch: int) -> None: + """PATCH agent endpoint to advance metricsMarker after a successful upload.""" + resp = await self._http.client.request( + method="PATCH", + url=self._agent_url, + headers=self._auth_headers, + json={"metricsMarker": date_epoch}, + timeout=30.0, + ) + if resp.status_code == 410: + self._registered = False + raise RuntimeError( + "Mavvrik FOCUS destination: connector is disconnected (410). " + "Re-enable the connection in the Mavvrik dashboard." + ) + if resp.status_code >= 400: + raise RuntimeError( + f"Mavvrik FOCUS destination: failed to update metricsMarker " + f"({resp.status_code}): {resp.text[:200]}" + ) + verbose_logger.debug( + "Mavvrik FOCUS destination: metricsMarker advanced to %s", date_epoch + ) + async def get_metrics_marker(self) -> Optional[int]: """Register with Mavvrik and return the current metricsMarker. - The metricsMarker is a Unix timestamp (seconds) representing the last - date Mavvrik has successfully ingested. Called on every scheduled run - so the logger can detect and catch up any dates missed due to previous - export failures. - Always calls the Mavvrik register API — unlike deliver() which skips registration once _registered is True, catch-up requires a fresh marker value on every run. @@ -300,8 +312,7 @@ class FocusMavvrikDestination(FocusDestination): ) if resp.status_code >= 400: raise RuntimeError( - f"Mavvrik FOCUS destination: register failed " - f"({resp.status_code}): {resp.text[:200]}" + f"Mavvrik FOCUS destination: register failed ({resp.status_code}): {resp.text[:200]}" ) self._registered = True metrics_marker = resp.json().get("metricsMarker", 0) @@ -321,14 +332,19 @@ class FocusMavvrikDestination(FocusDestination): Uses the start date of the time window as the object date key. """ + date_str = time_window.start_time.strftime("%Y-%m-%d") + date_epoch = int(time_window.start_time.timestamp()) + + await self._ensure_registered() + if not content: verbose_logger.debug( - "Mavvrik FOCUS destination: empty content, skipping upload" + "Mavvrik FOCUS destination: empty content for date=%s, advancing marker", + date_str, ) + await self._update_metrics_marker(date_epoch) return - date_str = time_window.start_time.strftime("%Y-%m-%d") - verbose_logger.debug( "Mavvrik FOCUS destination: uploading %d bytes for date=%s (%s)", len(content), @@ -336,9 +352,9 @@ class FocusMavvrikDestination(FocusDestination): filename, ) - await self._ensure_registered() signed_url = await self._get_signed_url(date_str) await self._upload_to_gcs(signed_url, content) + await self._update_metrics_marker(date_epoch) verbose_logger.debug( "Mavvrik FOCUS destination: upload complete for date=%s", date_str diff --git a/litellm/integrations/mavvrik_focus/mavvrik_focus_logger.py b/litellm/integrations/mavvrik_focus/mavvrik_focus_logger.py index 47d3e1da7bc..49030ffc2e1 100644 --- a/litellm/integrations/mavvrik_focus/mavvrik_focus_logger.py +++ b/litellm/integrations/mavvrik_focus/mavvrik_focus_logger.py @@ -72,6 +72,16 @@ def _parse_metrics_marker( return None +def _is_empty_metrics_marker(marker: Optional[object]) -> bool: + if marker is None: + return True + if isinstance(marker, (int, float)): + return marker == 0 + if isinstance(marker, str): + return not marker.strip() + return False + + class MavvrikFocusLogger(FocusLogger): """FOCUS-based export logger that routes to the Mavvrik destination.""" @@ -122,19 +132,17 @@ class MavvrikFocusLogger(FocusLogger): window.start_time.date(), window.end_time.date(), ) + payload = b"" if data.is_empty(): verbose_proxy_logger.debug( "Mavvrik FOCUS export: no usage data for window %s", window ) - return - normalized = engine._transformer.transform(data) - if normalized.is_empty(): - return - payload = engine._serializer.serialize(normalized) - if not payload: - return + else: + normalized = engine._transformer.transform(data) + if not normalized.is_empty(): + payload = engine._serializer.serialize(normalized) await engine._destination.deliver( - content=payload, + content=payload or b"", time_window=window, filename=engine._build_filename(window), ) @@ -149,8 +157,8 @@ class MavvrikFocusLogger(FocusLogger): On each run: 1. Register with Mavvrik → get metricsMarker (last successfully ingested date) - 2. If metricsMarker is behind yesterday, catch up missed dates (capped at - _MAX_CATCHUP_DAYS to avoid runaway loops on long outages) + 2. If metricsMarker is behind yesterday (or 0/None for a fresh connector), + catch up missed dates (capped at _MAX_CATCHUP_DAYS) 3. Export yesterday (today's daily window) This ensures a failed export on day N is automatically retried on day N+1 @@ -177,13 +185,19 @@ class MavvrikFocusLogger(FocusLogger): last_ingested = _parse_metrics_marker(marker) - # Catch up missed dates, capped at _MAX_CATCHUP_DAYS - if last_ingested and last_ingested < yesterday: - # Never go further back than _MAX_CATCHUP_DAYS from yesterday - earliest_catchup = yesterday - timedelta(days=self._MAX_CATCHUP_DAYS - 1) - catch_up_date = max(last_ingested + timedelta(days=1), earliest_catchup) + is_empty_marker = _is_empty_metrics_marker(marker) + earliest_catchup = yesterday - timedelta(days=self._MAX_CATCHUP_DAYS - 1) + if is_empty_marker or (last_ingested is not None and last_ingested < yesterday): + catch_up_date = ( + earliest_catchup + if last_ingested is None + else max(last_ingested + timedelta(days=1), earliest_catchup) + ) - if last_ingested + timedelta(days=1) < earliest_catchup: + if ( + last_ingested is not None + and last_ingested + timedelta(days=1) < earliest_catchup + ): verbose_proxy_logger.warning( "Mavvrik FOCUS export: metricsMarker is more than %d days behind " "(%s). Catching up from %s only; earlier data will not be re-exported.", @@ -197,18 +211,24 @@ class MavvrikFocusLogger(FocusLogger): "Mavvrik FOCUS export: catching up missed date %s", catch_up_date.date(), ) + # Use now as end_time for catch-up windows too — rows for old dates + # may have been flushed to DB well after their calendar day ended. + catch_up_end = min(catch_up_date + timedelta(days=1), now) window = FocusTimeWindow( start_time=catch_up_date, - end_time=catch_up_date + timedelta(days=1), + end_time=catch_up_end, frequency="daily", ) await self._export_window(window=window, limit=None) catch_up_date += timedelta(days=1) - # Export yesterday's window (the normal daily run) + # Export yesterday's window (the normal daily run). + # Use `now` as end_time so spend rows flushed after midnight are included. + # LiteLLM's DailyUserSpend rows for a given date keep getting updated_at + # bumped as the flush job runs; capping at midnight would miss those updates. window = FocusTimeWindow( start_time=yesterday, - end_time=yesterday + timedelta(days=1), + end_time=now, frequency="daily", ) await self._export_window(window=window, limit=None) @@ -253,6 +273,21 @@ class MavvrikFocusLogger(FocusLogger): ) if type(cb) is MavvrikFocusLogger ] + if not loggers and "mavvrik" in litellm.callbacks: + # The logger is registered as the string "mavvrik" but hasn't been + # instantiated yet (lazy init happens on first LLM call). Force it now + # so the scheduler can register the daily export job at startup. + from litellm.litellm_core_utils.litellm_logging import ( # noqa: PLC0415 + _init_custom_logger_compatible_class, + ) + + instance = _init_custom_logger_compatible_class( + logging_integration="mavvrik", + internal_usage_cache=None, + llm_router=None, + ) + if isinstance(instance, MavvrikFocusLogger): + loggers = [instance] if not loggers: verbose_proxy_logger.debug( "No MavvrikFocusLogger registered; skipping scheduler" diff --git a/litellm/llms/anthropic/chat/handler.py b/litellm/llms/anthropic/chat/handler.py index b1212f93059..7154a6f3595 100644 --- a/litellm/llms/anthropic/chat/handler.py +++ b/litellm/llms/anthropic/chat/handler.py @@ -11,7 +11,6 @@ from typing import ( Dict, List, Literal, - Optional, Tuple, Union, cast, @@ -73,17 +72,17 @@ if TYPE_CHECKING: async def make_call( - client: Optional[AsyncHTTPHandler], + client: AsyncHTTPHandler | None, api_base: str, headers: dict, data: str, model: str, messages: list, logging_obj, - timeout: Optional[Union[float, httpx.Timeout]], + timeout: Union[float, httpx.Timeout] | None, json_mode: bool, - speed: Optional[str] = None, - tool_name_reverse_map: Optional[Dict[str, str]] = None, + speed: str | None = None, + tool_name_reverse_map: Dict[str, str] | None = None, ) -> Tuple[Any, httpx.Headers]: if client is None: client = litellm.module_level_aclient @@ -133,17 +132,17 @@ async def make_call( def make_sync_call( - client: Optional[HTTPHandler], + client: HTTPHandler | None, api_base: str, headers: dict, data: str, model: str, messages: list, logging_obj, - timeout: Optional[Union[float, httpx.Timeout]], + timeout: Union[float, httpx.Timeout] | None, json_mode: bool, - speed: Optional[str] = None, - tool_name_reverse_map: Optional[Dict[str, str]] = None, + speed: str | None = None, + tool_name_reverse_map: Dict[str, str] | None = None, ) -> Tuple[Any, httpx.Headers]: if client is None: client = litellm.module_level_client # re-use a module level client @@ -213,7 +212,7 @@ class AnthropicChatCompletion(BaseLLM): model_response: ModelResponse, print_verbose: Callable, timeout: Union[float, httpx.Timeout], - client: Optional[AsyncHTTPHandler], + client: AsyncHTTPHandler | None, encoding, api_key, logging_obj, @@ -277,7 +276,7 @@ class AnthropicChatCompletion(BaseLLM): provider_config: "BaseConfig", logger_fn=None, headers={}, - client: Optional[AsyncHTTPHandler] = None, + client: AsyncHTTPHandler | None = None, ) -> Union[ModelResponse, "CustomStreamWrapper"]: async_handler = client or get_async_httpx_client( llm_provider=litellm.LlmProviders.ANTHROPIC @@ -539,9 +538,9 @@ class ModelResponseIterator: self, streaming_response, sync_stream: bool, - json_mode: Optional[bool] = False, - speed: Optional[str] = None, - tool_name_reverse_map: Optional[Dict[str, str]] = None, + json_mode: bool | None = False, + speed: str | None = None, + tool_name_reverse_map: Dict[str, str] | None = None, ): self.streaming_response = streaming_response self.response_iterator = self.streaming_response @@ -571,7 +570,7 @@ class ModelResponseIterator: # Track current content block type to avoid emitting tool calls for non-tool blocks # See: https://github.com/BerriAI/litellm/issues/17254 - self.current_content_block_type: Optional[str] = None + self.current_content_block_type: str | None = None # Accumulate web_search_tool_result blocks for multi-turn reconstruction # See: https://github.com/BerriAI/litellm/issues/17737 @@ -587,8 +586,8 @@ class ModelResponseIterator: # Track server tool use inputs and results for code_interpreter_results self._server_tool_inputs: Dict[str, Any] = {} self.tool_results: List[Dict[str, Any]] = [] - self._current_server_tool_id: Optional[str] = None - self._container_id: Optional[str] = None + self._current_server_tool_id: str | None = None + self._container_id: str | None = None def check_empty_tool_call_args(self) -> bool: """ @@ -629,16 +628,18 @@ class ModelResponseIterator: self, chunk: dict ) -> Tuple[ str, - Optional[ChatCompletionToolCallChunk], + ChatCompletionToolCallChunk | None, List[Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock]], Dict[str, Any], + str | None, ]: """ Helper function to handle the content block delta """ text = "" - tool_use: Optional[ChatCompletionToolCallChunk] = None + tool_use: ChatCompletionToolCallChunk | None = None provider_specific_fields = {} + reasoning_content: str | None = None content_block = ContentBlockDelta(**chunk) # type: ignore thinking_blocks: List[ Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock] @@ -673,14 +674,31 @@ class ModelResponseIterator: thinking_content = content_block["delta"].get("thinking") if isinstance(thinking_content, str) and thinking_content: self.reasoning_content_chunks.append(thinking_content) - thinking_blocks = [ - ChatCompletionThinkingBlock( - type="thinking", - thinking=thinking_content or "", - signature=str(content_block["delta"].get("signature") or ""), - ) - ] - provider_specific_fields["thinking_blocks"] = thinking_blocks + reasoning_content = thinking_content + thinking_blocks = [ + ChatCompletionThinkingBlock( + type="thinking", + thinking=thinking_content, + ) + ] + provider_specific_fields["thinking_blocks"] = thinking_blocks + + signature = content_block["delta"].get("signature") + if isinstance(signature, str) and signature: + thinking_blocks = [ + ChatCompletionThinkingBlock( + type="thinking", + thinking="".join( + cast(str, block["delta"].get("thinking")) + for block in self.content_blocks + if isinstance(block["delta"].get("thinking"), str) + ), + signature=signature, + ) + ] + provider_specific_fields["thinking_blocks"] = thinking_blocks + if reasoning_content is None: + reasoning_content = "" elif ( "content" in content_block["delta"] and content_block["delta"].get("type") == "compaction_delta" @@ -691,25 +709,13 @@ class ModelResponseIterator: "content": content_block["delta"]["content"], } - return text, tool_use, thinking_blocks, provider_specific_fields - - def _handle_reasoning_content( - self, - thinking_blocks: List[ - Union[ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock] - ], - ) -> Optional[str]: - """ - Handle the reasoning content - """ - reasoning_content = None - for block in thinking_blocks: - thinking_content = cast(Optional[str], block.get("thinking")) - if reasoning_content is None: - reasoning_content = "" - if thinking_content is not None: - reasoning_content += thinking_content - return reasoning_content + return ( + text, + tool_use, + thinking_blocks, + provider_specific_fields, + reasoning_content, + ) def _handle_redacted_thinking_content( self, @@ -780,18 +786,19 @@ class ModelResponseIterator: type_chunk = chunk.get("type", "") or "" text = "" - tool_use: Optional[ChatCompletionToolCallChunk] = None + tool_use: ChatCompletionToolCallChunk | None = None finish_reason = "" - usage: Optional[Usage] = None + usage: Usage | None = None provider_specific_fields: Dict[str, Any] = {} - reasoning_content: Optional[str] = None - thinking_blocks: Optional[ + reasoning_content: str | None = None + thinking_blocks: ( List[ Union[ ChatCompletionThinkingBlock, ChatCompletionRedactedThinkingBlock ] ] - ] = None + | None + ) = None # Always use index=0 for OpenAI choice format (fixes multi-choice errors) index = 0 @@ -805,11 +812,8 @@ class ModelResponseIterator: tool_use, thinking_blocks, provider_specific_fields, + reasoning_content, ) = self._content_block_delta_helper(chunk=chunk) - if thinking_blocks: - reasoning_content = self._handle_reasoning_content( - thinking_blocks=thinking_blocks - ) elif type_chunk == "content_block_start": """ event: content_block_start @@ -1061,8 +1065,8 @@ class ModelResponseIterator: raise ValueError(f"Failed to decode JSON from chunk: {chunk}") def _handle_json_mode_chunk( - self, text: str, tool_use: Optional[ChatCompletionToolCallChunk] - ) -> Tuple[str, Optional[ChatCompletionToolCallChunk]]: + self, text: str, tool_use: ChatCompletionToolCallChunk | None + ) -> Tuple[str, ChatCompletionToolCallChunk | None]: """ If JSON mode is enabled, convert the tool call to a message. @@ -1110,7 +1114,7 @@ class ModelResponseIterator: def _handle_message_delta( self, chunk: dict - ) -> Tuple[str, Optional[Usage], Optional[Dict[str, Any]]]: + ) -> Tuple[str, Usage | None, Dict[str, Any] | None]: """ Handle message_delta event for finish_reason, usage, and container. @@ -1134,7 +1138,7 @@ class ModelResponseIterator: def _handle_accumulated_json_chunk( self, data_str: str - ) -> Optional[ModelResponseStream]: + ) -> ModelResponseStream | None: """ Handle partial JSON chunks by accumulating them until valid JSON is received. @@ -1159,7 +1163,7 @@ class ModelResponseIterator: # If it's not valid JSON yet, continue to the next chunk return None - def _parse_sse_data(self, str_line: str) -> Optional[ModelResponseStream]: + def _parse_sse_data(self, str_line: str) -> ModelResponseStream | None: """ Parse SSE data line, handling both complete and partial JSON chunks. diff --git a/litellm/llms/base_llm/chat/transformation.py b/litellm/llms/base_llm/chat/transformation.py index 8f9d5cad7c4..4f7e98af780 100644 --- a/litellm/llms/base_llm/chat/transformation.py +++ b/litellm/llms/base_llm/chat/transformation.py @@ -377,6 +377,17 @@ class BaseConfig(ABC): ) -> "ModelResponse": pass + def transform_parsed_response_dict(self, parsed_response: dict) -> dict: + """ + Repair a parsed OpenAI-format response dict before generic conversion. + + Providers routed through the OpenAI SDK handler bypass transform_response, + which calls convert_to_model_response_object directly on the SDK's parsed + output. Override this to normalize a malformed response (e.g. github_copilot + returning empty choices for Anthropic-native Claude responses). + """ + return parsed_response + @abstractmethod def get_error_class( self, error_message: str, status_code: int, headers: Union[dict, httpx.Headers] diff --git a/litellm/llms/base_llm/rerank/transformation.py b/litellm/llms/base_llm/rerank/transformation.py index 166f876ba04..eac44ba85c5 100644 --- a/litellm/llms/base_llm/rerank/transformation.py +++ b/litellm/llms/base_llm/rerank/transformation.py @@ -1,5 +1,5 @@ from abc import ABC, abstractmethod -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union +from typing import TYPE_CHECKING, Any, Dict, List, Tuple, Union import httpx @@ -22,8 +22,8 @@ class BaseRerankConfig(ABC): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> dict: pass @@ -33,7 +33,7 @@ class BaseRerankConfig(ABC): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: return {} @@ -44,7 +44,7 @@ class BaseRerankConfig(ABC): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, @@ -54,9 +54,9 @@ class BaseRerankConfig(ABC): @abstractmethod def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: """ OPTIONAL @@ -79,12 +79,13 @@ class BaseRerankConfig(ABC): drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: pass @@ -100,9 +101,9 @@ class BaseRerankConfig(ABC): def calculate_rerank_cost( self, model: str, - custom_llm_provider: Optional[str] = None, - billed_units: Optional[RerankBilledUnits] = None, - model_info: Optional[ModelInfo] = None, + custom_llm_provider: str | None = None, + billed_units: RerankBilledUnits | None = None, + model_info: ModelInfo | None = None, ) -> Tuple[float, float]: """ Calculates the cost per query for a given rerank model. diff --git a/litellm/llms/bedrock/common_utils.py b/litellm/llms/bedrock/common_utils.py index 69c0a8529b4..44312eb3926 100644 --- a/litellm/llms/bedrock/common_utils.py +++ b/litellm/llms/bedrock/common_utils.py @@ -601,6 +601,15 @@ def extract_model_name_from_bedrock_arn(model: str) -> str: return model +def is_bedrock_application_inference_profile_arn(model: str) -> bool: + """ + An application inference profile ARN ends in an opaque id with no provider + substring, so the invoke path cannot resolve a provider from it. Such ARNs + must use the converse route, which needs no provider. + """ + return ":application-inference-profile/" in model + + def strip_bedrock_routing_prefix(model: str) -> str: """Strip LiteLLM routing prefixes from model name.""" for prefix in ["bedrock/", "converse/", "invoke/", "openai/", "nova-2/", "nova/"]: @@ -919,6 +928,9 @@ class BedrockModelInfo(BaseLLMModelInfo): ) or _model_after_bedrock.startswith("nova/"): return "converse" + if is_bedrock_application_inference_profile_arn(model): + return "converse" + base_model = BedrockModelInfo.get_base_model(model) alt_model = BedrockModelInfo.get_non_litellm_routing_model_name(model=model) if ( diff --git a/litellm/llms/bedrock/embed/embedding.py b/litellm/llms/bedrock/embed/embedding.py index bc72f04deac..e07ccb8c11b 100644 --- a/litellm/llms/bedrock/embed/embedding.py +++ b/litellm/llms/bedrock/embed/embedding.py @@ -226,6 +226,10 @@ class BedrockEmbedding(BaseAWSLLM): returned_response = AmazonTitanV2Config()._transform_response( response_list=response_list, model=model ) + elif model == "amazon.titan-embed-g1-text-02": + returned_response = AmazonTitanG1Config()._transform_response( + response_list=response_list, model=model + ) elif provider == "twelvelabs": returned_response = ( TwelveLabsMarengoEmbeddingConfig()._transform_response( @@ -449,6 +453,7 @@ class BedrockEmbedding(BaseAWSLLM): "amazon.titan-embed-image-v1", "amazon.titan-embed-text-v1", "amazon.titan-embed-text-v2:0", + "amazon.titan-embed-g1-text-02", ]: batch_data = [] for i in input: @@ -466,6 +471,10 @@ class BedrockEmbedding(BaseAWSLLM): transformed_request = AmazonTitanV2Config()._transform_request( input=i, inference_params=inference_params ) + elif model == "amazon.titan-embed-g1-text-02": + transformed_request = AmazonTitanG1Config()._transform_request( + input=i, inference_params=inference_params + ) else: raise Exception( "Unmapped model. Received={}. Expected={}".format( @@ -474,6 +483,7 @@ class BedrockEmbedding(BaseAWSLLM): "amazon.titan-embed-image-v1", "amazon.titan-embed-text-v1", "amazon.titan-embed-text-v2:0", + "amazon.titan-embed-g1-text-02", ], ) ) diff --git a/litellm/llms/cohere/rerank/guardrail_translation/handler.py b/litellm/llms/cohere/rerank/guardrail_translation/handler.py index e9a5823d2b8..0824e1cca41 100644 --- a/litellm/llms/cohere/rerank/guardrail_translation/handler.py +++ b/litellm/llms/cohere/rerank/guardrail_translation/handler.py @@ -26,11 +26,18 @@ class CohereRerankHandler(BaseTranslation): The handler specifically processes: - The 'query' parameter (string) + - The 'instruction' parameter (string), when present Note: Documents are not processed by guardrails as they are the corpus being searched, not user input. """ + # User-controlled free-text fields that reach the model and must be + # scanned. 'instruction' is folded into the prompt by instruction-aware + # rerankers (e.g. hosted vLLM / Qwen3-Reranker), so it is as sensitive as + # 'query'; omitting it would let a caller smuggle content past guardrails. + _SCANNED_FIELDS = ("query", "instruction") + async def process_input_messages( self, data: dict, @@ -38,42 +45,55 @@ class CohereRerankHandler(BaseTranslation): litellm_logging_obj: Optional[Any] = None, ) -> Any: """ - Process input query by applying guardrails. + Process input text fields ('query' and 'instruction') by applying + guardrails and writing the sanitized values back. Args: - data: Request data dictionary containing 'query' + data: Request data dictionary containing 'query' and optionally + 'instruction' guardrail_to_apply: The guardrail instance to apply Returns: - Modified data with guardrails applied to query only + Modified data with guardrails applied to query/instruction only """ - # Process query only - query = data.get("query") - if query is not None and isinstance(query, str): - inputs = GenericGuardrailAPIInputs(texts=[query]) - # Include model information if available - model = data.get("model") - if model: - inputs["model"] = model - guardrailed_inputs = await guardrail_to_apply.apply_guardrail( - inputs=inputs, - request_data=data, - input_type="request", - logging_obj=litellm_logging_obj, + # Collect every scannable text field in a stable order so the + # guardrailed results can be written back to the right key by index. + fields_to_scan = [ + (key, data[key]) + for key in self._SCANNED_FIELDS + if isinstance(data.get(key), str) + ] + if not fields_to_scan: + verbose_proxy_logger.debug( + "Rerank: No query/instruction to process or not strings" ) - guardrailed_texts = guardrailed_inputs.get("texts", []) - data["query"] = guardrailed_texts[0] if guardrailed_texts else query + return data - verbose_proxy_logger.debug( - "Rerank: Applied guardrail to query. " - "Original length: %d, New length: %d", - len(query), - len(data["query"]), - ) - else: - verbose_proxy_logger.debug( - "Rerank: No query to process or query is not a string" - ) + inputs = GenericGuardrailAPIInputs(texts=[value for _, value in fields_to_scan]) + # Include model information if available + model = data.get("model") + if model: + inputs["model"] = model + guardrailed_inputs = await guardrail_to_apply.apply_guardrail( + inputs=inputs, + request_data=data, + input_type="request", + logging_obj=litellm_logging_obj, + ) + guardrailed_texts = guardrailed_inputs.get("texts", []) + + for idx, (key, original) in enumerate(fields_to_scan): + # Defensive: only write back when the guardrail returned a value for + # this index; otherwise keep the original (never forward unscanned). + if idx < len(guardrailed_texts): + data[key] = guardrailed_texts[idx] + verbose_proxy_logger.debug( + "Rerank: Applied guardrail to %s. " + "Original length: %d, New length: %d", + key, + len(original), + len(data[key]), + ) return data diff --git a/litellm/llms/cohere/rerank/transformation.py b/litellm/llms/cohere/rerank/transformation.py index 64ae8e8ffa7..dd3f0f1a446 100644 --- a/litellm/llms/cohere/rerank/transformation.py +++ b/litellm/llms/cohere/rerank/transformation.py @@ -1,4 +1,4 @@ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union import httpx @@ -22,9 +22,9 @@ class CohereRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: if api_base: # Remove trailing slashes and ensure clean base URL @@ -46,17 +46,18 @@ class CohereRerankConfig(BaseRerankConfig): def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: """ Map Cohere rerank params @@ -78,8 +79,8 @@ class CohereRerankConfig(BaseRerankConfig): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> dict: if api_key is None: api_key = ( @@ -111,7 +112,7 @@ class CohereRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: if "query" not in optional_rerank_params: raise ValueError("query is required for Cohere rerank") @@ -134,7 +135,7 @@ class CohereRerankConfig(BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, diff --git a/litellm/llms/cohere/rerank_v2/transformation.py b/litellm/llms/cohere/rerank_v2/transformation.py index 4c800d6455d..7c68a431a90 100644 --- a/litellm/llms/cohere/rerank_v2/transformation.py +++ b/litellm/llms/cohere/rerank_v2/transformation.py @@ -1,4 +1,4 @@ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union from litellm.llms.cohere.rerank.transformation import CohereRerankConfig from litellm.types.rerank import OptionalRerankParams, RerankRequest @@ -14,9 +14,9 @@ class CohereRerankV2Config(CohereRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: if api_base: # Remove trailing slashes and ensure clean base URL @@ -38,17 +38,18 @@ class CohereRerankV2Config(CohereRerankConfig): def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: """ Map Cohere rerank params @@ -71,7 +72,7 @@ class CohereRerankV2Config(CohereRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: if "query" not in optional_rerank_params: raise ValueError("query is required for Cohere rerank") diff --git a/litellm/llms/dashscope/rerank/transformation.py b/litellm/llms/dashscope/rerank/transformation.py index 629f3cf4af7..365e15fdd7a 100644 --- a/litellm/llms/dashscope/rerank/transformation.py +++ b/litellm/llms/dashscope/rerank/transformation.py @@ -22,7 +22,7 @@ as supported only for gte-rerank-v2 / qwen3-vl-rerank. Docs - https://help.aliyun.com/zh/model-studio/text-rerank-api """ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union import httpx @@ -59,9 +59,9 @@ class DashScopeRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: if api_base is None: api_base = get_secret_str("DASHSCOPE_API_BASE_RERANK") or DEFAULT_RERANK_URL @@ -83,8 +83,8 @@ class DashScopeRerankConfig(BaseRerankConfig): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> dict: if api_key is None: api_key = get_secret_str("DASHSCOPE_API_KEY") @@ -105,17 +105,18 @@ class DashScopeRerankConfig(BaseRerankConfig): def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: # qwen3-rerank accepts query/documents/top_n/return_documents. The # rest (rank_fields, max_*_per_doc) are silently dropped. @@ -134,7 +135,7 @@ class DashScopeRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: if "query" not in optional_rerank_params: raise ValueError("query is required for DashScope rerank") @@ -158,10 +159,10 @@ class DashScopeRerankConfig(BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, - request_data: Optional[dict] = None, - optional_params: Optional[dict] = None, - litellm_params: Optional[dict] = None, + api_key: str | None = None, + request_data: dict | None = None, + optional_params: dict | None = None, + litellm_params: dict | None = None, ) -> RerankResponse: request_data = request_data or {} optional_params = optional_params or {} diff --git a/litellm/llms/deepinfra/rerank/transformation.py b/litellm/llms/deepinfra/rerank/transformation.py index e4bfbcb2513..385aa051d00 100644 --- a/litellm/llms/deepinfra/rerank/transformation.py +++ b/litellm/llms/deepinfra/rerank/transformation.py @@ -2,7 +2,7 @@ Translate between Cohere's `/rerank` format and Deepinfra's `/rerank` format. """ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union import httpx @@ -30,9 +30,9 @@ class DeepinfraRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: """ Constructs the complete DeepInfra inference endpoint URL for rerank. @@ -67,8 +67,8 @@ class DeepinfraRerankConfig(BaseRerankConfig): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> dict: if api_key is None: api_key = get_secret_str("DEEPINFRA_API_KEY") @@ -98,12 +98,13 @@ class DeepinfraRerankConfig(BaseRerankConfig): drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: # Start with the basic parameters optional_rerank_params = {} @@ -132,7 +133,7 @@ class DeepinfraRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: # Convert OptionalRerankParams to dict as expected by parent class if optional_rerank_params is None: @@ -145,7 +146,7 @@ class DeepinfraRerankConfig(BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, diff --git a/litellm/llms/deepseek/chat/transformation.py b/litellm/llms/deepseek/chat/transformation.py index 7ed3e484535..b90b1e1aa21 100644 --- a/litellm/llms/deepseek/chat/transformation.py +++ b/litellm/llms/deepseek/chat/transformation.py @@ -146,6 +146,86 @@ class DeepSeekChatConfig(OpenAIGPTConfig): and (optional_params.get("thinking") or {}).get("type") == "enabled" ) + @staticmethod + def _drop_unsupported_tools(optional_params: dict) -> dict: + """ + DeepSeek's /chat/completions only accepts tools of type "function". + + Requests bridged from /v1/responses can carry responses-API-native tool + types (e.g. a Codex CLI tool typed "namespace"); DeepSeek rejects the + whole request with `unknown variant '', expected 'function'` (issue + #30722). Drop the unsupported entries so the function tools still go + through, and drop the now-dangling tool_choice/parallel_tool_calls when + nothing callable survives. + + When a specific `tool_choice` points at a dropped tool, clear it so the + sanitized request does not reference a tool DeepSeek will never receive. + """ + tools = optional_params.get("tools") + if not isinstance(tools, list) or not tools: + return optional_params + + def _is_function_tool(tool: object) -> bool: + return isinstance(tool, dict) and tool.get("type") == "function" + + def _get_function_tool_name(tool: object) -> str | None: + if not isinstance(tool, dict): + return None + function = tool.get("function") + if not isinstance(function, dict): + return None + name = function.get("name") + return name if isinstance(name, str) else None + + def _tool_choice_matches_function_tool( + tool_choice: object, function_tool_names: set[str] + ) -> bool: + if not isinstance(tool_choice, dict): + return True + if tool_choice.get("type") != "function": + return False + function = tool_choice.get("function") + if not isinstance(function, dict): + return False + name = function.get("name") + return isinstance(name, str) and name in function_tool_names + + function_tools = [tool for tool in tools if _is_function_tool(tool)] + if len(function_tools) == len(tools): + return optional_params + + dropped_types = sorted( + { + str(tool.get("type")) if isinstance(tool, dict) else type(tool).__name__ + for tool in tools + if not _is_function_tool(tool) + } + ) + litellm.verbose_logger.warning( + "DeepSeek chat completions only supports function tools; dropping " + "unsupported tool type(s) %s before sending the request", + dropped_types, + ) + + cleaned = {k: v for k, v in optional_params.items() if k != "tools"} + if function_tools: + function_tool_names = { + name + for tool in function_tools + for name in (_get_function_tool_name(tool),) + if name is not None + } + if not _tool_choice_matches_function_tool( + cleaned.get("tool_choice"), function_tool_names + ): + cleaned = {k: v for k, v in cleaned.items() if k != "tool_choice"} + return {**cleaned, "tools": function_tools} + return { + k: v + for k, v in cleaned.items() + if k not in ("tool_choice", "parallel_tool_calls") + } + def transform_request( self, model: str, @@ -163,6 +243,7 @@ class DeepSeekChatConfig(OpenAIGPTConfig): (user explicitly enabled it), preventing spurious injection on models like deepseek-v3.2 that support thinking as opt-in but not always-on. """ + optional_params = self._drop_unsupported_tools(optional_params) if self._thinking_mode_active(model=model, optional_params=optional_params): messages = self._fill_reasoning_content(messages) return super().transform_request( @@ -185,6 +266,7 @@ class DeepSeekChatConfig(OpenAIGPTConfig): Async equivalent of transform_request — applies the same reasoning_content fix for multi-turn thinking-mode conversations. """ + optional_params = self._drop_unsupported_tools(optional_params) if self._thinking_mode_active(model=model, optional_params=optional_params): messages = self._fill_reasoning_content(messages) return await super().async_transform_request( diff --git a/litellm/llms/fireworks_ai/rerank/transformation.py b/litellm/llms/fireworks_ai/rerank/transformation.py index 4a7b64b9b77..400d511a02f 100644 --- a/litellm/llms/fireworks_ai/rerank/transformation.py +++ b/litellm/llms/fireworks_ai/rerank/transformation.py @@ -4,7 +4,7 @@ Fireworks AI Rerank API transformation Reference: https://docs.fireworks.ai/inference-api-reference/rerank """ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union import httpx @@ -29,9 +29,9 @@ class FireworksAIRerankConfig(FireworksAIMixin, BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: if api_base: # Remove trailing slashes and ensure clean base URL @@ -56,17 +56,18 @@ class FireworksAIRerankConfig(FireworksAIMixin, BaseRerankConfig): def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict[str, Any]: """ Map Cohere rerank params to Fireworks AI rerank params @@ -101,8 +102,8 @@ class FireworksAIRerankConfig(FireworksAIMixin, BaseRerankConfig): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> dict: api_key = self._get_api_key(api_key) if api_key is None: @@ -127,7 +128,7 @@ class FireworksAIRerankConfig(FireworksAIMixin, BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: """ Transform request to Fireworks AI rerank format @@ -175,7 +176,7 @@ class FireworksAIRerankConfig(FireworksAIMixin, BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, @@ -220,7 +221,7 @@ class FireworksAIRerankConfig(FireworksAIMixin, BaseRerankConfig): rerank_meta = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens) # Extract results - Fireworks AI uses "data" instead of "results" - _results: Optional[List[dict]] = raw_response_json.get( + _results: List[dict] | None = raw_response_json.get( "data" ) or raw_response_json.get("results") diff --git a/litellm/llms/github_copilot/chat/transformation.py b/litellm/llms/github_copilot/chat/transformation.py index 72dacb59f8a..54cbf69a4ac 100644 --- a/litellm/llms/github_copilot/chat/transformation.py +++ b/litellm/llms/github_copilot/chat/transformation.py @@ -1,5 +1,5 @@ import json -from typing import Any, List, Optional, Tuple +from typing import Any, List, Tuple import os @@ -22,8 +22,8 @@ from ..common_utils import ( class GithubCopilotConfig(OpenAIConfig): def __init__( self, - api_key: Optional[str] = None, - api_base: Optional[str] = None, + api_key: str | None = None, + api_base: str | None = None, custom_llm_provider: str = "openai", ) -> None: super().__init__() @@ -32,10 +32,10 @@ class GithubCopilotConfig(OpenAIConfig): def _get_openai_compatible_provider_info( self, model: str, - api_base: Optional[str], - api_key: Optional[str], + api_base: str | None, + api_key: str | None, custom_llm_provider: str, - ) -> Tuple[Optional[str], Optional[str], str]: + ) -> Tuple[str | None, str | None, str]: dynamic_api_base = ( api_base or self.authenticator.get_api_base() @@ -85,8 +85,8 @@ class GithubCopilotConfig(OpenAIConfig): messages: List[AllMessageValues], optional_params: dict, litellm_params: dict, - api_key: Optional[str] = None, - api_base: Optional[str] = None, + api_key: str | None = None, + api_base: str | None = None, ) -> dict: # Get base headers from parent validated_headers = super().validate_environment( @@ -173,7 +173,7 @@ class GithubCopilotConfig(OpenAIConfig): @staticmethod def _parse_anthropic_native_content( content_blocks: List[Any], - ) -> Tuple[str, List[ChatCompletionToolCallChunk], Optional[List[Any]]]: + ) -> Tuple[str, List[ChatCompletionToolCallChunk], List[Any] | None]: """ Parse Anthropic-native content blocks into OpenAI-compatible fields. @@ -194,6 +194,88 @@ class GithubCopilotConfig(OpenAIConfig): ) return text_content, tool_calls, thinking_blocks + @staticmethod + def _normalize_anthropic_usage(usage: dict) -> dict: + normalized = dict(usage) + if "input_tokens" in usage and "prompt_tokens" not in usage: + normalized["prompt_tokens"] = usage["input_tokens"] + if "output_tokens" in usage and "completion_tokens" not in usage: + normalized["completion_tokens"] = usage["output_tokens"] + if "total_tokens" not in normalized: + normalized["total_tokens"] = normalized.get( + "prompt_tokens", 0 + ) + normalized.get("completion_tokens", 0) + return normalized + + @classmethod + def _synthesize_choices_for_anthropic_native(cls, response_json: dict) -> dict: + """ + Synthesize a `choices` array from an Anthropic-native Copilot response. + + Newer Copilot Claude models (e.g. opus-4.7, opus-4.8) return content + blocks and `stop_reason` without an OpenAI-style `choices` array, and the + max_tokens=1 probe returns no content at all. Returns the response + unchanged when it already carries choices. + + See: https://github.com/BerriAI/litellm/issues/29391 + """ + if response_json.get("choices"): + return response_json + + content = "" + tool_calls: List[ChatCompletionToolCallChunk] = [] + thinking_blocks: List[Any] | None = None + raw_content = response_json.get("content") + if isinstance(raw_content, list): + content, tool_calls, thinking_blocks = cls._parse_anthropic_native_content( + raw_content + ) + elif isinstance(raw_content, str): + content = raw_content + + stop_reason = response_json.get("stop_reason") + finish_reason_map = { + "end_turn": "stop", + "max_tokens": "length", + "stop_sequence": "stop", + "tool_use": "tool_calls", + } + if tool_calls: + finish_reason = "tool_calls" + elif stop_reason in finish_reason_map: + finish_reason = finish_reason_map[stop_reason] + elif content: + finish_reason = "stop" + else: + finish_reason = "length" + + message: dict = { + "role": "assistant", + "content": content if content or not tool_calls else None, + } + if tool_calls: + message["tool_calls"] = tool_calls + if thinking_blocks: + message["thinking_blocks"] = thinking_blocks + + synthesized = { + **response_json, + "choices": [ + {"index": 0, "message": message, "finish_reason": finish_reason} + ], + } + usage = response_json.get("usage") + if isinstance(usage, dict): + synthesized["usage"] = cls._normalize_anthropic_usage(usage) + return synthesized + + def transform_parsed_response_dict(self, parsed_response: dict) -> dict: + """ + Repair the OpenAI-SDK-parsed response on the handler path that bypasses + transform_response. See: https://github.com/BerriAI/litellm/issues/30927 + """ + return self._synthesize_choices_for_anthropic_native(parsed_response) + def transform_response( self, model: str, @@ -205,18 +287,9 @@ class GithubCopilotConfig(OpenAIConfig): optional_params: dict, litellm_params: dict, encoding: Any, - api_key: Optional[str] = None, - json_mode: Optional[bool] = None, + api_key: str | None = None, + json_mode: bool | None = None, ) -> "ModelResponse": - """ - Handle newer Copilot models (e.g. claude-opus-4.7, claude-opus-4.8) that - return Anthropic-native format responses without a `choices` array. - - Synthesizes the missing `choices` from Anthropic-native fields, then - delegates to the parent so all standard post-processing applies. - - See: https://github.com/BerriAI/litellm/issues/29391 - """ try: response_json = raw_response.json() except Exception: @@ -235,70 +308,12 @@ class GithubCopilotConfig(OpenAIConfig): ) if not response_json.get("choices"): - content = "" - tool_calls: List[ChatCompletionToolCallChunk] = [] - thinking_blocks: Optional[List[Any]] = None - if "content" in response_json and isinstance( - response_json["content"], list - ): - content, tool_calls, thinking_blocks = ( - self._parse_anthropic_native_content(response_json["content"]) - ) - elif isinstance(response_json.get("content"), str): - content = response_json["content"] - - stop_reason = response_json.get("stop_reason") - finish_reason_map = { - "end_turn": "stop", - "max_tokens": "length", - "stop_sequence": "stop", - "tool_use": "tool_calls", - } - # Prefer tool_calls when blocks were extracted; otherwise map stop_reason. - if tool_calls: - finish_reason = "tool_calls" - elif stop_reason in finish_reason_map: - finish_reason = finish_reason_map[stop_reason] - elif content: - finish_reason = "stop" - else: - finish_reason = "length" - - message: dict = { - "role": "assistant", - "content": content if content or not tool_calls else None, - } - if tool_calls: - message["tool_calls"] = tool_calls - if thinking_blocks: - message["thinking_blocks"] = thinking_blocks - - response_json["choices"] = [ - { - "index": 0, - "message": message, - "finish_reason": finish_reason, - } - ] - - if "usage" in response_json: - usage = response_json["usage"] - if "input_tokens" in usage and "prompt_tokens" not in usage: - usage["prompt_tokens"] = usage["input_tokens"] - if "output_tokens" in usage and "completion_tokens" not in usage: - usage["completion_tokens"] = usage["output_tokens"] - if "total_tokens" not in usage: - usage["total_tokens"] = usage.get("prompt_tokens", 0) + usage.get( - "completion_tokens", 0 - ) - - # Build a patched response so super() sees valid JSON with choices - patched = httpx.Response( + response_json = self._synthesize_choices_for_anthropic_native(response_json) + raw_response = httpx.Response( status_code=raw_response.status_code, headers=raw_response.headers, content=json.dumps(response_json).encode(), ) - raw_response = patched return super().transform_response( model=model, diff --git a/litellm/llms/hosted_vllm/rerank/transformation.py b/litellm/llms/hosted_vllm/rerank/transformation.py index 60b6dc7d23d..47495350460 100644 --- a/litellm/llms/hosted_vllm/rerank/transformation.py +++ b/litellm/llms/hosted_vllm/rerank/transformation.py @@ -2,7 +2,7 @@ Transformation logic for Hosted VLLM rerank """ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union import httpx @@ -28,7 +28,7 @@ class HostedVLLMRerankError(BaseLLMException): self, status_code: int, message: str, - headers: Optional[Union[dict, httpx.Headers]] = None, + headers: Union[dict, httpx.Headers] | None = None, ): super().__init__(status_code=status_code, message=message, headers=headers) @@ -39,9 +39,9 @@ class HostedVLLMRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: if api_base: # Remove trailing slashes and ensure clean base URL @@ -61,21 +61,23 @@ class HostedVLLMRerankConfig(BaseRerankConfig): "top_n", "rank_fields", "return_documents", + "instruction", ] def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: """ Map parameters for Hosted VLLM rerank @@ -83,22 +85,28 @@ class HostedVLLMRerankConfig(BaseRerankConfig): if max_chunks_per_doc is not None: raise ValueError("Hosted VLLM does not support max_chunks_per_doc") - return dict( - OptionalRerankParams( - query=query, - documents=documents, - top_n=top_n, - rank_fields=rank_fields, - return_documents=return_documents, - ) + mapped_params = OptionalRerankParams( + query=query, + documents=documents, + top_n=top_n, + rank_fields=rank_fields, + return_documents=return_documents, ) + # `instruction` is a vLLM-supported passthrough (folded into the model's + # chat_template_kwargs). Only forward it when explicitly set so omitting + # it leaves the request unchanged. + if instruction is not None: + mapped_params["instruction"] = instruction + + return dict(mapped_params) + def validate_environment( self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> dict: if api_key is None: api_key = get_secret_str("HOSTED_VLLM_API_KEY") or "fake-api-key" @@ -121,7 +129,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: if "query" not in optional_rerank_params: raise ValueError("query is required for Hosted VLLM rerank") @@ -135,6 +143,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig): top_n=optional_rerank_params.get("top_n", None), rank_fields=optional_rerank_params.get("rank_fields", None), return_documents=optional_rerank_params.get("return_documents", None), + instruction=optional_rerank_params.get("instruction", None), ) return rerank_request.model_dump(exclude_none=True) @@ -144,7 +153,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, @@ -178,7 +187,7 @@ class HostedVLLMRerankConfig(BaseRerankConfig): rerank_meta = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens) # Extract results - _results: Optional[List[dict]] = response.get("results") + _results: List[dict] | None = response.get("results") if _results is None: raise ValueError(f"No results found in the response={response}") diff --git a/litellm/llms/huggingface/rerank/transformation.py b/litellm/llms/huggingface/rerank/transformation.py index 2c847b617ef..c94fc65acbd 100644 --- a/litellm/llms/huggingface/rerank/transformation.py +++ b/litellm/llms/huggingface/rerank/transformation.py @@ -1,5 +1,5 @@ import os -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Union +from typing import TYPE_CHECKING, Any, Dict, List, Tuple, Union import httpx from typing_extensions import TypedDict @@ -35,7 +35,7 @@ class HuggingFaceRerankResponseItem(TypedDict): index: int score: float - text: Optional[str] # Optional, included when return_text=True + text: str | None # Optional, included when return_text=True class HuggingFaceRerankResponse(TypedDict): @@ -50,7 +50,7 @@ HuggingFaceRerankResponseList = List[HuggingFaceRerankResponseItem] class HuggingFaceRerankConfig(BaseRerankConfig): - def get_api_base(self, model: str, api_base: Optional[str]) -> str: + def get_api_base(self, model: str, api_base: str | None) -> str: if api_base is not None: return api_base elif os.getenv("HF_API_BASE") is not None: @@ -62,9 +62,9 @@ class HuggingFaceRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: """ Get the complete URL for the API call, including the /rerank suffix if necessary. @@ -89,17 +89,18 @@ class HuggingFaceRerankConfig(BaseRerankConfig): def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: optional_rerank_params = {} if non_default_params is not None: @@ -121,9 +122,9 @@ class HuggingFaceRerankConfig(BaseRerankConfig): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, - api_base: Optional[str] = None, + api_key: str | None = None, + optional_params: dict | None = None, + api_base: str | None = None, ) -> dict: # Get API credentials api_key, api_base = self.get_api_credentials(api_key=api_key, api_base=api_base) @@ -146,7 +147,7 @@ class HuggingFaceRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Union[OptionalRerankParams, dict], headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: if "query" not in optional_rerank_params: raise ValueError("query is required for HuggingFace rerank") @@ -172,7 +173,7 @@ class HuggingFaceRerankConfig(BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LoggingClass, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, @@ -275,9 +276,9 @@ class HuggingFaceRerankConfig(BaseRerankConfig): def get_api_credentials( self, - api_key: Optional[str] = None, - api_base: Optional[str] = None, - ) -> Tuple[Optional[str], Optional[str]]: + api_key: str | None = None, + api_base: str | None = None, + ) -> Tuple[str | None, str | None]: """ Get API key and base URL from multiple sources. Returns tuple of (api_key, api_base). diff --git a/litellm/llms/jina_ai/rerank/transformation.py b/litellm/llms/jina_ai/rerank/transformation.py index 56be754fc34..cabaf079edc 100644 --- a/litellm/llms/jina_ai/rerank/transformation.py +++ b/litellm/llms/jina_ai/rerank/transformation.py @@ -6,7 +6,7 @@ Why separate file? Make it easy to see how transformation works Docs - https://jina.ai/reranker """ -from typing import Any, Dict, List, Optional, Tuple, Union +from typing import Any, Dict, List, Tuple, Union from httpx import URL, Response @@ -39,12 +39,13 @@ class JinaAIRerankConfig(BaseRerankConfig): drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: optional_params = {} supported_params = self.get_supported_cohere_rerank_params(model) @@ -59,9 +60,9 @@ class JinaAIRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: base_path = "/v1/rerank" @@ -78,7 +79,7 @@ class JinaAIRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: Dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> Dict: return {"model": model, **optional_rerank_params} @@ -88,7 +89,7 @@ class JinaAIRerankConfig(BaseRerankConfig): raw_response: Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: Dict = {}, optional_params: Dict = {}, litellm_params: Dict = {}, @@ -104,7 +105,7 @@ class JinaAIRerankConfig(BaseRerankConfig): _tokens = RerankTokens(**_json_response.get("usage", {})) rerank_meta = RerankResponseMeta(billed_units=_billed_units, tokens=_tokens) - _results: Optional[List[dict]] = _json_response.get("results") + _results: List[dict] | None = _json_response.get("results") if _results is None: raise ValueError(f"No results found in the response={_json_response}") @@ -136,8 +137,8 @@ class JinaAIRerankConfig(BaseRerankConfig): self, headers: Dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> Dict: if api_key is None: raise ValueError( @@ -152,9 +153,9 @@ class JinaAIRerankConfig(BaseRerankConfig): def calculate_rerank_cost( self, model: str, - custom_llm_provider: Optional[str] = None, - billed_units: Optional[RerankBilledUnits] = None, - model_info: Optional[ModelInfo] = None, + custom_llm_provider: str | None = None, + billed_units: RerankBilledUnits | None = None, + model_info: ModelInfo | None = None, ) -> Tuple[float, float]: """ Jina AI reranker is priced at $0.000000018 per token. diff --git a/litellm/llms/moonshot/chat/transformation.py b/litellm/llms/moonshot/chat/transformation.py index 587fa0ed8d6..a5ac696aae2 100644 --- a/litellm/llms/moonshot/chat/transformation.py +++ b/litellm/llms/moonshot/chat/transformation.py @@ -242,11 +242,11 @@ class MoonshotChatConfig(OpenAIGPTConfig): https://platform.moonshot.ai/docs/guide/migrating-from-openai-to-kimi#about-tool_choice """ - messages.append( + optional_params.pop("tool_choice") + return [ + *messages, { "role": "user", "content": "Please select a tool to handle the current issue.", # Usually, the Kimi large language model understands the intention to invoke a tool and selects one for invocation - } - ) - optional_params.pop("tool_choice") - return messages + }, + ] diff --git a/litellm/llms/nvidia_nim/rerank/transformation.py b/litellm/llms/nvidia_nim/rerank/transformation.py index 05f545c2944..7ae3f913297 100644 --- a/litellm/llms/nvidia_nim/rerank/transformation.py +++ b/litellm/llms/nvidia_nim/rerank/transformation.py @@ -1,4 +1,4 @@ -from typing import Any, Dict, List, Literal, Optional, Union +from typing import Any, Dict, List, Literal, Union import httpx from typing_extensions import Required, TypedDict @@ -64,9 +64,9 @@ class NvidiaNimRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: """ Construct the Nvidia NIM rerank URL. @@ -106,17 +106,18 @@ class NvidiaNimRerankConfig(BaseRerankConfig): def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: """ Map Cohere/OpenAI rerank params to Nvidia NIM format. @@ -145,8 +146,8 @@ class NvidiaNimRerankConfig(BaseRerankConfig): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> dict: """ Validate that the Nvidia NIM API key is present. @@ -177,7 +178,7 @@ class NvidiaNimRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: """ Transform request to Nvidia NIM format. @@ -258,7 +259,7 @@ class NvidiaNimRerankConfig(BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, diff --git a/litellm/llms/openai/image_generation/cost_calculator.py b/litellm/llms/openai/image_generation/cost_calculator.py index d009a085fab..dab277a7ba8 100644 --- a/litellm/llms/openai/image_generation/cost_calculator.py +++ b/litellm/llms/openai/image_generation/cost_calculator.py @@ -7,7 +7,10 @@ These models use token-based pricing instead of pixel-based pricing like DALL-E. from typing import Optional from litellm import verbose_logger -from litellm.litellm_core_utils.llm_cost_calc.utils import generic_cost_per_token +from litellm.litellm_core_utils.llm_cost_calc.utils import ( + calculate_image_response_cost_from_usage, + generic_cost_per_token, +) from litellm.types.utils import ImageResponse, Usage @@ -16,54 +19,40 @@ def cost_calculator( image_response: ImageResponse, custom_llm_provider: Optional[str] = None, ) -> float: - """ - Calculate cost for OpenAI gpt-image models. - - Uses the same usage format as Responses API, so we reuse the helper - to transform to chat completion format and use generic_cost_per_token. - - Args: - model: The model name (e.g., "gpt-image-1", "gpt-image-2") - image_response: The ImageResponse containing usage data - custom_llm_provider: Optional provider name - - Returns: - float: Total cost in USD - """ + """Calculate cost for OpenAI gpt-image models (token-based pricing).""" usage = getattr(image_response, "usage", None) - if usage is None: verbose_logger.debug( f"No usage data available for {model}, cannot calculate token-based cost" ) return 0.0 - # If usage is already a Usage object with completion_tokens_details set, - # use it directly (it was already transformed in convert_to_image_response) + provider = custom_llm_provider or "openai" + + # A chat Usage with an explicit output breakdown: cost via generic_cost_per_token. if isinstance(usage, Usage) and usage.completion_tokens_details is not None: - chat_usage = usage - else: - # Transform ImageUsage to Usage using the existing helper - # ImageUsage has the same format as ResponseAPIUsage - from litellm.responses.utils import ResponseAPILoggingUtils - - chat_usage = ( - ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage(usage) + prompt_cost, completion_cost = generic_cost_per_token( + model=model, usage=usage, custom_llm_provider=provider ) + return prompt_cost + completion_cost - # Use generic_cost_per_token for cost calculation - prompt_cost, completion_cost = generic_cost_per_token( - model=model, - usage=chat_usage, - custom_llm_provider=custom_llm_provider or "openai", - ) + # ImageUsage / ResponseAPIUsage: reuse the shared helper (same path as + # azure_ai/gemini/vertex_ai). It prices generated output tokens at + # output_cost_per_image_token, classifying them as image tokens when the provider + # does not itemize output and splitting text/image when it does. + if getattr(usage, "input_tokens", None) is not None: + token_based_cost = calculate_image_response_cost_from_usage( + model=model, image_response=image_response, custom_llm_provider=provider + ) + if token_based_cost is not None: + return token_based_cost - total_cost = prompt_cost + completion_cost + # Fallback: a Usage with no output breakdown that the image helper can't read — + # cost via generic_cost_per_token (text rate) instead of returning 0.0. + if isinstance(usage, Usage): + prompt_cost, completion_cost = generic_cost_per_token( + model=model, usage=usage, custom_llm_provider=provider + ) + return prompt_cost + completion_cost - verbose_logger.debug( - f"OpenAI gpt-image cost calculation for {model}: " - f"prompt_cost=${prompt_cost:.6f}, completion_cost=${completion_cost:.6f}, " - f"total=${total_cost:.6f}" - ) - - return total_cost + return 0.0 diff --git a/litellm/llms/openai/openai.py b/litellm/llms/openai/openai.py index 59acbac6e15..0b90381ba59 100644 --- a/litellm/llms/openai/openai.py +++ b/litellm/llms/openai/openai.py @@ -785,7 +785,11 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM): ) logging_obj.model_call_details["response_headers"] = headers - stringified_response = response.model_dump() + stringified_response = ( + provider_config.transform_parsed_response_dict( + response.model_dump() + ) + ) logging_obj.post_call( input=messages, api_key=api_key, @@ -933,7 +937,9 @@ class OpenAIChatCompletion(BaseLLM, BaseOpenAILLM): timeout=timeout, logging_obj=logging_obj, ) - stringified_response = response.model_dump() + stringified_response = provider_config.transform_parsed_response_dict( + response.model_dump() + ) logging_obj.post_call( input=data["messages"], api_key=api_key, diff --git a/litellm/llms/vertex_ai/rerank/transformation.py b/litellm/llms/vertex_ai/rerank/transformation.py index 3b84972e946..382a8498d40 100644 --- a/litellm/llms/vertex_ai/rerank/transformation.py +++ b/litellm/llms/vertex_ai/rerank/transformation.py @@ -4,7 +4,7 @@ Translates from Cohere's `/v1/rerank` input format to Vertex AI Discovery Engine Why separate file? Make it easy to see how transformation works """ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union import httpx @@ -36,9 +36,9 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[Dict] = None, + optional_params: Dict | None = None, ) -> str: """ Get the complete URL for the Vertex AI Discovery Engine ranking API @@ -76,8 +76,8 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[Dict] = None, + api_key: str | None = None, + optional_params: Dict | None = None, ) -> dict: """ Validate and set up authentication for Vertex AI Discovery Engine API @@ -112,7 +112,7 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: """ Transform the request from Cohere format to Vertex AI Discovery Engine format @@ -161,7 +161,7 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, @@ -236,12 +236,13 @@ class VertexAIRerankConfig(BaseRerankConfig, VertexBase): drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: """ Map Cohere rerank params to Vertex AI format diff --git a/litellm/llms/voyage/rerank/transformation.py b/litellm/llms/voyage/rerank/transformation.py index d64450a1211..05991d4cc8c 100644 --- a/litellm/llms/voyage/rerank/transformation.py +++ b/litellm/llms/voyage/rerank/transformation.py @@ -4,7 +4,7 @@ Transformation logic for Voyage AI's /v1/rerank endpoint. Docs - https://docs.voyageai.com/docs/reranker """ -from typing import Any, Dict, List, Optional, Tuple, Union +from typing import Any, Dict, List, Tuple, Union import httpx @@ -33,12 +33,13 @@ class VoyageRerankConfig(BaseRerankConfig): drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: # Voyage AI uses 'top_k' instead of 'top_n' optional_params: Dict[str, Any] = {"query": query, "documents": documents} @@ -52,9 +53,9 @@ class VoyageRerankConfig(BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: if api_base is None: return "https://api.voyageai.com/v1/rerank" @@ -71,7 +72,7 @@ class VoyageRerankConfig(BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: Dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> Dict: return {"model": model, **optional_rerank_params} @@ -81,7 +82,7 @@ class VoyageRerankConfig(BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: Dict = {}, optional_params: Dict = {}, litellm_params: Dict = {}, @@ -102,7 +103,7 @@ class VoyageRerankConfig(BaseRerankConfig): ) # Voyage AI returns results in "data" key, not "results" - _results: Optional[List[dict]] = _json_response.get("data") + _results: List[dict] | None = _json_response.get("data") if _results is None: raise ValueError(f"No results found in the response={_json_response}") @@ -136,8 +137,8 @@ class VoyageRerankConfig(BaseRerankConfig): self, headers: Dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> Dict: if api_key is None: api_key = get_secret_str("VOYAGE_API_KEY") or get_secret_str( @@ -155,9 +156,9 @@ class VoyageRerankConfig(BaseRerankConfig): def calculate_rerank_cost( self, model: str, - custom_llm_provider: Optional[str] = None, - billed_units: Optional[RerankBilledUnits] = None, - model_info: Optional[ModelInfo] = None, + custom_llm_provider: str | None = None, + billed_units: RerankBilledUnits | None = None, + model_info: ModelInfo | None = None, ) -> Tuple[float, float]: if ( model_info is None diff --git a/litellm/llms/watsonx/rerank/transformation.py b/litellm/llms/watsonx/rerank/transformation.py index 202760f68a6..790606c7e6d 100644 --- a/litellm/llms/watsonx/rerank/transformation.py +++ b/litellm/llms/watsonx/rerank/transformation.py @@ -5,7 +5,7 @@ Docs - https://cloud.ibm.com/apidocs/watsonx-ai#text-rerank """ import uuid -from typing import Any, Dict, List, Optional, Union, cast +from typing import Any, Dict, List, Union, cast import httpx @@ -31,9 +31,9 @@ class IBMWatsonXRerankConfig(IBMWatsonXMixin, BaseRerankConfig): def get_complete_url( self, - api_base: Optional[str], + api_base: str | None, model: str, - optional_params: Optional[dict] = None, + optional_params: dict | None = None, ) -> str: base_url = self._get_base_url(api_base=api_base) endpoint = WatsonXAIEndpoint.RERANK.value @@ -60,8 +60,8 @@ class IBMWatsonXRerankConfig(IBMWatsonXMixin, BaseRerankConfig): self, headers: dict, model: str, - api_key: Optional[str] = None, - optional_params: Optional[dict] = None, + api_key: str | None = None, + optional_params: dict | None = None, ) -> Dict: optional_params = optional_params or {} @@ -73,11 +73,11 @@ class IBMWatsonXRerankConfig(IBMWatsonXMixin, BaseRerankConfig): if "Authorization" in headers: return {**default_headers, **headers} token = cast( - Optional[str], + str | None, optional_params.pop("token", None) or get_secret_str("WATSONX_TOKEN"), ) zen_api_key = cast( - Optional[str], + str | None, optional_params.pop("zen_api_key", None) or get_secret_str("WATSONX_ZENAPIKEY"), ) @@ -93,17 +93,18 @@ class IBMWatsonXRerankConfig(IBMWatsonXMixin, BaseRerankConfig): def map_cohere_rerank_params( self, - non_default_params: Optional[dict], + non_default_params: dict | None, model: str, drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, ) -> Dict: """ Map Cohere rerank params to IBM watsonx.ai rerank params @@ -143,7 +144,7 @@ class IBMWatsonXRerankConfig(IBMWatsonXMixin, BaseRerankConfig): model: str, optional_rerank_params: Dict, headers: dict, - litellm_params: Optional[dict] = None, + litellm_params: dict | None = None, ) -> dict: """ Transform request to IBM watsonx.ai rerank format @@ -162,7 +163,7 @@ class IBMWatsonXRerankConfig(IBMWatsonXMixin, BaseRerankConfig): raw_response: httpx.Response, model_response: RerankResponse, logging_obj: LiteLLMLoggingObj, - api_key: Optional[str] = None, + api_key: str | None = None, request_data: dict = {}, optional_params: dict = {}, litellm_params: dict = {}, @@ -179,7 +180,7 @@ class IBMWatsonXRerankConfig(IBMWatsonXMixin, BaseRerankConfig): headers=raw_response.headers, ) - _results: Optional[List[dict]] = raw_response_json.get("results") + _results: List[dict] | None = raw_response_json.get("results") if _results is None: raise ValueError(f"No results found in the response={raw_response_json}") diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json index d44fc654a56..1fa6cb1ec71 100644 --- a/litellm/model_prices_and_context_window_backup.json +++ b/litellm/model_prices_and_context_window_backup.json @@ -583,6 +583,15 @@ "output_cost_per_token": 0.0, "output_vector_size": 1536 }, + "amazon.titan-embed-g1-text-02": { + "input_cost_per_token": 1e-07, + "litellm_provider": "bedrock", + "max_input_tokens": 8192, + "max_tokens": 8192, + "mode": "embedding", + "output_cost_per_token": 0.0, + "output_vector_size": 1536 + }, "amazon.titan-embed-text-v2:0": { "input_cost_per_token": 2e-08, "litellm_provider": "bedrock", @@ -14156,6 +14165,14 @@ "notes": "APISerpent deep search (/api/search), multi-engine (Google, Bing, Yahoo, DuckDuckGo). Pricing: $0.60/1k searches." } }, + "tinyfish/search": { + "input_cost_per_query": 0.0, + "litellm_provider": "tinyfish", + "mode": "search", + "metadata": { + "notes": "TinyFish Search API" + } + }, "elevenlabs/scribe_v1": { "input_cost_per_second": 6.11e-05, "litellm_provider": "elevenlabs", @@ -31414,13 +31431,13 @@ "output_cost_per_token": 0.0 }, "sambanova/MiniMax-M2.7": { - "input_cost_per_token": 3e-07, + "input_cost_per_token": 6e-07, "litellm_provider": "sambanova", - "max_input_tokens": 204800, + "max_input_tokens": 196608, "max_output_tokens": 131072, "max_tokens": 131072, "mode": "chat", - "output_cost_per_token": 1.2e-06, + "output_cost_per_token": 2.4e-06, "source": "https://cloud.sambanova.ai/plans/pricing", "supports_function_calling": true, "supports_reasoning": true, @@ -31437,6 +31454,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/DeepSeek-R1-Distill-Llama-70B": { + "deprecation_date": "2026-03-20", "input_cost_per_token": 7e-07, "litellm_provider": "sambanova", "max_input_tokens": 131072, @@ -31447,6 +31465,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/DeepSeek-V3-0324": { + "deprecation_date": "2026-04-14", "input_cost_per_token": 3e-06, "litellm_provider": "sambanova", "max_input_tokens": 32768, @@ -31477,6 +31496,7 @@ "supports_vision": true }, "sambanova/Llama-4-Scout-17B-16E-Instruct": { + "deprecation_date": "2025-06-19", "input_cost_per_token": 4e-07, "litellm_provider": "sambanova", "max_input_tokens": 8192, @@ -31493,6 +31513,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-3.1-405B-Instruct": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 5e-06, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31506,6 +31527,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-3.1-8B-Instruct": { + "deprecation_date": "2026-04-14", "input_cost_per_token": 1e-07, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31519,6 +31541,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-3.2-1B-Instruct": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 4e-08, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31529,6 +31552,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/Meta-Llama-3.2-3B-Instruct": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 8e-08, "litellm_provider": "sambanova", "max_input_tokens": 4096, @@ -31552,6 +31576,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-Guard-3-8B": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 3e-07, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31562,6 +31587,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/QwQ-32B": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 5e-07, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31572,6 +31598,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/Qwen2-Audio-7B-Instruct": { + "deprecation_date": "2025-06-19", "input_cost_per_token": 5e-07, "litellm_provider": "sambanova", "max_input_tokens": 4096, @@ -31583,6 +31610,7 @@ "supports_audio_input": true }, "sambanova/Qwen3-32B": { + "deprecation_date": "2026-04-06", "input_cost_per_token": 4e-07, "litellm_provider": "sambanova", "max_input_tokens": 8192, @@ -31596,9 +31624,9 @@ "supports_tool_choice": true }, "sambanova/DeepSeek-V3.1": { - "max_tokens": 32768, - "max_input_tokens": 32768, - "max_output_tokens": 32768, + "max_tokens": 131072, + "max_input_tokens": 131072, + "max_output_tokens": 131072, "input_cost_per_token": 3e-06, "output_cost_per_token": 4.5e-06, "litellm_provider": "sambanova", @@ -31612,8 +31640,8 @@ "max_tokens": 131072, "max_input_tokens": 131072, "max_output_tokens": 131072, - "input_cost_per_token": 3e-06, - "output_cost_per_token": 4.5e-06, + "input_cost_per_token": 2.2e-07, + "output_cost_per_token": 5.9e-07, "litellm_provider": "sambanova", "mode": "chat", "supports_function_calling": true, @@ -31621,21 +31649,55 @@ "supports_reasoning": true, "source": "https://cloud.sambanova.ai/plans/pricing" }, - "snowflake/claude-3-5-sonnet": { - "litellm_provider": "snowflake", - "max_input_tokens": 18000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat", - "supports_computer_use": true - }, - "snowflake/deepseek-r1": { - "litellm_provider": "snowflake", + "sambanova/DeepSeek-V3.2": { + "max_tokens": 32768, "max_input_tokens": 32768, - "max_output_tokens": 8192, - "max_tokens": 8192, + "max_output_tokens": 32768, + "input_cost_per_token": 3e-06, + "output_cost_per_token": 4.5e-06, + "litellm_provider": "sambanova", "mode": "chat", - "supports_reasoning": true + "supports_function_calling": true, + "supports_tool_choice": true, + "source": "https://cloud.sambanova.ai/plans/pricing" + }, + "sambanova/gemma-4-31B-it": { + "max_tokens": 131072, + "max_input_tokens": 131072, + "max_output_tokens": 131072, + "input_cost_per_token": 3.8e-07, + "output_cost_per_token": 1.15e-06, + "litellm_provider": "sambanova", + "mode": "chat", + "supports_vision": true, + "source": "https://cloud.sambanova.ai/plans/pricing" + }, + "snowflake/claude-3-5-sonnet": { + "litellm_provider": "snowflake", + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "input_cost_per_token": 0.000003, + "output_cost_per_token": 0.000015, + "cache_read_input_token_cost": 0.0000003, + "supports_computer_use": true, + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/deepseek-r1": { + "litellm_provider": "snowflake", + "max_input_tokens": 128000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "input_cost_per_token": 0.00000135, + "output_cost_per_token": 0.0000054, + "supports_reasoning": true, + "supports_system_messages": true }, "snowflake/gemma-7b": { "litellm_provider": "snowflake", @@ -31689,23 +31751,34 @@ "snowflake/llama3.1-405b": { "litellm_provider": "snowflake", "max_input_tokens": 128000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat" + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "input_cost_per_token": 0.0000012, + "output_cost_per_token": 0.0000012, + "supports_function_calling": true, + "supports_system_messages": true }, "snowflake/llama3.1-70b": { "litellm_provider": "snowflake", "max_input_tokens": 128000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat" + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "input_cost_per_token": 0.00000072, + "output_cost_per_token": 0.00000072, + "supports_function_calling": true, + "supports_system_messages": true }, "snowflake/llama3.1-8b": { "litellm_provider": "snowflake", "max_input_tokens": 128000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat" + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "input_cost_per_token": 0.00000024, + "output_cost_per_token": 0.00000024, + "supports_system_messages": true }, "snowflake/llama3.2-1b": { "litellm_provider": "snowflake", @@ -31721,13 +31794,17 @@ "max_tokens": 8192, "mode": "chat" }, - "snowflake/llama3.3-70b": { - "litellm_provider": "snowflake", + "snowflake/llama3.3-70b": { + "max_tokens": 16384, "max_input_tokens": 128000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat" - }, + "max_output_tokens": 16384, + "input_cost_per_token": 0.00000072, + "output_cost_per_token": 0.00000072, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_system_messages": true + }, "snowflake/mistral-7b": { "litellm_provider": "snowflake", "max_input_tokens": 32000, @@ -31742,12 +31819,17 @@ "max_tokens": 8192, "mode": "chat" }, - "snowflake/mistral-large2": { + "snowflake/mistral-large2": { "litellm_provider": "snowflake", "max_input_tokens": 128000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat" + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "input_cost_per_token": 0.000002, + "output_cost_per_token": 0.000006, + "supports_function_calling": true, + "supports_system_messages": true, + "supports_response_schema": true }, "snowflake/mixtral-8x7b": { "litellm_provider": "snowflake", @@ -31784,13 +31866,17 @@ "max_tokens": 8192, "mode": "chat" }, - "snowflake/snowflake-llama-3.3-70b": { + "snowflake/snowflake-llama-3.3-70b": { + "max_tokens": 16384, + "max_input_tokens": 128000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.00000072, + "output_cost_per_token": 0.00000072, "litellm_provider": "snowflake", - "max_input_tokens": 8000, - "max_output_tokens": 8192, - "max_tokens": 8192, - "mode": "chat" - }, + "mode": "chat", + "supports_function_calling": true, + "supports_system_messages": true + }, "stability/sd3": { "litellm_provider": "stability", "mode": "image_generation", @@ -32133,6 +32219,11 @@ "litellm_provider": "tavily", "mode": "search" }, + "you_com/search": { + "input_cost_per_query": 0.0, + "litellm_provider": "you_com", + "mode": "search" + }, "text-completion-codestral/codestral-2405": { "input_cost_per_token": 0.0, "litellm_provider": "text-completion-codestral", @@ -36652,17 +36743,7 @@ "max_input_tokens": 32000, "max_tokens": 32000, "mode": "embedding", - "output_cost_per_token": 0.0, - "supports_vision": true - }, - "voyage/voyage-multimodal-3.5": { - "input_cost_per_token": 1.2e-07, - "litellm_provider": "voyage", - "max_input_tokens": 32000, - "max_tokens": 32000, - "mode": "embedding", - "output_cost_per_token": 0.0, - "supports_vision": true + "output_cost_per_token": 0.0 }, "wandb/openai/gpt-oss-120b": { "max_tokens": 131072, @@ -40327,6 +40408,178 @@ "supports_tool_choice": true, "supports_vision": true }, + "scaleway/qwen/qwen3.5-397b-a17b": { + "input_cost_per_token": 6e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 256000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 3.6e-06, + "supports_function_calling": true, + "supports_reasoning": true, + "supports_vision": true + }, + "scaleway/qwen/qwen3.6-35b-a3b": { + "input_cost_per_token": 2.5e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 256000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 1.5e-06, + "supports_function_calling": true, + "supports_vision": true, + "supports_reasoning": true + }, + "scaleway/qwen/qwen3-235b-a22b-instruct-2507": { + "input_cost_per_token": 7.5e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 256000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 2.25e-06, + "supports_function_calling": true + }, + "scaleway/qwen/qwen3-embedding-8b": { + "input_cost_per_token": 1e-07, + "litellm_provider": "scaleway", + "mode": "embedding", + "output_cost_per_token": 0.0 + }, + "scaleway/qwen/qwen3-coder-30b-a3b-instruct": { + "input_cost_per_token": 2e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 128000, + "max_output_tokens": 32768, + "max_tokens": 32768, + "mode": "chat", + "output_cost_per_token": 8e-07, + "supports_function_calling": true + }, + "scaleway/openai/gpt-oss-120b": { + "input_cost_per_token": 1.5e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 128000, + "max_output_tokens": 32768, + "max_tokens": 32768, + "mode": "chat", + "output_cost_per_token": 6e-07, + "supports_function_calling": true + }, + "scaleway/openai/whisper-large-v3": { + "input_cost_per_audio_token": 0.0, + "litellm_provider": "scaleway", + "mode": "audio_transcription", + "output_cost_per_token": 0.0 + }, + "scaleway/google/gemma-4-26b-a4b-it": { + "input_cost_per_token": 2.5e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 256000, + "max_output_tokens": 32768, + "max_tokens": 32768, + "mode": "chat", + "output_cost_per_token": 5e-07, + "supports_function_calling": true, + "supports_reasoning": true, + "supports_vision": true + }, + "scaleway/google/gemma-3-27b-it": { + "input_cost_per_token": 2.5e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 40000, + "max_output_tokens": 8192, + "max_tokens": 8192, + "mode": "chat", + "output_cost_per_token": 5e-07, + "supports_function_calling": true, + "supports_vision": true + }, + "scaleway/hcompany/holo2-30b-a3b": { + "input_cost_per_token": 3e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 22000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 7e-07, + "supports_reasoning": true, + "supports_vision": true + }, + "scaleway/mistralai/mistral-medium-3.5-128b": { + "input_cost_per_token": 1.5e-06, + "litellm_provider": "scaleway", + "max_input_tokens": 256000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 7.5e-06, + "supports_reasoning": true, + "supports_function_calling": true, + "supports_vision": true, + "supports_tool_choice": true + }, + "scaleway/mistralai/devstral-2-123b-instruct-2512": { + "input_cost_per_token": 4e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 2e-06, + "supports_function_calling": true + }, + "scaleway/mistralai/voxtral-small-24b-2507": { + "input_cost_per_audio_token": 1.5e-07, + "input_cost_per_token": 1.5e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 32000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 3.5e-07, + "supports_audio_input": true + }, + "scaleway/mistralai/mistral-small-3.2-24b-instruct-2506": { + "input_cost_per_token": 1.5e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 128000, + "max_output_tokens": 32768, + "max_tokens": 32768, + "mode": "chat", + "output_cost_per_token": 3.5e-07, + "supports_function_calling": true, + "supports_vision": true + }, + "scaleway/mistralai/pixtral-12b-2409": { + "input_cost_per_token": 2e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 128000, + "max_output_tokens": 4096, + "max_tokens": 4096, + "mode": "chat", + "output_cost_per_token": 2e-07, + "supports_vision": true, + "supports_function_calling": true + }, + "scaleway/BAAI/bge-multilingual-gemma2": { + "input_cost_per_token": 1e-07, + "litellm_provider": "scaleway", + "mode": "embedding", + "output_cost_per_token": 0.0 + }, + "scaleway/meta/llama-3.3-70b-instruct": { + "input_cost_per_token": 9e-07, + "litellm_provider": "scaleway", + "max_input_tokens": 128000, + "max_output_tokens": 16384, + "max_tokens": 16384, + "mode": "chat", + "output_cost_per_token": 9e-07, + "supports_function_calling": true + }, "novita/deepseek/deepseek-v3.2": { "litellm_provider": "novita", "mode": "chat", @@ -43108,192 +43361,362 @@ "supports_native_structured_output": true, "supports_pdf_input": true }, - "soniox/stt-async-v4": { - "litellm_provider": "soniox", - "max_output_tokens": 8000, - "max_tokens": 8000, - "input_cost_per_second": 0.0, - "output_cost_per_second": 0.0000277778, - "mode": "audio_transcription", - "source": "https://soniox.com/pricing", - "supported_endpoints": [ - "/v1/audio/transcriptions" - ], - "supports_audio_input": true - }, - "soniox/stt-async-v5": { - "litellm_provider": "soniox", - "max_output_tokens": 8000, - "max_tokens": 8000, - "input_cost_per_second": 0.0, - "output_cost_per_second": 0.0000277778, - "mode": "audio_transcription", - "source": "https://soniox.com/pricing", - "supported_endpoints": [ - "/v1/audio/transcriptions" - ], - "supports_audio_input": true - }, - "tensormesh/Qwen/Qwen3.5-397B-A17B-FP8": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 6e-07, - "output_cost_per_token": 3.6e-06, - "cache_read_input_token_cost": 0, - "max_input_tokens": 262144, - "max_output_tokens": 262144, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 4.5e-07, - "output_cost_per_token": 1.8e-06, - "cache_read_input_token_cost": 0, - "max_input_tokens": 262144, - "max_output_tokens": 262144, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/Qwen/Qwen3.6-27B-FP8": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 3.2e-07, - "output_cost_per_token": 3.2e-06, - "cache_read_input_token_cost": 0, - "max_input_tokens": 262144, - "max_output_tokens": 262144, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/lukealonso/GLM-5.1-NVFP4-MTP": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 1.4e-06, - "output_cost_per_token": 4.4e-06, - "cache_read_input_token_cost": 0, - "max_input_tokens": 202752, - "max_output_tokens": 202752, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/deepseek-ai/DeepSeek-V4-Flash": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 1.4e-07, - "output_cost_per_token": 2.8e-07, - "cache_read_input_token_cost": 0, - "max_input_tokens": 32768, - "max_output_tokens": 32768, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/moonshotai/Kimi-K2.6": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 9.6e-07, - "output_cost_per_token": 4e-06, - "cache_read_input_token_cost": 0, - "max_input_tokens": 32768, - "max_output_tokens": 32768, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/MiniMaxAI/MiniMax-M2.5": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 3e-07, - "output_cost_per_token": 1.2e-06, - "cache_read_input_token_cost": 0, - "max_input_tokens": 196608, - "max_output_tokens": 196608, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/google/gemma-4-31B-it": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 1.4e-07, - "output_cost_per_token": 5.6e-07, - "cache_read_input_token_cost": 0, - "max_input_tokens": 32768, - "max_output_tokens": 32768, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/openai/gpt-oss-120b": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 1.5e-07, - "output_cost_per_token": 6e-07, - "cache_read_input_token_cost": 0, - "max_input_tokens": 131072, - "max_output_tokens": 131072, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - }, - "tensormesh/openai/gpt-oss-20b": { - "litellm_provider": "tensormesh", - "mode": "chat", - "input_cost_per_token": 7e-08, - "output_cost_per_token": 2.8e-07, - "cache_read_input_token_cost": 0, - "max_input_tokens": 131072, - "max_output_tokens": 131072, - "supports_function_calling": true, - "supports_tool_choice": true, - "supports_response_schema": true, - "supports_prompt_caching": true, - "supports_system_messages": true, - "supports_reasoning": true, - "source": "https://serverless.tensormesh.ai/v1/models/openrouter" - } -, + "snowflake/claude-sonnet-4-5": { + "max_tokens": 16384, + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.000003, + "output_cost_per_token": 0.000015, + "cache_read_input_token_cost": 0.0000003, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/claude-sonnet-4-6": { + "max_tokens": 16384, + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.000003, + "output_cost_per_token": 0.000015, + "cache_read_input_token_cost": 0.0000003, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/claude-4-sonnet": { + "max_tokens": 16384, + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.000003, + "output_cost_per_token": 0.000015, + "cache_read_input_token_cost": 0.0000003, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/claude-4-opus": { + "max_tokens": 16384, + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.000005, + "output_cost_per_token": 0.000025, + "cache_read_input_token_cost": 0.0000005, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "supports_response_schema": true + }, + "snowflake/claude-haiku-4-5": { + "max_tokens": 16384, + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.000001, + "output_cost_per_token": 0.000005, + "cache_read_input_token_cost": 0.0000001, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/claude-3-7-sonnet": { + "max_tokens": 16384, + "max_input_tokens": 200000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.000003, + "output_cost_per_token": 0.000015, + "cache_read_input_token_cost": 0.0000003, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "supports_response_schema": true + }, + "snowflake/openai-gpt-4.1": { + "max_tokens": 16384, + "max_input_tokens": 300000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.000002, + "output_cost_per_token": 0.000008, + "cache_read_input_token_cost": 0.0000005, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/openai-gpt-5": { + "max_tokens": 16384, + "max_input_tokens": 300000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.00000125, + "output_cost_per_token": 0.00001, + "cache_read_input_token_cost": 0.000000125, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_vision": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "supports_response_schema": true + }, + "snowflake/openai-gpt-5-mini": { + "max_tokens": 16384, + "max_input_tokens": 1000000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.0000003, + "output_cost_per_token": 0.0000012, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/openai-gpt-5-nano": { + "max_tokens": 16384, + "max_input_tokens": 5000000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.00000015, + "output_cost_per_token": 0.0000006, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_system_messages": true, + "supports_response_schema": true + }, + "snowflake/llama4-maverick": { + "max_tokens": 16384, + "max_input_tokens": 128000, + "max_output_tokens": 16384, + "input_cost_per_token": 0.00000024, + "output_cost_per_token": 0.00000097, + "litellm_provider": "snowflake", + "mode": "chat", + "supports_function_calling": true, + "supports_system_messages": true + }, + "snowflake/snowflake-arctic-embed-l-v2.0": { + "max_tokens": 8192, + "max_input_tokens": 8192, + "input_cost_per_token": 0.00000007, + "output_cost_per_token": 0.0, + "litellm_provider": "snowflake", + "mode": "embedding" + }, + "snowflake/snowflake-arctic-embed-m-v2.0": { + "max_tokens": 8192, + "max_input_tokens": 8192, + "input_cost_per_token": 0.00000007, + "output_cost_per_token": 0.0, + "litellm_provider": "snowflake", + "mode": "embedding" + }, + "soniox/stt-async-v4": { + "litellm_provider": "soniox", + "max_output_tokens": 8000, + "max_tokens": 8000, + "input_cost_per_second": 0.0, + "output_cost_per_second": 0.0000277778, + "mode": "audio_transcription", + "source": "https://soniox.com/pricing", + "supported_endpoints": ["/v1/audio/transcriptions"], + "supports_audio_input": true + }, + "soniox/stt-async-v5": { + "litellm_provider": "soniox", + "max_output_tokens": 8000, + "max_tokens": 8000, + "input_cost_per_second": 0.0, + "output_cost_per_second": 0.0000277778, + "mode": "audio_transcription", + "source": "https://soniox.com/pricing", + "supported_endpoints": ["/v1/audio/transcriptions"], + "supports_audio_input": true + }, + "tensormesh/Qwen/Qwen3.5-397B-A17B-FP8": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 6e-07, + "output_cost_per_token": 3.6e-06, + "cache_read_input_token_cost": 0, + "max_input_tokens": 262144, + "max_output_tokens": 262144, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 4.5e-07, + "output_cost_per_token": 1.8e-06, + "cache_read_input_token_cost": 0, + "max_input_tokens": 262144, + "max_output_tokens": 262144, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/Qwen/Qwen3.6-27B-FP8": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 3.2e-07, + "output_cost_per_token": 3.2e-06, + "cache_read_input_token_cost": 0, + "max_input_tokens": 262144, + "max_output_tokens": 262144, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/lukealonso/GLM-5.1-NVFP4-MTP": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 1.4e-06, + "output_cost_per_token": 4.4e-06, + "cache_read_input_token_cost": 0, + "max_input_tokens": 202752, + "max_output_tokens": 202752, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/deepseek-ai/DeepSeek-V4-Flash": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 1.4e-07, + "output_cost_per_token": 2.8e-07, + "cache_read_input_token_cost": 0, + "max_input_tokens": 32768, + "max_output_tokens": 32768, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/moonshotai/Kimi-K2.6": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 9.6e-07, + "output_cost_per_token": 4e-06, + "cache_read_input_token_cost": 0, + "max_input_tokens": 32768, + "max_output_tokens": 32768, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/MiniMaxAI/MiniMax-M2.5": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 3e-07, + "output_cost_per_token": 1.2e-06, + "cache_read_input_token_cost": 0, + "max_input_tokens": 196608, + "max_output_tokens": 196608, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/google/gemma-4-31B-it": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 1.4e-07, + "output_cost_per_token": 5.6e-07, + "cache_read_input_token_cost": 0, + "max_input_tokens": 32768, + "max_output_tokens": 32768, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/openai/gpt-oss-120b": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 1.5e-07, + "output_cost_per_token": 6e-07, + "cache_read_input_token_cost": 0, + "max_input_tokens": 131072, + "max_output_tokens": 131072, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + }, + "tensormesh/openai/gpt-oss-20b": { + "litellm_provider": "tensormesh", + "mode": "chat", + "input_cost_per_token": 7e-08, + "output_cost_per_token": 2.8e-07, + "cache_read_input_token_cost": 0, + "max_input_tokens": 131072, + "max_output_tokens": 131072, + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_response_schema": true, + "supports_prompt_caching": true, + "supports_system_messages": true, + "supports_reasoning": true, + "source": "https://serverless.tensormesh.ai/v1/models/openrouter" + } + , "deepseek-v4-flash": { "cache_creation_input_token_cost": 0.0, "cache_read_input_token_cost": 2.8e-09, @@ -43427,5 +43850,83 @@ "supports_system_messages": true, "supports_tool_choice": true, "supports_vision": false + }, + "pinstripes/ps/glm-4.5-air": { + "max_tokens": 128000, + "max_input_tokens": 128000, + "max_output_tokens": 128000, + "input_cost_per_token": 0.000000125, + "output_cost_per_token": 0.00000045, + "litellm_provider": "pinstripes", + "mode": "chat", + "supports_function_calling": true, + "supports_assistant_prefill": true, + "supports_reasoning": true, + "source": "https://pinstripes.io/pricing" + }, + "pinstripes/ps/qwen3.6-35b-a3b": { + "max_tokens": 131072, + "max_input_tokens": 131072, + "max_output_tokens": 131072, + "input_cost_per_token": 0.00000014, + "output_cost_per_token": 0.00000045, + "litellm_provider": "pinstripes", + "mode": "chat", + "supports_function_calling": true, + "supports_assistant_prefill": true, + "supports_reasoning": true, + "source": "https://pinstripes.io/pricing" + }, + "pinstripes/ps/qwen3-30b-a3b": { + "max_tokens": 131072, + "max_input_tokens": 131072, + "max_output_tokens": 131072, + "input_cost_per_token": 0.00000009, + "output_cost_per_token": 0.0000002, + "litellm_provider": "pinstripes", + "mode": "chat", + "supports_function_calling": true, + "supports_assistant_prefill": true, + "supports_reasoning": true, + "source": "https://pinstripes.io/pricing" + }, + "pinstripes/ps/qwen3-coder-30b-a3b": { + "max_tokens": 131072, + "max_input_tokens": 131072, + "max_output_tokens": 131072, + "input_cost_per_token": 0.0000003, + "output_cost_per_token": 0.0000006, + "litellm_provider": "pinstripes", + "mode": "chat", + "supports_function_calling": true, + "supports_assistant_prefill": true, + "supports_reasoning": false, + "source": "https://pinstripes.io/pricing" + }, + "pinstripes/ps/deepseek-v4-flash": { + "max_tokens": 163840, + "max_input_tokens": 163840, + "max_output_tokens": 163840, + "input_cost_per_token": 0.0000001, + "output_cost_per_token": 0.0000002, + "litellm_provider": "pinstripes", + "mode": "chat", + "supports_function_calling": true, + "supports_assistant_prefill": true, + "supports_reasoning": true, + "source": "https://pinstripes.io/pricing" + }, + "pinstripes/ps/minimax-m2.7": { + "max_tokens": 1000192, + "max_input_tokens": 1000192, + "max_output_tokens": 1000192, + "input_cost_per_token": 0.000000255, + "output_cost_per_token": 0.00000055, + "litellm_provider": "pinstripes", + "mode": "chat", + "supports_function_calling": true, + "supports_assistant_prefill": true, + "supports_reasoning": false, + "source": "https://pinstripes.io/pricing" } } diff --git a/litellm/proxy/spend_tracking/cold_storage_handler.py b/litellm/proxy/spend_tracking/cold_storage_handler.py index 57c41bafccd..01c698c7122 100644 --- a/litellm/proxy/spend_tracking/cold_storage_handler.py +++ b/litellm/proxy/spend_tracking/cold_storage_handler.py @@ -4,8 +4,6 @@ This module is responsible for handling Getting/Setting the proxy server request It allows fetching a dict of the proxy server request from s3 or GCS bucket. """ -from typing import Optional - import litellm from litellm import _custom_logger_compatible_callbacks_literal from litellm.integrations.custom_logger import CustomLogger @@ -16,12 +14,18 @@ class ColdStorageHandler: This class is responsible for handling Getting/Setting the proxy server request from cold storage. It allows fetching a dict of the proxy server request from s3 or GCS bucket. + + The cold storage logger can be injected for testing; when omitted it is + resolved from the configured ``litellm.cold_storage_custom_logger``. """ + def __init__(self, cold_storage_logger: CustomLogger | None = None): + self._injected_cold_storage_logger = cold_storage_logger + async def get_proxy_server_request_from_cold_storage_with_object_key( self, object_key: str, - ) -> Optional[dict]: + ) -> dict | None: """ Get the proxy server request from cold storage using the object key directly. @@ -31,38 +35,31 @@ class ColdStorageHandler: Returns: Optional[dict]: The proxy server request dict or None if not found """ - - # select the custom logger to use for cold storage - custom_logger_name: Optional[_custom_logger_compatible_callbacks_literal] = ( - self._select_custom_logger_for_cold_storage() + custom_logger = ( + self._injected_cold_storage_logger or self._resolve_cold_storage_logger() ) - - # if no custom logger name is configured, return None - if custom_logger_name is None: + if custom_logger is None: return None - # get the active/initialized custom logger - custom_logger: Optional[CustomLogger] = ( + return await custom_logger.get_proxy_server_request_from_cold_storage_with_object_key( + object_key=object_key, + ) + + def _resolve_cold_storage_logger(self) -> CustomLogger | None: + custom_logger_name = self._select_custom_logger_for_cold_storage() + if custom_logger_name is None: + return None + return ( litellm.logging_callback_manager.get_active_custom_logger_for_callback_name( custom_logger_name ) ) - # if no custom logger is found, return None - if custom_logger is None: - return None - - proxy_server_request = await custom_logger.get_proxy_server_request_from_cold_storage_with_object_key( - object_key=object_key, - ) - - return proxy_server_request - def _select_custom_logger_for_cold_storage( self, - ) -> Optional[_custom_logger_compatible_callbacks_literal]: - cold_storage_custom_logger: Optional[ - _custom_logger_compatible_callbacks_literal - ] = litellm.cold_storage_custom_logger + ) -> _custom_logger_compatible_callbacks_literal | None: + cold_storage_custom_logger: ( + _custom_logger_compatible_callbacks_literal | None + ) = litellm.cold_storage_custom_logger return cold_storage_custom_logger diff --git a/litellm/proxy/spend_tracking/spend_management_endpoints.py b/litellm/proxy/spend_tracking/spend_management_endpoints.py index 29fc6d7c30f..8ca048a0caa 100644 --- a/litellm/proxy/spend_tracking/spend_management_endpoints.py +++ b/litellm/proxy/spend_tracking/spend_management_endpoints.py @@ -3,7 +3,16 @@ import collections import json import os from datetime import datetime, timedelta, timezone -from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional +from typing import ( + TYPE_CHECKING, + Any, + Dict, + List, + Literal, + Mapping, + NamedTuple, + Union, +) import fastapi from fastapi import APIRouter, Depends, HTTPException, Request, status @@ -29,6 +38,7 @@ from litellm.repositories.verification_token_repository import ( if TYPE_CHECKING: from litellm.proxy.proxy_server import PrismaClient + from litellm.proxy.spend_tracking.cold_storage_handler import ColdStorageHandler else: PrismaClient = Any @@ -104,7 +114,7 @@ def _strip_password_from_users(users) -> None: include_in_schema=False, ) async def spend_user_fn( - user_id: Optional[str] = fastapi.Query( + user_id: str | None = fastapi.Query( default=None, description="Get User Table row for user_id", ), @@ -185,11 +195,11 @@ async def spend_user_fn( }, ) async def view_spend_tags( - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing key spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view key spend", ), @@ -290,11 +300,11 @@ async def get_global_activity_internal_user( include_in_schema=False, ) async def get_global_activity( - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view spend", ), @@ -437,11 +447,11 @@ async def get_global_activity_model_internal_user( include_in_schema=False, ) async def get_global_activity_model( - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view spend", ), @@ -597,11 +607,11 @@ async def get_global_activity_exceptions_per_deployment( model_group: str = fastapi.Query( description="Filter by model group", ), - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view spend", ), @@ -750,11 +760,11 @@ async def get_global_activity_exceptions( model_group: str = fastapi.Query( description="Filter by model group", ), - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view spend", ), @@ -857,11 +867,11 @@ async def get_global_activity_exceptions( }, ) async def get_global_spend_provider( - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view spend", ), @@ -992,31 +1002,31 @@ async def get_global_spend_provider( }, ) async def get_global_spend_report( - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view spend", ), - group_by: Optional[Literal["team", "customer", "api_key"]] = fastapi.Query( + group_by: Literal["team", "customer", "api_key"] | None = fastapi.Query( default="team", description="Group spend by internal team or customer or api_key", ), - api_key: Optional[str] = fastapi.Query( + api_key: str | None = fastapi.Query( default=None, description="View spend for a specific api_key. Example api_key='sk-1234", ), - internal_user_id: Optional[str] = fastapi.Query( + internal_user_id: str | None = fastapi.Query( default=None, description="View spend for a specific internal_user_id. Example internal_user_id='1234", ), - team_id: Optional[str] = fastapi.Query( + team_id: str | None = fastapi.Query( default=None, description="View spend for a specific team_id. Example team_id='1234", ), - customer_id: Optional[str] = fastapi.Query( + customer_id: str | None = fastapi.Query( default=None, description="View spend for a specific customer_id. Example customer_id='1234. Can be used in conjunction with team_id as well.", ), @@ -1412,15 +1422,15 @@ async def global_get_all_tag_names(): }, ) async def global_view_spend_tags( - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing key spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view key spend", ), - tags: Optional[str] = fastapi.Query( + tags: str | None = fastapi.Query( default=None, description="comman separated tags to filter on", ), @@ -1639,7 +1649,7 @@ async def calculate_spend(request: SpendCalculateRequest): # check if model in llm_router _model_in_llm_router = None - cost_per_token: Optional[CostPerToken] = None + cost_per_token: CostPerToken | None = None if llm_router is not None: if ( llm_router.model_group_alias is not None @@ -1732,35 +1742,35 @@ async def calculate_spend(request: SpendCalculateRequest): ) async def ui_view_spend_logs( request: Request, - api_key: Optional[str] = fastapi.Query( + api_key: str | None = fastapi.Query( default=None, description="Get spend logs based on api key", ), - user_id: Optional[str] = fastapi.Query( + user_id: str | None = fastapi.Query( default=None, description="Get spend logs based on user_id", ), - request_id: Optional[str] = fastapi.Query( + request_id: str | None = fastapi.Query( default=None, description="request_id to get spend logs for specific request_id", ), - team_id: Optional[str] = fastapi.Query( + team_id: str | None = fastapi.Query( default=None, description="Filter spend logs by team_id", ), - min_spend: Optional[float] = fastapi.Query( + min_spend: float | None = fastapi.Query( default=None, description="Filter logs with spend greater than or equal to this value", ), - max_spend: Optional[float] = fastapi.Query( + max_spend: float | None = fastapi.Query( default=None, description="Filter logs with spend less than or equal to this value", ), - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing key spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view key spend", ), @@ -1771,36 +1781,34 @@ async def ui_view_spend_logs( default=50, description="Number of items per page", ge=1, le=100 ), user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth), - status_filter: Optional[str] = fastapi.Query( + status_filter: str | None = fastapi.Query( default=None, description="Filter logs by status (e.g., success, failure)" ), - model: Optional[str] = fastapi.Query( - default=None, description="Filter logs by model" - ), - model_id: Optional[str] = fastapi.Query( + model: str | None = fastapi.Query(default=None, description="Filter logs by model"), + model_id: str | None = fastapi.Query( default=None, description="Filter logs by model ID (litellm model deployment id)", ), - model_group: Optional[str] = fastapi.Query( + model_group: str | None = fastapi.Query( default=None, description="Filter logs by model group" ), - key_alias: Optional[str] = fastapi.Query( + key_alias: str | None = fastapi.Query( default=None, description="Filter logs by key alias" ), - end_user: Optional[str] = fastapi.Query( + end_user: str | None = fastapi.Query( default=None, description="Filter logs by end user" ), - error_code: Optional[str] = fastapi.Query( + error_code: str | None = fastapi.Query( default=None, description="Filter logs by error code (e.g., '404', '500')" ), - error_message: Optional[str] = fastapi.Query( + error_message: str | None = fastapi.Query( default=None, description="Filter logs by error message (partial string match)" ), sort_by: str = fastapi.Query( default="startTime", description="Sort logs by field: spend, total_tokens, startTime, endTime, request_duration_ms, model, or ttft_ms", ), - sort_order: Optional[str] = fastapi.Query( + sort_order: str | None = fastapi.Query( default="desc", description="Sort order: asc or desc", ), @@ -1964,7 +1972,7 @@ async def ui_view_spend_logs( if max_spend is not None: where_conditions["spend"]["lte"] = max_spend is_admin_view = _is_admin_view_safe(user_api_key_dict=user_api_key_dict) - permitted_team_ids: Optional[List[str]] = None + permitted_team_ids: List[str] | None = None if not is_admin_view: if team_id is not None: can_view_team = await _can_team_member_view_log( @@ -2171,6 +2179,89 @@ async def ui_view_spend_logs( raise handle_exception_on_proxy(e) +class RequestResponsePayload(NamedTuple): + messages: Union[str, list, dict] | None + response: Union[str, list, dict] | None + proxy_server_request: Union[str, dict] | None + + +_EMPTY_SPEND_LOG_VALUES = frozenset({"", "{}", "[]", "null"}) + + +def _spend_log_field_has_content(value: Union[str, list, dict] | None) -> bool: + if value is None: + return False + if isinstance(value, str): + return value.strip() not in _EMPTY_SPEND_LOG_VALUES + if isinstance(value, (list, dict)): + return len(value) > 0 + return True + + +def _cold_storage_object_key_from_metadata( + metadata: Union[str, dict] | None, +) -> str | None: + if isinstance(metadata, str): + try: + metadata = json.loads(metadata) + except (json.JSONDecodeError, TypeError): + return None + if not isinstance(metadata, dict): + return None + object_key = metadata.get("cold_storage_object_key") + return object_key if isinstance(object_key, str) and object_key else None + + +async def _resolve_request_response_payload( + row: Mapping[str, Any], + cold_storage_handler: "ColdStorageHandler", +) -> RequestResponsePayload: + """ + Decide where the prompt/response come from for a single spend-log row. + + PG holds the content when ``store_prompts_in_spend_logs`` is on; otherwise it + holds ``"{}"`` placeholders and the real payload lives in cold storage keyed + by ``metadata.cold_storage_object_key``. The choice is made on actual row + content, not config flags, so historical and mixed-storage rows both resolve + correctly. + """ + messages = row.get("messages") + response = row.get("response") + proxy_server_request = row.get("proxy_server_request") + + pg_payload = RequestResponsePayload(messages, response, proxy_server_request) + if ( + _spend_log_field_has_content(messages) + or _spend_log_field_has_content(response) + or _spend_log_field_has_content(proxy_server_request) + ): + return pg_payload + + object_key = _cold_storage_object_key_from_metadata(row.get("metadata")) + if object_key is None: + return pg_payload + + try: + payload = await cold_storage_handler.get_proxy_server_request_from_cold_storage_with_object_key( + object_key=object_key + ) + except Exception: + verbose_proxy_logger.warning( + "Failed to fetch cold storage payload for key %s; falling back to DB values", + object_key, + exc_info=True, + ) + return pg_payload + if payload is None: + return pg_payload + + return RequestResponsePayload( + messages=payload.get("messages"), + response=payload.get("response"), + proxy_server_request=payload.get("proxy_server_request"), + ) + + @router.get( "/spend/logs/ui/{request_id}", tags=["Budget & Spend Tracking"], @@ -2179,11 +2270,11 @@ async def ui_view_spend_logs( ) async def ui_view_request_response_for_request_id( request_id: str, - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing key spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view key spend", ), @@ -2215,8 +2306,8 @@ async def ui_view_request_response_for_request_id( ) custom_loggers = litellm.logging_callback_manager.get_active_additional_logging_utils_from_custom_logger() - start_date_obj: Optional[datetime] = None - end_date_obj: Optional[datetime] = None + start_date_obj: datetime | None = None + end_date_obj: datetime | None = None if start_date is not None: start_date_obj = datetime.strptime(start_date, "%Y-%m-%d %H:%M:%S").replace( tzinfo=timezone.utc @@ -2235,26 +2326,27 @@ async def ui_view_request_response_for_request_id( if payload is not None: return payload - # Fallback: fetch heavy columns directly from the database. - # The list endpoint (/spend/logs/ui) intentionally excludes messages, - # response, and proxy_server_request for performance. When no custom - # logger (S3, GCS, etc.) is configured, we still need to serve these - # fields from the DB for the detail/drawer view. + # Fallback: the list endpoint omits the heavy columns for performance, so + # serve them here. When prompts were offloaded to cold storage the DB holds + # only placeholders, so _resolve_request_response_payload fetches the real + # payload from the configured cold storage backend by object key. if prisma_client is not None: + from litellm.proxy.spend_tracking.cold_storage_handler import ( + ColdStorageHandler, + ) + sql_query = """ - SELECT messages, response, proxy_server_request + SELECT messages, response, proxy_server_request, metadata FROM "LiteLLM_SpendLogs" WHERE request_id = $1 LIMIT 1 """ db_result = await prisma_client.db.query_raw(sql_query, request_id) if db_result and len(db_result) > 0: - row = db_result[0] - return { - "messages": row.get("messages"), - "response": row.get("response"), - "proxy_server_request": row.get("proxy_server_request"), - } + resolved = await _resolve_request_response_payload( + db_result[0], cold_storage_handler=ColdStorageHandler() + ) + return resolved._asdict() return None @@ -2268,23 +2360,23 @@ async def ui_view_request_response_for_request_id( }, ) async def view_spend_logs( - api_key: Optional[str] = fastapi.Query( + api_key: str | None = fastapi.Query( default=None, description="Get spend logs based on api key", ), - user_id: Optional[str] = fastapi.Query( + user_id: str | None = fastapi.Query( default=None, description="Get spend logs based on user_id", ), - request_id: Optional[str] = fastapi.Query( + request_id: str | None = fastapi.Query( default=None, description="request_id to get spend logs for specific request_id. If none passed then pass spend logs for all requests", ), - start_date: Optional[str] = fastapi.Query( + start_date: str | None = fastapi.Query( default=None, description="Time from which to start viewing key spend", ), - end_date: Optional[str] = fastapi.Query( + end_date: str | None = fastapi.Query( default=None, description="Time till which to view key spend", ), @@ -2622,7 +2714,7 @@ async def global_spend_refresh(): async def global_spend_for_internal_user( - api_key: Optional[str] = None, + api_key: str | None = None, user_api_key_dict: UserAPIKeyAuth = Depends(user_api_key_auth), ): from litellm.proxy.proxy_server import prisma_client @@ -2666,7 +2758,7 @@ async def global_spend_for_internal_user( include_in_schema=False, ) async def global_spend_logs( - api_key: Optional[str] = fastapi.Query( + api_key: str | None = fastapi.Query( default=None, description="API Key to get global spend (spend per day for last 30d). Admin-only endpoint", ), @@ -3025,7 +3117,7 @@ async def global_view_all_end_users(): dependencies=[Depends(user_api_key_auth)], include_in_schema=False, ) -async def global_spend_end_users(data: Optional[GlobalEndUsersSpend] = None): +async def global_spend_end_users(data: GlobalEndUsersSpend | None = None): """ [BETA] This is a beta endpoint. It will change. @@ -3258,8 +3350,8 @@ async def get_spend_by_tags( async def ui_get_spend_by_tags( start_date: str, end_date: str, - prisma_client: Optional[PrismaClient] = None, - tags_str: Optional[str] = None, + prisma_client: PrismaClient | None = None, + tags_str: str | None = None, ): """ Should cover 2 cases: @@ -3270,7 +3362,7 @@ async def ui_get_spend_by_tags( # tags_str is a list of strings csv of tags # tags_str = tag1,tag2,tag3 # convert to list if it's not None - tags_list: Optional[List[str]] = None + tags_list: List[str] | None = None if tags_str is not None and len(tags_str) > 0: tags_list = tags_str.split(",") @@ -3534,7 +3626,7 @@ async def _build_ui_spend_logs_response( } -def _build_status_filter_condition(status_filter: Optional[str]) -> Dict[str, Any]: +def _build_status_filter_condition(status_filter: str | None) -> Dict[str, Any]: """ Helper function to build the status filter condition for database queries. @@ -3573,7 +3665,7 @@ def _is_admin_view_safe(user_api_key_dict: UserAPIKeyAuth) -> bool: async def _can_team_member_view_log( prisma_client, user_api_key_dict: UserAPIKeyAuth, - team_id: Optional[str], + team_id: str | None, ) -> bool: """ Check if the requesting user can view spend logs for the given team. diff --git a/litellm/rerank_api/main.py b/litellm/rerank_api/main.py index e40e12e9197..35d40423bca 100644 --- a/litellm/rerank_api/main.py +++ b/litellm/rerank_api/main.py @@ -1,7 +1,7 @@ import asyncio import contextvars from functools import partial -from typing import Any, Coroutine, Dict, List, Literal, Optional, Union +from typing import Any, Coroutine, Dict, List, Literal, Union import litellm from litellm._logging import verbose_logger @@ -30,15 +30,16 @@ async def arerank( model: str, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[ + custom_llm_provider: ( Literal[ "cohere", "together_ai", "deepinfra", "fireworks_ai", "voyage", "watsonx" ] - ] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = None, - max_chunks_per_doc: Optional[int] = None, + | None + ) = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = None, + max_chunks_per_doc: int | None = None, **kwargs, ) -> Union[RerankResponse, Coroutine[Any, Any, RerankResponse]]: """ @@ -79,7 +80,7 @@ def rerank( model: str, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[ + custom_llm_provider: ( Literal[ "cohere", "together_ai", @@ -92,20 +93,26 @@ def rerank( "voyage", "watsonx", ] - ] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, + | None + ) = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, **kwargs, ) -> Union[RerankResponse, Coroutine[Any, Any, RerankResponse]]: """ Reranks a list of documents based on their relevance to the query """ - headers: Optional[dict] = kwargs.get("headers") # type: ignore + # `instruction` is read from kwargs rather than declared as a named param. + # The router forwards rerank calls via an untyped `**kwargs` unpack, and a + # typed named param there would trip the basedpyright budget gate without + # adding real safety; it stays typed downstream via get_optional_rerank_params. + instruction: str | None = kwargs.get("instruction", None) + headers: dict | None = kwargs.get("headers") # type: ignore litellm_logging_obj: LiteLLMLoggingObj = kwargs.get("litellm_logging_obj") # type: ignore - litellm_call_id: Optional[str] = kwargs.get("litellm_call_id", None) + litellm_call_id: str | None = kwargs.get("litellm_call_id", None) proxy_server_request = kwargs.get("proxy_server_request", None) model_info = kwargs.get("model_info", None) user = kwargs.get("user", None) @@ -155,6 +162,7 @@ def rerank( return_documents=return_documents, max_chunks_per_doc=max_chunks_per_doc, max_tokens_per_doc=max_tokens_per_doc, + instruction=instruction, non_default_params=kwargs, ) verbose_logger.info(f"optional_rerank_params: {optional_rerank_params}") @@ -187,11 +195,11 @@ def rerank( or _custom_llm_provider == litellm.LlmProviders.LITELLM_PROXY ): # Implement Cohere rerank logic - api_key: Optional[str] = ( + api_key: str | None = ( dynamic_api_key or optional_params.api_key or litellm.api_key ) - api_base: Optional[str] = ( + api_base: str | None = ( dynamic_api_base or optional_params.api_base or litellm.api_base diff --git a/litellm/rerank_api/rerank_utils.py b/litellm/rerank_api/rerank_utils.py index 38e599ef824..856029e45ea 100644 --- a/litellm/rerank_api/rerank_utils.py +++ b/litellm/rerank_api/rerank_utils.py @@ -1,4 +1,4 @@ -from typing import Any, Dict, List, Optional, Union +from typing import Any, Dict, List, Union from litellm.llms.base_llm.rerank.transformation import BaseRerankConfig @@ -9,13 +9,14 @@ def get_optional_rerank_params( drop_params: bool, query: str, documents: List[Union[str, Dict[str, Any]]], - custom_llm_provider: Optional[str] = None, - top_n: Optional[int] = None, - rank_fields: Optional[List[str]] = None, - return_documents: Optional[bool] = True, - max_chunks_per_doc: Optional[int] = None, - max_tokens_per_doc: Optional[int] = None, - non_default_params: Optional[dict] = None, + custom_llm_provider: str | None = None, + top_n: int | None = None, + rank_fields: List[str] | None = None, + return_documents: bool | None = True, + max_chunks_per_doc: int | None = None, + max_tokens_per_doc: int | None = None, + instruction: str | None = None, + non_default_params: dict | None = None, ) -> Dict: all_non_default_params = non_default_params or {} if query is not None: @@ -30,6 +31,11 @@ def get_optional_rerank_params( all_non_default_params["max_chunks_per_doc"] = max_chunks_per_doc if max_tokens_per_doc is not None: all_non_default_params["max_tokens_per_doc"] = max_tokens_per_doc + if instruction is not None: + # Also surfaced in non_default_params so providers that read it from + # there (e.g. DeepInfra) keep working now that `rerank()` consumes + # `instruction` as a named param instead of leaving it in **kwargs. + all_non_default_params["instruction"] = instruction return rerank_provider_config.map_cohere_rerank_params( model=model, drop_params=drop_params, @@ -41,5 +47,6 @@ def get_optional_rerank_params( return_documents=return_documents, max_chunks_per_doc=max_chunks_per_doc, max_tokens_per_doc=max_tokens_per_doc, + instruction=instruction, non_default_params=all_non_default_params, ) diff --git a/litellm/router.py b/litellm/router.py index 1aba259a328..20f8eba22ff 100644 --- a/litellm/router.py +++ b/litellm/router.py @@ -320,6 +320,7 @@ class Router: "latency-based-routing", "cost-based-routing", "usage-based-routing-v2", + "lar1", ] = "simple-shuffle", optional_pre_call_checks: Optional[OptionalPreCallChecks] = None, routing_strategy_args: dict = {}, # just for latency-based @@ -639,10 +640,15 @@ class Router: """ ### ROUTING SETUP ### - self.routing_strategy_init( - routing_strategy=routing_strategy, - routing_strategy_args=routing_strategy_args, - ) + if self._normalize_strategy(routing_strategy) == "lar1": + from litellm.router_strategy.lar1_routing import apply_lar1_routing_strategy + + apply_lar1_routing_strategy(self, routing_strategy_args) + else: + self.routing_strategy_init( + routing_strategy=routing_strategy, + routing_strategy_args=routing_strategy_args, + ) self._init_routing_groups(self._routing_groups_input) self.access_groups = None ## USAGE TRACKING ## @@ -863,7 +869,9 @@ class Router: self, routing_strategy: Union[RoutingStrategy, str, None] ) -> None: # See: https://github.com/BerriAI/litellm/issues/11330 - valid_strategy_strings = ["simple-shuffle"] + [s.value for s in RoutingStrategy] + valid_strategy_strings = ["simple-shuffle", "lar1"] + [ + s.value for s in RoutingStrategy + ] if routing_strategy is None: return is_valid_string = ( @@ -953,6 +961,7 @@ class Router: ): verbose_router_logger.info(f"Routing strategy: {routing_strategy}") self._validate_routing_strategy(routing_strategy) + self._reset_custom_routing_strategy() self._unregister_router_selectors( [ @@ -10601,6 +10610,7 @@ class Router: _existing_router_settings = self.get_settings() rebuild_routing_groups = False + relink_lar1_from_args = False for var in kwargs: if var in _allowed_settings: if var in _int_settings: @@ -10615,17 +10625,37 @@ class Router: if var == "routing_strategy": value = self._normalize_strategy(value) if _existing_router_settings["routing_strategy"] != value: - self.routing_strategy_init( - routing_strategy=value, - routing_strategy_args=kwargs.get( - "routing_strategy_args", {} - ), - ) + if value == "lar1": + from litellm.router_strategy.lar1_routing import ( + apply_lar1_routing_strategy, + ) + + apply_lar1_routing_strategy( + self, + kwargs.get("routing_strategy_args"), + ) + else: + self.routing_strategy_init( + routing_strategy=value, + routing_strategy_args=kwargs.get( + "routing_strategy_args", {} + ), + ) rebuild_routing_groups = True + elif var == "routing_strategy_args": + relink_lar1_from_args = True setattr(self, var, value) else: verbose_router_logger.debug("Setting {} is not allowed".format(var)) + if ( + relink_lar1_from_args + and self._normalize_strategy(self.routing_strategy) == "lar1" + ): + from litellm.router_strategy.lar1_routing import apply_lar1_routing_strategy + + apply_lar1_routing_strategy(self, self.routing_strategy_args) + if rebuild_routing_groups: self._init_routing_groups(self._routing_groups_input) verbose_router_logger.debug(f"Updated Router settings: {self.get_settings()}") @@ -12189,6 +12219,11 @@ class Router: CustomRoutingStrategy.async_get_available_deployment, ) + def _reset_custom_routing_strategy(self) -> None: + for attr in ("get_available_deployment", "async_get_available_deployment"): + if attr in self.__dict__: + delattr(self, attr) + def flush_cache(self): litellm.cache = None self.cache.flush_cache() diff --git a/litellm/router_strategy/lar1_routing.py b/litellm/router_strategy/lar1_routing.py new file mode 100644 index 00000000000..31aa5a96bad --- /dev/null +++ b/litellm/router_strategy/lar1_routing.py @@ -0,0 +1,192 @@ +""" +LAR-1 Semantic Routing Strategy + +Routes requests based on agent confidence level (LAR-1 protocol). +Thresholds are configurable via routing_strategy_args in router config. + +LAR-1 metadata passed via request_kwargs["metadata"]["lar1"] +""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import TYPE_CHECKING, Optional, Union + +from pydantic import ValidationError + +from litellm._logging import verbose_router_logger +from litellm.router import CustomRoutingStrategyBase +from litellm.types.lar1 import LAR1Metadata, LAR1Time + +if TYPE_CHECKING: + from litellm.router import Router + +DEFAULT_THRESHOLDS: dict[str, float] = {"low": 0.3, "medium": 0.5, "high": 0.7} + + +def _coerce_threshold(value: object, default: float) -> float: + if isinstance(value, (int, float)): + return float(value) + return default + + +def lar1_thresholds_from_args( + routing_strategy_args: Optional[Mapping[str, object]] = None, +) -> dict[str, float]: + args = routing_strategy_args or {} + return { + "low": _coerce_threshold( + args.get("confidence_threshold_low"), DEFAULT_THRESHOLDS["low"] + ), + "medium": _coerce_threshold( + args.get("confidence_threshold_medium"), DEFAULT_THRESHOLDS["medium"] + ), + "high": _coerce_threshold( + args.get("confidence_threshold_high"), DEFAULT_THRESHOLDS["high"] + ), + } + + +def apply_lar1_routing_strategy( + router: Router, + routing_strategy_args: Optional[Mapping[str, object]] = None, +) -> None: + strategy = LAR1RoutingStrategy( + router_instance=router, + thresholds=lar1_thresholds_from_args(routing_strategy_args), + ) + router.routing_strategy = "lar1" + router.set_custom_routing_strategy(strategy) + + +def _normalize_thresholds(thresholds: Optional[dict[str, float]]) -> dict[str, float]: + merged = {**DEFAULT_THRESHOLDS, **(thresholds or {})} + low = merged["low"] + medium = merged["medium"] + high = merged["high"] + if not (0 < low < medium < high < 1): + raise ValueError( + "LAR-1 thresholds must satisfy 0 < low < medium < high < 1, " + f"got low={low}, medium={medium}, high={high}" + ) + return merged + + +def _parse_lar1_metadata(request_kwargs: dict) -> LAR1Metadata: + lar1_raw = request_kwargs.get("metadata", {}).get("lar1", {}) + if not isinstance(lar1_raw, dict): + verbose_router_logger.warning( + f"[LAR-1] Invalid lar1 metadata type: {type(lar1_raw).__name__}. Using defaults" + ) + return LAR1Metadata() + try: + return LAR1Metadata.model_validate(lar1_raw) + except ValidationError as exc: + verbose_router_logger.warning( + f"[LAR-1] Invalid lar1 metadata: {exc}. Using defaults" + ) + return LAR1Metadata() + + +class LAR1RoutingStrategy(CustomRoutingStrategyBase): + def __init__( + self, + router_instance: Optional[Router] = None, + thresholds: Optional[dict[str, float]] = None, + ): + self._router = router_instance + self.thresholds = _normalize_thresholds(thresholds) + + async def async_get_available_deployment( + self, + model: str, + messages: Optional[list[dict[str, str]]] = None, + input: Optional[Union[str, list]] = None, + specific_deployment: Optional[bool] = False, + request_kwargs: Optional[dict] = None, + ): + if request_kwargs is None: + request_kwargs = {} + if self._router is None: + return None + + lar1 = _parse_lar1_metadata(request_kwargs) + confidence = lar1.confidence + evidence = tuple(e.value for e in lar1.evidence) + time_dim = lar1.time.value + + healthy = await self._router.async_get_healthy_deployments( + model=model, + request_kwargs=request_kwargs, + messages=messages, + input=input, + specific_deployment=specific_deployment, + ) + if isinstance(healthy, dict): + return healthy + + if not healthy: + return None + + target = self._classify_request(confidence, evidence, time_dim) + selected, exact_match = self._select_deployment(target, healthy) + + if selected is None: + return None + if exact_match: + verbose_router_logger.info(f"[LAR-1] confidence={confidence} -> {target}") + else: + actual_type = selected.get("model_info", {}).get("type", "unknown") + verbose_router_logger.warning( + f"[LAR-1] No deployment for type '{target}', " + f"fallback to deployment type '{actual_type}'" + ) + return selected + + def _classify_request( + self, + confidence: float, + evidence: tuple[str, ...], + time_dim: str, + ) -> str: + if "UNVERIFIED" in evidence: + return "cloud-smart" + + if time_dim == LAR1Time.MEM.value: + return "cloud-fast" + + t = self.thresholds + if confidence < t["low"]: + return "cloud-smart" + if confidence < t["medium"]: + return "cloud-fast" + if confidence < t["high"]: + return "local" + return "deep" + + def _select_deployment( + self, + target_type: str, + deployments: list[dict], + ) -> tuple[Optional[dict], bool]: + if not deployments: + return None, False + + for deployment in deployments: + if not isinstance(deployment, dict): + continue + model_type = deployment.get("model_info", {}).get("type", "") + if model_type == target_type: + return deployment, True + + for deployment in deployments: + if isinstance(deployment, dict): + return deployment, False + + return None, False + + def get_available_deployment(self, *args, **kwargs): + raise NotImplementedError( + "LAR-1 routing only supports async routing. " + "Enable async_only_mode on the router or use acompletion." + ) diff --git a/litellm/router_utils/fallback_event_handlers.py b/litellm/router_utils/fallback_event_handlers.py index f0edc7fc9db..891d80d785a 100644 --- a/litellm/router_utils/fallback_event_handlers.py +++ b/litellm/router_utils/fallback_event_handlers.py @@ -72,7 +72,7 @@ def get_fallback_model_group( elif list(item.keys())[0] == "*": # check generic fallback generic_fallback_idx = idx elif isinstance(item, str): - fallback_model_group = [fallbacks.pop(idx)] # returns single-item list + fallback_model_group = [item] ## if none, check for generic fallback if fallback_model_group is None: if stripped_model_fallback is not None: diff --git a/litellm/types/lar1.py b/litellm/types/lar1.py new file mode 100644 index 00000000000..5998b665ece --- /dev/null +++ b/litellm/types/lar1.py @@ -0,0 +1,39 @@ +from enum import Enum + +from pydantic import BaseModel, Field + + +class LAR1Act(str, Enum): + INF = "INF" + OBS = "OBS" + RET = "RET" + GEN = "GEN" + + +class LAR1Time(str, Enum): + NOW = "NOW" + MEM = "MEM" + CTX = "CTX" + PRE = "PRE" + + +class LAR1Mind(str, Enum): + REF = "REF" + REC = "REC" + HYP = "HYP" + ACT = "ACT" + + +class LAR1Evidence(str, Enum): + SYNTH = "SYNTH" + RETRIEVED = "RETRIEVED" + UNVERIFIED = "UNVERIFIED" + CONFIRMED = "CONFIRMED" + + +class LAR1Metadata(BaseModel): + act: LAR1Act = LAR1Act.INF + time: LAR1Time = LAR1Time.NOW + mind: LAR1Mind = LAR1Mind.REF + confidence: float = Field(default=0.5, ge=0.0, le=1.0) + evidence: list[LAR1Evidence] = [] diff --git a/litellm/types/rerank.py b/litellm/types/rerank.py index d2c252a1e92..376d6f66603 100644 --- a/litellm/types/rerank.py +++ b/litellm/types/rerank.py @@ -19,6 +19,10 @@ class RerankRequest(BaseModel): return_documents: Optional[bool] = None max_chunks_per_doc: Optional[int] = None max_tokens_per_doc: Optional[int] = None + # Optional task/query instruction passed through to providers that support it + # (e.g. hosted vLLM / Qwen3-Reranker, DeepInfra). Omitted from the outgoing + # request when None, so this is fully backward-compatible. + instruction: Optional[str] = None class OptionalRerankParams(TypedDict, total=False): @@ -29,6 +33,7 @@ class OptionalRerankParams(TypedDict, total=False): return_documents: Optional[bool] max_chunks_per_doc: Optional[int] max_tokens_per_doc: Optional[int] + instruction: Optional[str] class RerankBilledUnits(TypedDict, total=False): diff --git a/litellm/types/utils.py b/litellm/types/utils.py index c3f99b1d18b..44c4ee3d05c 100644 --- a/litellm/types/utils.py +++ b/litellm/types/utils.py @@ -2446,7 +2446,7 @@ class ImageResponse(OpenAIImageResponse, BaseLiteLLMOpenAIResponseObject): class TranscriptionUsageDurationObject(BaseModel): type: Literal["duration"] - seconds: int + seconds: float class TranscriptionUsageInputTokenDetailsObject(BaseModel): diff --git a/litellm/utils.py b/litellm/utils.py index e0aa8575473..cfff7e914c6 100644 --- a/litellm/utils.py +++ b/litellm/utils.py @@ -5854,6 +5854,9 @@ def _is_potential_model_name_in_model_cost( ) +_ABOVE_THRESHOLD_COST_KEY = re.compile(r"_above_\d+k?_tokens$") + + def _get_model_info_helper( model: str, custom_llm_provider: Optional[str] = None, @@ -6031,7 +6034,7 @@ def _get_model_info_helper( ) _output_cost_per_token = 0 - return ModelInfoBase( + returned_model_info = ModelInfoBase( key=key, max_tokens=_model_info.get("max_tokens", None), max_input_tokens=_model_info.get("max_input_tokens", None), @@ -6248,6 +6251,13 @@ def _get_model_info_helper( uses_embed_content=_model_info.get("uses_embed_content", None), supports_image_size=_model_info.get("supports_image_size", None), ) + for cost_key, cost_value in _model_info.items(): + if ( + cost_key not in returned_model_info + and _ABOVE_THRESHOLD_COST_KEY.search(cost_key) is not None + ): + returned_model_info[cost_key] = cost_value # type: ignore[literal-required] + return returned_model_info except Exception as e: verbose_logger.debug(f"Error getting model info: {e}") raise Exception( diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json index a174c1b5efd..5d2ed4244e0 100644 --- a/model_prices_and_context_window.json +++ b/model_prices_and_context_window.json @@ -583,6 +583,15 @@ "output_cost_per_token": 0.0, "output_vector_size": 1536 }, + "amazon.titan-embed-g1-text-02": { + "input_cost_per_token": 1e-07, + "litellm_provider": "bedrock", + "max_input_tokens": 8192, + "max_tokens": 8192, + "mode": "embedding", + "output_cost_per_token": 0.0, + "output_vector_size": 1536 + }, "amazon.titan-embed-text-v2:0": { "input_cost_per_token": 2e-08, "litellm_provider": "bedrock", @@ -22570,7 +22579,7 @@ "input_cost_per_token_batches": 3.75e-07, "input_cost_per_token_priority": 1.5e-06, "litellm_provider": "openai", - "max_input_tokens": 272000, + "max_input_tokens": 1050000, "max_output_tokens": 128000, "max_tokens": 128000, "mode": "chat", @@ -22618,7 +22627,7 @@ "input_cost_per_token_batches": 3.75e-07, "input_cost_per_token_priority": 1.5e-06, "litellm_provider": "openai", - "max_input_tokens": 272000, + "max_input_tokens": 1050000, "max_output_tokens": 128000, "max_tokens": 128000, "mode": "chat", @@ -22664,7 +22673,7 @@ "input_cost_per_token_flex": 1e-07, "input_cost_per_token_batches": 1e-07, "litellm_provider": "openai", - "max_input_tokens": 272000, + "max_input_tokens": 1050000, "max_output_tokens": 128000, "max_tokens": 128000, "mode": "chat", @@ -22709,7 +22718,7 @@ "input_cost_per_token_flex": 1e-07, "input_cost_per_token_batches": 1e-07, "litellm_provider": "openai", - "max_input_tokens": 272000, + "max_input_tokens": 1050000, "max_output_tokens": 128000, "max_tokens": 128000, "mode": "chat", @@ -22750,9 +22759,9 @@ "input_cost_per_token": 1.5e-05, "input_cost_per_token_batches": 7.5e-06, "litellm_provider": "openai", - "max_input_tokens": 128000, - "max_output_tokens": 272000, - "max_tokens": 272000, + "max_input_tokens": 400000, + "max_output_tokens": 128000, + "max_tokens": 128000, "mode": "responses", "output_cost_per_token": 0.00012, "output_cost_per_token_batches": 6e-05, @@ -22786,9 +22795,9 @@ "input_cost_per_token": 1.5e-05, "input_cost_per_token_batches": 7.5e-06, "litellm_provider": "openai", - "max_input_tokens": 128000, - "max_output_tokens": 272000, - "max_tokens": 272000, + "max_input_tokens": 400000, + "max_output_tokens": 128000, + "max_tokens": 128000, "mode": "responses", "output_cost_per_token": 0.00012, "output_cost_per_token_batches": 6e-05, @@ -30087,6 +30096,22 @@ "supports_reasoning": true, "supports_tool_choice": true }, + "openrouter/z-ai/glm-5.1": { + "input_cost_per_token": 1.05e-06, + "output_cost_per_token": 3.5e-06, + "cache_read_input_token_cost": 5.25e-07, + "cache_creation_input_token_cost": 0.0, + "litellm_provider": "openrouter", + "max_input_tokens": 202752, + "max_output_tokens": 65535, + "max_tokens": 65535, + "mode": "chat", + "source": "https://openrouter.ai/z-ai/glm-5.1", + "supports_function_calling": true, + "supports_prompt_caching": true, + "supports_reasoning": true, + "supports_tool_choice": true + }, "openrouter/minimax/minimax-m2.1": { "input_cost_per_token": 2.7e-07, "output_cost_per_token": 1.2e-06, @@ -31586,13 +31611,13 @@ "output_cost_per_token": 0.0 }, "sambanova/MiniMax-M2.7": { - "input_cost_per_token": 3e-07, + "input_cost_per_token": 6e-07, "litellm_provider": "sambanova", - "max_input_tokens": 204800, + "max_input_tokens": 196608, "max_output_tokens": 131072, "max_tokens": 131072, "mode": "chat", - "output_cost_per_token": 1.2e-06, + "output_cost_per_token": 2.4e-06, "source": "https://cloud.sambanova.ai/plans/pricing", "supports_function_calling": true, "supports_reasoning": true, @@ -31609,6 +31634,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/DeepSeek-R1-Distill-Llama-70B": { + "deprecation_date": "2026-03-20", "input_cost_per_token": 7e-07, "litellm_provider": "sambanova", "max_input_tokens": 131072, @@ -31619,6 +31645,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/DeepSeek-V3-0324": { + "deprecation_date": "2026-04-14", "input_cost_per_token": 3e-06, "litellm_provider": "sambanova", "max_input_tokens": 32768, @@ -31649,6 +31676,7 @@ "supports_vision": true }, "sambanova/Llama-4-Scout-17B-16E-Instruct": { + "deprecation_date": "2025-06-19", "input_cost_per_token": 4e-07, "litellm_provider": "sambanova", "max_input_tokens": 8192, @@ -31665,6 +31693,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-3.1-405B-Instruct": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 5e-06, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31678,6 +31707,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-3.1-8B-Instruct": { + "deprecation_date": "2026-04-14", "input_cost_per_token": 1e-07, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31691,6 +31721,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-3.2-1B-Instruct": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 4e-08, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31701,6 +31732,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/Meta-Llama-3.2-3B-Instruct": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 8e-08, "litellm_provider": "sambanova", "max_input_tokens": 4096, @@ -31724,6 +31756,7 @@ "supports_tool_choice": true }, "sambanova/Meta-Llama-Guard-3-8B": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 3e-07, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31734,6 +31767,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/QwQ-32B": { + "deprecation_date": "2025-06-25", "input_cost_per_token": 5e-07, "litellm_provider": "sambanova", "max_input_tokens": 16384, @@ -31744,6 +31778,7 @@ "source": "https://cloud.sambanova.ai/plans/pricing" }, "sambanova/Qwen2-Audio-7B-Instruct": { + "deprecation_date": "2025-06-19", "input_cost_per_token": 5e-07, "litellm_provider": "sambanova", "max_input_tokens": 4096, @@ -31755,6 +31790,7 @@ "supports_audio_input": true }, "sambanova/Qwen3-32B": { + "deprecation_date": "2026-04-06", "input_cost_per_token": 4e-07, "litellm_provider": "sambanova", "max_input_tokens": 8192, @@ -31768,9 +31804,9 @@ "supports_tool_choice": true }, "sambanova/DeepSeek-V3.1": { - "max_tokens": 32768, - "max_input_tokens": 32768, - "max_output_tokens": 32768, + "max_tokens": 131072, + "max_input_tokens": 131072, + "max_output_tokens": 131072, "input_cost_per_token": 3e-06, "output_cost_per_token": 4.5e-06, "litellm_provider": "sambanova", @@ -31784,13 +31820,36 @@ "max_tokens": 131072, "max_input_tokens": 131072, "max_output_tokens": 131072, + "input_cost_per_token": 2.2e-07, + "output_cost_per_token": 5.9e-07, + "litellm_provider": "sambanova", + "mode": "chat", + "supports_function_calling": true, + "supports_tool_choice": true, + "supports_reasoning": true, + "source": "https://cloud.sambanova.ai/plans/pricing" + }, + "sambanova/DeepSeek-V3.2": { + "max_tokens": 32768, + "max_input_tokens": 32768, + "max_output_tokens": 32768, "input_cost_per_token": 3e-06, "output_cost_per_token": 4.5e-06, "litellm_provider": "sambanova", "mode": "chat", "supports_function_calling": true, "supports_tool_choice": true, - "supports_reasoning": true, + "source": "https://cloud.sambanova.ai/plans/pricing" + }, + "sambanova/gemma-4-31B-it": { + "max_tokens": 131072, + "max_input_tokens": 131072, + "max_output_tokens": 131072, + "input_cost_per_token": 3.8e-07, + "output_cost_per_token": 1.15e-06, + "litellm_provider": "sambanova", + "mode": "chat", + "supports_vision": true, "source": "https://cloud.sambanova.ai/plans/pricing" }, "snowflake/claude-3-5-sonnet": { @@ -38121,6 +38180,21 @@ "supports_tool_choice": true, "source": "https://docs.z.ai/guides/overview/pricing" }, + "zai/glm-5.1": { + "cache_creation_input_token_cost": 0, + "cache_read_input_token_cost": 2.6e-07, + "input_cost_per_token": 1.4e-06, + "output_cost_per_token": 4.4e-06, + "litellm_provider": "zai", + "max_input_tokens": 200000, + "max_output_tokens": 128000, + "mode": "chat", + "supports_function_calling": true, + "supports_prompt_caching": true, + "supports_reasoning": true, + "supports_tool_choice": true, + "source": "https://docs.z.ai/guides/overview/pricing" + }, "zai/glm-5-code": { "cache_creation_input_token_cost": 0, "cache_read_input_token_cost": 3e-07, @@ -38151,6 +38225,21 @@ "supports_tool_choice": true, "source": "https://docs.z.ai/guides/overview/pricing" }, + "zai/glm-4.7-flash": { + "cache_creation_input_token_cost": 0, + "cache_read_input_token_cost": 0, + "input_cost_per_token": 0, + "output_cost_per_token": 0, + "litellm_provider": "zai", + "max_input_tokens": 200000, + "max_output_tokens": 128000, + "mode": "chat", + "supports_function_calling": true, + "supports_prompt_caching": true, + "supports_reasoning": true, + "supports_tool_choice": true, + "source": "https://docs.z.ai/guides/overview/pricing" + }, "zai/glm-4.6": { "cache_creation_input_token_cost": 0, "cache_read_input_token_cost": 1.1e-07, diff --git a/tests/litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py b/tests/litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py index a2f45e7188b..66d7e0bcbf9 100644 --- a/tests/litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py +++ b/tests/litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py @@ -166,3 +166,24 @@ class TestDeepSeekThinkingParams: ) assert "thinking" not in result + + def test_drop_unsupported_tools_removes_dangling_tool_choice(self): + optional_params = { + "tools": [ + {"type": "namespace", "name": "local_shell"}, + {"type": "function", "function": {"name": "get_weather"}}, + ], + "tool_choice": { + "type": "function", + "function": {"name": "local_shell"}, + }, + "parallel_tool_calls": True, + } + + result = self.config._drop_unsupported_tools(optional_params) + + assert result["tools"] == [ + {"type": "function", "function": {"name": "get_weather"}} + ] + assert "tool_choice" not in result + assert result["parallel_tool_calls"] is True diff --git a/tests/llm_translation/test_bedrock_embedding.py b/tests/llm_translation/test_bedrock_embedding.py index 92c22f582d9..2bc4192833b 100644 --- a/tests/llm_translation/test_bedrock_embedding.py +++ b/tests/llm_translation/test_bedrock_embedding.py @@ -34,6 +34,11 @@ img_base_64 = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAGQAAABkBAMAAACCzIh "text", titan_embedding_response, ), # V2 text model + ( + "bedrock/amazon.titan-embed-g1-text-02", + "text", + titan_embedding_response, + ), # G1 text model ( "bedrock/amazon.titan-embed-image-v1", "image", @@ -459,3 +464,13 @@ def test_bedrock_embedding_region_bug_reproduction(): os.environ["AWS_REGION_NAME"] = original_region_name else: os.environ.pop("AWS_REGION_NAME", None) + + +def test_bedrock_titan_g1_text_02_model_info(): + """Test that amazon.titan-embed-g1-text-02 has correct pricing metadata""" + model_info = litellm.get_model_info("amazon.titan-embed-g1-text-02") + assert model_info is not None, "Model info should not be None" + assert model_info["litellm_provider"] == "bedrock" + assert model_info["mode"] == "embedding" + assert model_info["input_cost_per_token"] == 1e-07 + assert model_info["max_input_tokens"] == 8192 diff --git a/tests/llm_translation/test_llm_response_utils/test_convert_dict_to_chat_completion.py b/tests/llm_translation/test_llm_response_utils/test_convert_dict_to_chat_completion.py index 4d67f1e426a..5683d973ac9 100644 --- a/tests/llm_translation/test_llm_response_utils/test_convert_dict_to_chat_completion.py +++ b/tests/llm_translation/test_llm_response_utils/test_convert_dict_to_chat_completion.py @@ -1627,7 +1627,12 @@ class TestMissingChoicesGuard: assert "no 'choices'" in exc_info.value.message def test_convert_to_model_response_object_empty_choices_raises_api_error(self): - """Empty choices list raises APIError.""" + """Empty choices list raises APIError, same as missing/null choices. + + Provider-specific repair (e.g. github_copilot synthesizing choices for + Anthropic-native responses) happens before this guard, in the provider + config; the core utility keeps treating empty choices as an error. + """ from litellm.exceptions import APIError response_object = { @@ -1683,7 +1688,9 @@ class TestMissingChoicesGuard: assert "no 'choices'" in exc_info.value.message - def test_convert_to_model_response_object_stream_true_no_choices_raises_api_error(self): + def test_convert_to_model_response_object_stream_true_no_choices_raises_api_error( + self, + ): """Missing choices via stream=True path raises APIError when generator is consumed.""" from litellm.exceptions import APIError @@ -2475,6 +2482,13 @@ class TestConvertToModelResponseObjectCompletion: def test_model_response_none_raises(self): with pytest.raises(Exception): convert_to_model_response_object( - response_object={"choices": [{"message": {"content": "hi", "role": "assistant"}, "finish_reason": "stop"}]}, + response_object={ + "choices": [ + { + "message": {"content": "hi", "role": "assistant"}, + "finish_reason": "stop", + } + ] + }, model_response_object=None, ) diff --git a/tests/local_testing/test_router_custom_routing.py b/tests/local_testing/test_router_custom_routing.py index 698ccac10dc..3ebd79a7b2a 100644 --- a/tests/local_testing/test_router_custom_routing.py +++ b/tests/local_testing/test_router_custom_routing.py @@ -80,6 +80,31 @@ class CustomRoutingStrategy(CustomRoutingStrategyBase): pass +def test_reset_custom_routing_strategy(): + """ + Setting a custom routing strategy installs instance-level overrides for + get_available_deployment / async_get_available_deployment. Re-initializing the + routing strategy must clear them so the class implementations are used again. + """ + router = _create_router() + router.set_custom_routing_strategy(CustomRoutingStrategy(router)) + + assert "get_available_deployment" in router.__dict__ + assert "async_get_available_deployment" in router.__dict__ + + router._reset_custom_routing_strategy() + + assert "get_available_deployment" not in router.__dict__ + assert "async_get_available_deployment" not in router.__dict__ + assert ( + router.async_get_available_deployment.__func__ + is Router.async_get_available_deployment + ) + + # idempotent: resetting again when nothing is overridden must not raise + router._reset_custom_routing_strategy() + + @pytest.mark.asyncio async def test_custom_routing(): litellm.set_verbose = True diff --git a/tests/test_litellm/integrations/focus/test_mavvrik_destination.py b/tests/test_litellm/integrations/focus/test_mavvrik_destination.py index 797238ae238..574c6186e07 100644 --- a/tests/test_litellm/integrations/focus/test_mavvrik_destination.py +++ b/tests/test_litellm/integrations/focus/test_mavvrik_destination.py @@ -2,8 +2,8 @@ from __future__ import annotations -from datetime import datetime, timezone -from unittest.mock import AsyncMock, MagicMock, patch +from datetime import datetime, timedelta, timezone +from unittest.mock import AsyncMock, MagicMock import pytest @@ -34,6 +34,12 @@ def _dest(**overrides) -> FocusMavvrikDestination: return FocusMavvrikDestination(prefix="mavvrik_focus_exports", config=config) +def _patch_resp(status: int = 204) -> MagicMock: + r = MagicMock() + r.status_code = status + return r + + def test_missing_api_key_raises(): with pytest.raises(ValueError, match="MAVVRIK_API_KEY"): FocusMavvrikDestination( @@ -83,11 +89,27 @@ def test_initializes_with_not_registered(): @pytest.mark.asyncio -async def test_deliver_skips_empty_content(): +async def test_deliver_skips_upload_but_advances_marker_for_empty_content(): dest = _dest() + + register_resp = MagicMock() + register_resp.status_code = 200 + register_resp.json.return_value = {"metricsMarker": 0} + + patch_resp = _patch_resp(204) + + mock_http = MagicMock() + mock_http.client = MagicMock() + mock_http.client.request = AsyncMock(side_effect=[register_resp, patch_resp]) + dest._http = mock_http + await dest.deliver(content=b"", time_window=_make_window(), filename="usage.csv") - # _registered still False — _ensure_registered was never called - assert dest._registered is False + + assert dest._registered is True + assert mock_http.client.request.call_count == 2 + patch_call = mock_http.client.request.call_args_list[1] + assert patch_call.kwargs["method"] == "PATCH" + assert "metricsMarker" in patch_call.kwargs["json"] @pytest.mark.asyncio @@ -112,9 +134,7 @@ async def test_large_content_uploads_in_multiple_chunks(): signed_url_resp = MagicMock() signed_url_resp.status_code = 200 - signed_url_resp.json.return_value = { - "url": "https://storage.googleapis.com/upload?sig=x" - } + signed_url_resp.json.return_value = {"url": "https://storage.googleapis.com/upload?sig=x"} init_resp = MagicMock() init_resp.status_code = 200 @@ -127,6 +147,8 @@ async def test_large_content_uploads_in_multiple_chunks(): chunk2_resp = MagicMock() chunk2_resp.status_code = 200 + patch_resp = _patch_resp(204) + mock_http = MagicMock() mock_http.client = MagicMock() mock_http.client.request = AsyncMock( @@ -136,6 +158,7 @@ async def test_large_content_uploads_in_multiple_chunks(): init_resp, chunk1_resp, chunk2_resp, + patch_resp, ] ) dest._http = mock_http @@ -152,15 +175,20 @@ async def test_large_content_uploads_in_multiple_chunks(): filename="usage.csv", ) - # register + get_signed_url + init + 2 chunk PUTs = 5 calls - assert mock_http.client.request.call_count == 5 + # register + get_signed_url + init + 2 chunk PUTs + PATCH = 6 calls + assert mock_http.client.request.call_count == 6 - # Check Content-Range headers - put_calls = mock_http.client.request.call_args_list[3:] + # Check Content-Range headers on the chunk PUTs (calls 3 and 4) + put_calls = mock_http.client.request.call_args_list[3:5] assert "bytes" in put_calls[0].kwargs["headers"]["Content-Range"] assert "/*" in put_calls[0].kwargs["headers"]["Content-Range"] # intermediate assert "/*" not in put_calls[1].kwargs["headers"]["Content-Range"] # final + # Verify the PATCH call advanced metricsMarker + patch_call = mock_http.client.request.call_args_list[5] + assert patch_call.kwargs["method"] == "PATCH" + assert "metricsMarker" in patch_call.kwargs["json"] + @pytest.mark.asyncio async def test_deliver_calls_register_get_url_and_upload(): @@ -180,12 +208,14 @@ async def test_deliver_calls_register_get_url_and_upload(): upload_resp = MagicMock() upload_resp.status_code = 200 + patch_resp = _patch_resp(204) + mock_http = MagicMock() mock_http.client = MagicMock() - # All 4 calls go through self._http.client.request: - # 1. register, 2. get_signed_url, 3. GCS session init POST, 4. GCS PUT + # All 5 calls go through self._http.client.request: + # 1. register, 2. get_signed_url, 3. GCS session init POST, 4. GCS PUT, 5. PATCH marker mock_http.client.request = AsyncMock( - side_effect=[register_resp, signed_url_resp, init_resp, upload_resp] + side_effect=[register_resp, signed_url_resp, init_resp, upload_resp, patch_resp] ) dest._http = mock_http @@ -196,10 +226,14 @@ async def test_deliver_calls_register_get_url_and_upload(): ) assert dest._registered is True - assert mock_http.client.request.call_count == 4 + assert mock_http.client.request.call_count == 5 # Verify Content-Range header was set on the PUT put_call = mock_http.client.request.call_args_list[3] assert "Content-Range" in put_call.kwargs["headers"] + # Verify PATCH was called last with metricsMarker + patch_call = mock_http.client.request.call_args_list[4] + assert patch_call.kwargs["method"] == "PATCH" + assert "metricsMarker" in patch_call.kwargs["json"] @pytest.mark.asyncio @@ -224,17 +258,19 @@ async def test_register_called_only_once_across_multiple_deliveries(): mock_http = MagicMock() mock_http.client = MagicMock() - # First delivery: register, get_signed_url, GCS init, GCS PUT - # Second delivery: get_signed_url, GCS init, GCS PUT (register skipped) + # First delivery: register, get_signed_url, GCS init, GCS PUT, PATCH + # Second delivery: get_signed_url, GCS init, GCS PUT, PATCH (register skipped) mock_http.client.request = AsyncMock( side_effect=[ register_resp, _signed_url_resp(), init_resp, upload_resp, + _patch_resp(204), _signed_url_resp(), init_resp, upload_resp, + _patch_resp(204), ] ) dest._http = mock_http @@ -243,8 +279,8 @@ async def test_register_called_only_once_across_multiple_deliveries(): await dest.deliver(content=b"header\nrow1\n", time_window=window, filename="1.csv") await dest.deliver(content=b"header\nrow2\n", time_window=window, filename="2.csv") - # 7 total: register(1) + [get_url+init+put](2) × 2 deliveries - assert mock_http.client.request.call_count == 7 + # 9 total: register(1) + [get_url+init+put+patch](4) × 2 deliveries + assert mock_http.client.request.call_count == 9 # First call was register first_call = mock_http.client.request.call_args_list[0] assert first_call.kwargs["method"] == "POST" @@ -343,9 +379,7 @@ async def test_deliver_raises_on_non_gcs_session_uri(): signed_url_resp = MagicMock() signed_url_resp.status_code = 200 # signed URL is valid GCS - signed_url_resp.json.return_value = { - "url": "https://storage.googleapis.com/upload?sig=abc" - } + signed_url_resp.json.return_value = {"url": "https://storage.googleapis.com/upload?sig=abc"} # Location header points to a non-GCS host init_resp = MagicMock() @@ -355,9 +389,7 @@ async def test_deliver_raises_on_non_gcs_session_uri(): mock_http = MagicMock() mock_http.client = MagicMock() # register, get_signed_url, GCS session init (returns bad Location) - mock_http.client.request = AsyncMock( - side_effect=[register_resp, signed_url_resp, init_resp] - ) + mock_http.client.request = AsyncMock(side_effect=[register_resp, signed_url_resp, init_resp]) dest._http = mock_http with pytest.raises(ValueError, match="GCS endpoint"): @@ -447,10 +479,11 @@ async def test_export_window_passes_max_rows_as_limit(monkeypatch): # Mock the engine internals so _export_window runs through our new code path db_mock = MagicMock() - db_mock.get_usage_data = AsyncMock(return_value=pl.DataFrame()) # empty → no upload + db_mock.get_usage_data = AsyncMock(return_value=pl.DataFrame()) # empty deliver engine_mock = MagicMock() engine_mock._database = db_mock + engine_mock._destination.deliver = AsyncMock() logger._engine = engine_mock window = FocusTimeWindow( @@ -545,6 +578,36 @@ async def test_run_scheduled_export_no_catchup_when_marker_is_current(): await logger._run_scheduled_export() # Only one call — yesterday's normal run, no catch-up + assert db_mock.get_usage_data.call_count == 1 + assert db_mock.get_usage_data.call_args.kwargs["start_time_utc"].date() == yesterday.date() + + +@pytest.mark.asyncio +async def test_run_scheduled_export_skips_catchup_when_marker_is_unparseable(): + import polars as pl + from litellm.integrations.mavvrik_focus.mavvrik_focus_logger import ( + MavvrikFocusLogger, + ) + from litellm.integrations.focus.destinations.mavvrik_destination import ( + FocusMavvrikDestination, + ) + + logger = MavvrikFocusLogger() + now = datetime.now(timezone.utc).replace(hour=0, minute=0, second=0, microsecond=0) + yesterday = now - timedelta(days=1) + + dest_mock = MagicMock(spec=FocusMavvrikDestination) + dest_mock.get_metrics_marker = AsyncMock(return_value="not-a-date") + + db_mock = MagicMock() + db_mock.get_usage_data = AsyncMock(return_value=pl.DataFrame()) + engine_mock = MagicMock() + engine_mock._database = db_mock + engine_mock._destination = dest_mock + logger._engine = engine_mock + + await logger._run_scheduled_export() + assert db_mock.get_usage_data.call_count == 1 assert ( db_mock.get_usage_data.call_args.kwargs["start_time_utc"].date() @@ -713,9 +776,7 @@ async def test_gcs_session_cancelled_on_chunk_failure(): signed_url_resp = MagicMock() signed_url_resp.status_code = 200 - signed_url_resp.json.return_value = { - "url": "https://storage.googleapis.com/upload?sig=x" - } + signed_url_resp.json.return_value = {"url": "https://storage.googleapis.com/upload?sig=x"} init_resp = MagicMock() init_resp.status_code = 200 @@ -749,3 +810,42 @@ async def test_gcs_session_cancelled_on_chunk_failure(): delete_call = calls[4] assert delete_call.kwargs["method"] == "DELETE" assert "storage.googleapis.com/session" in delete_call.kwargs["url"] + + +@pytest.mark.asyncio +async def test_update_metrics_marker_raises_on_non_410_error(): + dest = _dest() + + fail_resp = MagicMock() + fail_resp.status_code = 500 + fail_resp.text = "Internal Server Error" + + mock_http = MagicMock() + mock_http.client = MagicMock() + mock_http.client.request = AsyncMock(return_value=fail_resp) + dest._http = mock_http + + with pytest.raises(RuntimeError, match="failed to update metricsMarker"): + await dest._update_metrics_marker(1234567890) + assert mock_http.client.request.call_count == 1 + + +@pytest.mark.asyncio +async def test_update_metrics_marker_raises_on_410(): + """_update_metrics_marker must raise RuntimeError and reset _registered on 410.""" + dest = _dest() + dest._registered = True + + resp_410 = MagicMock() + resp_410.status_code = 410 + resp_410.text = "Gone" + + mock_http = MagicMock() + mock_http.client = MagicMock() + mock_http.client.request = AsyncMock(return_value=resp_410) + dest._http = mock_http + + with pytest.raises(RuntimeError, match="disconnected"): + await dest._update_metrics_marker(1234567890) + + assert dest._registered is False diff --git a/tests/test_litellm/integrations/mavvrik_focus/test_mavvrik_focus_logger.py b/tests/test_litellm/integrations/mavvrik_focus/test_mavvrik_focus_logger.py new file mode 100644 index 00000000000..cd21807e887 --- /dev/null +++ b/tests/test_litellm/integrations/mavvrik_focus/test_mavvrik_focus_logger.py @@ -0,0 +1,67 @@ +from datetime import datetime, timezone +from unittest.mock import AsyncMock, MagicMock + +import pytest + +from litellm.integrations.focus.destinations.base import FocusTimeWindow +from litellm.integrations.mavvrik_focus.mavvrik_focus_logger import MavvrikFocusLogger + + +class _Frame: + def __init__(self, *, empty: bool) -> None: + self._empty = empty + + def __len__(self) -> int: + return 0 if self._empty else 1 + + def is_empty(self) -> bool: + return self._empty + + +def _window() -> FocusTimeWindow: + return FocusTimeWindow( + start_time=datetime(2026, 1, 1, tzinfo=timezone.utc), + end_time=datetime(2026, 1, 2, tzinfo=timezone.utc), + frequency="daily", + ) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("data_empty", "normalized_empty", "serialized_payload"), + ( + (True, False, b"not-used"), + (False, True, b"not-used"), + (False, False, b""), + ), +) +async def test_export_window_delivers_empty_payload_for_empty_export( + data_empty: bool, + normalized_empty: bool, + serialized_payload: bytes, +) -> None: + database = MagicMock() + database.get_usage_data = AsyncMock(return_value=_Frame(empty=data_empty)) + transformer = MagicMock() + transformer.transform.return_value = _Frame(empty=normalized_empty) + serializer = MagicMock() + serializer.serialize.return_value = serialized_payload + destination = MagicMock() + destination.deliver = AsyncMock() + engine = MagicMock() + engine._database = database + engine._transformer = transformer + engine._serializer = serializer + engine._destination = destination + engine._build_filename.return_value = "metrics.csv" + logger = MavvrikFocusLogger() + logger._engine = engine + window = _window() + + await logger._export_window(window=window, limit=None) + + destination.deliver.assert_awaited_once_with( + content=b"", + time_window=window, + filename="metrics.csv", + ) diff --git a/tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py b/tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py index a5d1934237f..72fee9bfb40 100644 --- a/tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py +++ b/tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py @@ -384,6 +384,44 @@ def test_generic_cost_per_token_minimax_m3_above_512k_tokens(): assert round(completion_cost, 10) == round(expected_completion, 10) +def test_generic_cost_per_token_honors_non_standard_above_threshold(): + """Regression for #30344: get_model_info must keep arbitrary + input/output_cost_per_token_above__tokens thresholds, not only the hard-coded + 128k/200k/272k/512k set, so a custom tier boundary is applied past its limit.""" + model = "litellm-test-non-standard-tier" + custom_llm_provider = "openai" + litellm.register_model( + { + model: { + "litellm_provider": custom_llm_provider, + "mode": "chat", + "input_cost_per_token": 1e-6, + "output_cost_per_token": 2e-6, + "input_cost_per_token_above_500k_tokens": 9e-6, + "output_cost_per_token_above_500k_tokens": 18e-6, + } + } + ) + + try: + prompt_tokens = 600000 + completion_tokens = 1000 + usage = Usage( + prompt_tokens=prompt_tokens, + completion_tokens=completion_tokens, + total_tokens=prompt_tokens + completion_tokens, + ) + prompt_cost, completion_cost = generic_cost_per_token( + model=model, + usage=usage, + custom_llm_provider=custom_llm_provider, + ) + assert round(prompt_cost, 10) == round(9e-6 * prompt_tokens, 10) + assert round(completion_cost, 10) == round(18e-6 * completion_tokens, 10) + finally: + litellm.model_cost.pop(model, None) + + def test_generic_cost_per_token_gpt55(): """gpt-5.5: base pricing — $5/1M input, $30/1M output, $0.50/1M cached input.""" model = "gpt-5.5" diff --git a/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py b/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py index 2fdd639e74d..f934c7184f8 100644 --- a/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py +++ b/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_handler.py @@ -74,6 +74,150 @@ def test_redacted_thinking_content_block_delta(): assert "thinking_blocks" in model_response.choices[0].delta.provider_specific_fields +def test_streaming_thinking_blocks_are_replayable_after_signature_delta(): + model_response_iterator = ModelResponseIterator( + streaming_response=MagicMock(), sync_stream=True, json_mode=False + ) + chunks = [ + { + "type": "content_block_start", + "index": 0, + "content_block": {"type": "thinking", "thinking": ""}, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "thinking_delta", "thinking": "Step 1. "}, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "thinking_delta", "thinking": "Step 2."}, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "signature_delta", "signature": "sig-final"}, + }, + ] + + parsed_chunks = [ + model_response_iterator.chunk_parser(chunk=chunk) for chunk in chunks + ] + reasoning_content = "".join( + getattr(chunk.choices[0].delta, "reasoning_content", None) or "" + for chunk in parsed_chunks + ) + thinking_blocks = tuple( + block + for chunk in parsed_chunks + for block in (getattr(chunk.choices[0].delta, "thinking_blocks", None) or []) + ) + expected_delta_blocks = ( + {"type": "thinking", "thinking": "Step 1. "}, + {"type": "thinking", "thinking": "Step 2."}, + ) + expected_thinking_block = { + "type": "thinking", + "thinking": "Step 1. Step 2.", + "signature": "sig-final", + } + + assert reasoning_content == "Step 1. Step 2." + assert thinking_blocks == (*expected_delta_blocks, expected_thinking_block) + assert parsed_chunks[1].choices[0].delta.provider_specific_fields == { + "thinking_blocks": [expected_delta_blocks[0]] + } + assert parsed_chunks[-1].choices[0].delta.provider_specific_fields == { + "thinking_blocks": [expected_thinking_block] + } + + +def test_streaming_unsigned_thinking_deltas_keep_reasoning_content(): + model_response_iterator = ModelResponseIterator( + streaming_response=MagicMock(), sync_stream=True, json_mode=False + ) + chunks = [ + { + "type": "content_block_start", + "index": 0, + "content_block": {"type": "thinking", "thinking": ""}, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "thinking_delta", "thinking": "Step 1. "}, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "thinking_delta", "thinking": "Step 2."}, + }, + {"type": "content_block_stop", "index": 0}, + ] + + parsed_chunks = [ + model_response_iterator.chunk_parser(chunk=chunk) for chunk in chunks + ] + reasoning_content = "".join( + getattr(chunk.choices[0].delta, "reasoning_content", None) or "" + for chunk in parsed_chunks + ) + thinking_blocks = tuple( + block + for chunk in parsed_chunks + for block in (getattr(chunk.choices[0].delta, "thinking_blocks", None) or []) + ) + + assert reasoning_content == "Step 1. Step 2." + assert thinking_blocks == ( + {"type": "thinking", "thinking": "Step 1. "}, + {"type": "thinking", "thinking": "Step 2."}, + ) + + +def test_streaming_truncated_thinking_deltas_keep_reasoning_content(): + model_response_iterator = ModelResponseIterator( + streaming_response=MagicMock(), sync_stream=True, json_mode=False + ) + chunks = [ + { + "type": "content_block_start", + "index": 0, + "content_block": {"type": "thinking", "thinking": ""}, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "thinking_delta", "thinking": "Step 1. "}, + }, + { + "type": "content_block_delta", + "index": 0, + "delta": {"type": "thinking_delta", "thinking": "Step 2."}, + }, + ] + + parsed_chunks = [ + model_response_iterator.chunk_parser(chunk=chunk) for chunk in chunks + ] + reasoning_content = "".join( + getattr(chunk.choices[0].delta, "reasoning_content", None) or "" + for chunk in parsed_chunks + ) + thinking_blocks = tuple( + block + for chunk in parsed_chunks + for block in (getattr(chunk.choices[0].delta, "thinking_blocks", None) or []) + ) + + assert reasoning_content == "Step 1. Step 2." + assert thinking_blocks == ( + {"type": "thinking", "thinking": "Step 1. "}, + {"type": "thinking", "thinking": "Step 2."}, + ) + + def test_handle_json_mode_chunk_response_format_tool(): model_response_iterator = ModelResponseIterator( streaming_response=MagicMock(), sync_stream=True, json_mode=True diff --git a/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py b/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py index 6b869076044..301bcba99f4 100644 --- a/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py +++ b/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py @@ -158,6 +158,68 @@ def test_deepseek_cris(): assert bedrock_route == "converse" +def test_application_inference_profile_arn_routes_to_converse(): + """ + Regression for #18258: a bare application-inference-profile ARN passed as + `bedrock/arn:...` must route to converse. The ARN ends in an opaque id with + no provider substring, so the invoke path cannot build a provider-native + body and raises "Unknown provider=None". Converse needs no provider, so it + is the correct route. + """ + route = BedrockModelInfo.get_bedrock_route( + model="bedrock/arn:aws:bedrock:us-west-2:123412341234:application-inference-profile/a1b2c3" + ) + assert route == "converse" + + +def test_explicit_invoke_prefix_wins_over_application_inference_profile_arn(): + """ + An explicit invoke/ prefix is respected even for an application-inference-profile + ARN; only the bare `bedrock/arn:...` form is auto-routed to converse. The + explicit invoke path remains a dead end for these ARNs (no provider can be + derived, so completion raises "Unknown provider=None") by design: a caller + that explicitly asks for invoke gets invoke. The auto-route only rescues the + documented bare form. + """ + from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM + + model = "bedrock/invoke/arn:aws:bedrock:us-west-2:123412341234:application-inference-profile/a1b2c3" + assert BedrockModelInfo.get_bedrock_route(model) == "invoke" + assert BaseAWSLLM.get_bedrock_invoke_provider(model) is None + + +def test_system_defined_inference_profile_arn_still_routes_to_converse(): + """ + A system-defined cross-region inference-profile ARN embeds a known model, so + get_base_model resolves it and it already routes to converse. Guards that the + application-inference-profile fix does not change this working case. + """ + route = BedrockModelInfo.get_bedrock_route( + model="bedrock/arn:aws:bedrock:us-east-1:123:inference-profile/us.anthropic.claude-3-5-sonnet-20240620-v1:0" + ) + assert route == "converse" + + +def test_other_opaque_arn_types_still_route_to_invoke(): + """ + Only application-inference-profile ARNs are auto-routed to converse. Other + opaque ARNs (provisioned-model, imported-model, custom-model-deployment) + also yield no invoke provider, but they are frequently invoke-only with + provider-specific body formats, so routing them to converse could break + them. Guards the deliberate scope against an over-broad "any opaque ARN -> + converse" generalization. + """ + for arn_segment in ( + "provisioned-model/abcdefgh1234", + "imported-model/abcdefgh1234", + "custom-model-deployment/abcdefgh1234", + ): + route = BedrockModelInfo.get_bedrock_route( + model=f"bedrock/arn:aws:bedrock:us-east-1:123412341234:{arn_segment}" + ) + assert route == "invoke", f"{arn_segment} should stay on invoke route" + + def test_govcloud_cross_region_inference_prefix(): """ Test that GovCloud models with cross-region inference prefix (us-gov.) are parsed correctly diff --git a/tests/test_litellm/llms/cohere/rerank/test_rerank_guardrail_handler.py b/tests/test_litellm/llms/cohere/rerank/test_rerank_guardrail_handler.py index 88072cd7760..46c37e6af6c 100644 --- a/tests/test_litellm/llms/cohere/rerank/test_rerank_guardrail_handler.py +++ b/tests/test_litellm/llms/cohere/rerank/test_rerank_guardrail_handler.py @@ -2,10 +2,8 @@ Unit tests for Cohere Rerank Guardrail Translation Handler """ -import asyncio import os import sys -from typing import List, Optional, Tuple import pytest @@ -94,6 +92,74 @@ class TestInputProcessing: "id": "doc2", } + @pytest.mark.asyncio + async def test_process_query_and_instruction(self): + """Both query and instruction are guardrailed; documents untouched""" + handler = CohereRerankHandler() + guardrail = MockGuardrail(guardrail_name="test") + + data = { + "model": "qwen3-reranker", + "query": "What is machine learning?", + "instruction": "Rank by relevance to ML research", + "documents": ["Doc 1", "Doc 2"], + } + + result = await handler.process_input_messages(data, guardrail) + + # Both user-controlled text fields are scanned and written back + assert result["query"] == "What is machine learning? [GUARDRAILED]" + assert result["instruction"] == "Rank by relevance to ML research [GUARDRAILED]" + # Documents unchanged + assert result["documents"] == ["Doc 1", "Doc 2"] + + @pytest.mark.asyncio + async def test_instruction_masked_with_pii(self): + """A masking guardrail rewrites instruction, not just query""" + + class PIIMaskingGuardrail(CustomGuardrail): + async def apply_guardrail( + self, inputs: dict, request_data: dict, input_type: str, **kwargs + ) -> dict: + texts = inputs.get("texts", []) + return {"texts": [t.replace("John Doe", "[NAME_REDACTED]") for t in texts]} + + handler = CohereRerankHandler() + guardrail = PIIMaskingGuardrail(guardrail_name="mask_pii") + + data = { + "model": "qwen3-reranker", + "query": "find records", + "instruction": "prioritize anything authored by John Doe", + "documents": ["Doc 1"], + } + + result = await handler.process_input_messages(data, guardrail) + + # The sensitive value in instruction is sanitized before forwarding + assert "John Doe" not in result["instruction"] + assert "[NAME_REDACTED]" in result["instruction"] + assert result["documents"] == ["Doc 1"] + + @pytest.mark.asyncio + async def test_non_string_instruction_not_scanned(self): + """A non-string instruction is left as-is (only strings are scanned)""" + handler = CohereRerankHandler() + guardrail = MockGuardrail(guardrail_name="test") + + data = { + "model": "qwen3-reranker", + "query": "hello", + "instruction": 12345, # invalid type; backend will reject it + "documents": ["Doc 1"], + } + + result = await handler.process_input_messages(data, guardrail) + + # Query still guardrailed; non-string instruction untouched + assert result["query"] == "hello [GUARDRAILED]" + assert result["instruction"] == 12345 + @pytest.mark.asyncio async def test_process_no_query(self): """Test processing when query is missing""" diff --git a/tests/test_litellm/llms/deepseek/chat/__init__.py b/tests/test_litellm/llms/deepseek/chat/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/tests/test_litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py b/tests/test_litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py new file mode 100644 index 00000000000..ec51e5d303d --- /dev/null +++ b/tests/test_litellm/llms/deepseek/chat/test_deepseek_chat_transformation.py @@ -0,0 +1,103 @@ +from litellm.llms.deepseek.chat.transformation import DeepSeekChatConfig + + +def _function_tool(name: str) -> dict: + return { + "type": "function", + "function": {"name": name, "parameters": {"type": "object"}}, + } + + +def test_drop_unsupported_tools_keeps_function_tools_only(): + optional_params = { + "tools": [ + _function_tool("shell"), + {"type": "namespace", "name": "container.exec"}, + _function_tool("apply_patch"), + ], + "tool_choice": "auto", + } + + result = DeepSeekChatConfig._drop_unsupported_tools(optional_params) + + assert [tool["function"]["name"] for tool in result["tools"]] == [ + "shell", + "apply_patch", + ] + assert all(tool["type"] == "function" for tool in result["tools"]) + assert result["tool_choice"] == "auto" + + +def test_drop_unsupported_tools_drops_dangling_tool_choice_when_none_survive(): + optional_params = { + "tools": [{"type": "namespace", "name": "container.exec"}], + "tool_choice": "required", + "parallel_tool_calls": True, + "temperature": 0.2, + } + + result = DeepSeekChatConfig._drop_unsupported_tools(optional_params) + + assert "tools" not in result + assert "tool_choice" not in result + assert "parallel_tool_calls" not in result + assert result["temperature"] == 0.2 + + +def test_drop_unsupported_tools_is_noop_for_function_only(): + optional_params = { + "tools": [_function_tool("shell")], + "tool_choice": "auto", + } + + result = DeepSeekChatConfig._drop_unsupported_tools(optional_params) + + assert result is optional_params + + +def test_drop_unsupported_tools_is_noop_without_tools(): + optional_params = {"temperature": 0.7} + + result = DeepSeekChatConfig._drop_unsupported_tools(optional_params) + + assert result is optional_params + + +def test_transform_request_strips_unsupported_tools_from_body(): + config = DeepSeekChatConfig() + body = config.transform_request( + model="deepseek-chat", + messages=[{"role": "user", "content": "hi"}], + optional_params={ + "tools": [ + _function_tool("shell"), + {"type": "namespace", "name": "container.exec"}, + ], + "tool_choice": "auto", + }, + litellm_params={}, + headers={}, + ) + + assert [tool["type"] for tool in body["tools"]] == ["function"] + assert body["tools"][0]["function"]["name"] == "shell" + + +async def test_async_transform_request_strips_unsupported_tools_from_body(): + config = DeepSeekChatConfig() + body = await config.async_transform_request( + model="deepseek-chat", + messages=[{"role": "user", "content": "hi"}], + optional_params={ + "tools": [ + _function_tool("shell"), + {"type": "namespace", "name": "container.exec"}, + ], + "tool_choice": "auto", + }, + litellm_params={}, + headers={}, + ) + + assert [tool["type"] for tool in body["tools"]] == ["function"] + assert body["tools"][0]["function"]["name"] == "shell" diff --git a/tests/test_litellm/llms/github_copilot/test_github_copilot_transformation.py b/tests/test_litellm/llms/github_copilot/test_github_copilot_transformation.py index 5673ad81551..f69ba7df938 100644 --- a/tests/test_litellm/llms/github_copilot/test_github_copilot_transformation.py +++ b/tests/test_litellm/llms/github_copilot/test_github_copilot_transformation.py @@ -878,3 +878,107 @@ class TestGithubCopilotTransformResponse: litellm_params={}, encoding=None, ) + + +class TestGithubCopilotTransformParsedResponseDict: + """ + Tests for GithubCopilotConfig.transform_parsed_response_dict, the hook the + OpenAI SDK handler calls on its parsed response. That handler bypasses + transform_response, so this is the seam that repairs empty-choices responses + from newer Copilot Claude models on the live completion path. + + See: https://github.com/BerriAI/litellm/issues/30927 + """ + + def test_synthesizes_choices_from_anthropic_content(self): + config = GithubCopilotConfig() + + parsed = { + "id": "msg_vrtx_01", + "model": "claude-opus-4.8", + "object": "chat.completion", + "choices": [], + "content": [{"type": "text", "text": "Hello!"}], + "stop_reason": "end_turn", + "usage": {"input_tokens": 10, "output_tokens": 5}, + } + + repaired = config.transform_parsed_response_dict(parsed) + + assert len(repaired["choices"]) == 1 + choice = repaired["choices"][0] + assert choice["message"]["content"] == "Hello!" + assert choice["finish_reason"] == "stop" + assert repaired["usage"]["prompt_tokens"] == 10 + assert repaired["usage"]["completion_tokens"] == 5 + assert repaired["usage"]["total_tokens"] == 15 + + def test_passthrough_when_choices_present(self): + config = GithubCopilotConfig() + + parsed = { + "id": "chatcmpl-1", + "choices": [ + { + "index": 0, + "message": {"role": "assistant", "content": "ok"}, + "finish_reason": "stop", + } + ], + } + + assert config.transform_parsed_response_dict(parsed) is parsed + + +@patch("litellm.llms.openai.openai.OpenAIChatCompletion._get_openai_client") +@patch( + "litellm.llms.openai.openai.OpenAIChatCompletion.make_sync_openai_chat_completion_request" +) +def test_openai_handler_repairs_github_copilot_empty_choices( + mock_request, mock_get_client +): + """ + The OpenAI SDK handler calls convert_to_model_response_object directly on the + SDK's parsed output, bypassing transform_response. convert raises APIError on + empty choices, so the handler must route github_copilot responses through + transform_parsed_response_dict first. Removing that wiring (or resolving a + config without the override) fails this test with APIError. + + See: https://github.com/BerriAI/litellm/issues/30927 + """ + from litellm.llms.openai.openai import OpenAIChatCompletion + + mock_get_client.return_value = MagicMock() + + class _FakeSDKResponse: + def model_dump(self): + return { + "id": "msg_vrtx_01", + "model": "claude-opus-4.8", + "object": "chat.completion", + "choices": [], + "content": [{"type": "text", "text": "Hi there"}], + "stop_reason": "end_turn", + "usage": {"input_tokens": 12, "output_tokens": 3}, + } + + mock_request.return_value = ({}, _FakeSDKResponse()) + + result = OpenAIChatCompletion().completion( + model="claude-opus-4.8", + messages=[{"role": "user", "content": "Hi"}], + model_response=ModelResponse(), + timeout=60.0, + optional_params={}, + litellm_params={}, + logging_obj=MagicMock(), + custom_llm_provider="github_copilot", + client=MagicMock(), + api_key="gh.test-key-123456789", + acompletion=False, + ) + + assert isinstance(result, ModelResponse) + assert result.choices[0].message.content == "Hi there" + assert result.choices[0].finish_reason == "stop" + mock_request.assert_called_once() diff --git a/tests/test_litellm/llms/hosted_vllm/test_hosted_vllm_rerank_transformation.py b/tests/test_litellm/llms/hosted_vllm/test_hosted_vllm_rerank_transformation.py index 9e6fa608c50..6425e815db0 100644 --- a/tests/test_litellm/llms/hosted_vllm/test_hosted_vllm_rerank_transformation.py +++ b/tests/test_litellm/llms/hosted_vllm/test_hosted_vllm_rerank_transformation.py @@ -4,6 +4,7 @@ import sys import pytest from litellm.llms.hosted_vllm.rerank.transformation import HostedVLLMRerankConfig +from litellm.rerank_api.rerank_utils import get_optional_rerank_params from litellm.types.rerank import ( OptionalRerankParams, RerankBilledUnits, @@ -37,6 +38,54 @@ class TestHostedVLLMRerankTransform: assert params["rank_fields"] == ["field1"] assert params["return_documents"] is True + def test_map_cohere_rerank_params_omits_instruction_when_absent(self): + # Backward-compat: when no instruction is supplied, it must not appear + # in the mapped params (and therefore not in the outgoing request body). + params = self.config.map_cohere_rerank_params( + non_default_params=None, + model=self.model, + drop_params=False, + query="test query", + documents=["doc1", "doc2"], + ) + assert "instruction" not in params + + def test_map_cohere_rerank_params_passes_instruction_when_set(self): + params = self.config.map_cohere_rerank_params( + non_default_params=None, + model=self.model, + drop_params=False, + query="test query", + documents=["doc1", "doc2"], + instruction="Rank by relevance to genomics", + ) + assert params["instruction"] == "Rank by relevance to genomics" + + def test_transform_request_includes_instruction_when_set(self): + body = self.config.transform_rerank_request( + model=self.model, + optional_rerank_params={ + "query": "test query", + "documents": ["doc1", "doc2"], + "instruction": "Rank by relevance to genomics", + }, + headers={}, + ) + assert body["instruction"] == "Rank by relevance to genomics" + + def test_transform_request_omits_instruction_when_absent(self): + # exclude_none must drop the field entirely so the body matches the + # pre-existing (instruction-less) shape exactly. + body = self.config.transform_rerank_request( + model=self.model, + optional_rerank_params={ + "query": "test query", + "documents": ["doc1", "doc2"], + }, + headers={}, + ) + assert "instruction" not in body + def test_map_cohere_rerank_params_raises_on_max_chunks_per_doc(self): with pytest.raises( ValueError, match="Hosted VLLM does not support max_chunks_per_doc" @@ -74,6 +123,7 @@ class TestHostedVLLMRerankTransform: } result = self.config._transform_response(response_dict) assert result.id == "abc123" + assert result.results is not None assert len(result.results) == 2 assert result.results[0]["index"] == 0 assert result.results[0]["relevance_score"] == 0.9 @@ -94,3 +144,32 @@ class TestHostedVLLMRerankTransform: } with pytest.raises(ValueError, match="Missing required fields in the result="): self.config._transform_response(response_dict) + + +class TestGetOptionalRerankParamsInstruction: + """`instruction` is threaded through get_optional_rerank_params only when set.""" + + def setup_method(self): + self.config = HostedVLLMRerankConfig() + self.model = "hosted-vllm-model" + + def test_instruction_threaded_when_set(self): + params = get_optional_rerank_params( + rerank_provider_config=self.config, + model=self.model, + drop_params=False, + query="test query", + documents=["doc1", "doc2"], + instruction="Rank by relevance to genomics", + ) + assert params["instruction"] == "Rank by relevance to genomics" + + def test_instruction_absent_when_not_set(self): + params = get_optional_rerank_params( + rerank_provider_config=self.config, + model=self.model, + drop_params=False, + query="test query", + documents=["doc1", "doc2"], + ) + assert "instruction" not in params diff --git a/tests/test_litellm/llms/moonshot/test_moonshot_chat_transformation.py b/tests/test_litellm/llms/moonshot/test_moonshot_chat_transformation.py index 95ade4290e9..417dd4a767c 100644 --- a/tests/test_litellm/llms/moonshot/test_moonshot_chat_transformation.py +++ b/tests/test_litellm/llms/moonshot/test_moonshot_chat_transformation.py @@ -305,6 +305,34 @@ class TestMoonshotConfig: assert len(result["messages"]) == 2 assert result["messages"][1]["content"] == "Please select a tool to handle the current issue." + def test_tool_choice_required_does_not_mutate_input_messages(self): + """tool_choice='required' must not mutate the caller's messages list. + + The handling appends a "select a tool" user message; building it in + place corrupts the caller's conversation history and makes + transform_request non-idempotent across retries. + """ + config = MoonshotChatConfig() + + messages = [{"role": "user", "content": "What's the weather like?"}] + + for _ in range(2): + optional_params = { + "tool_choice": "required", + "tools": [{"type": "function", "function": {"name": "get_weather"}}], + } + result = config.transform_request( + model="moonshot-v1-8k", + messages=messages, + optional_params=optional_params, + litellm_params={}, + headers={}, + ) + # The returned request carries the extra message. + assert len(result["messages"]) == 2 + # The caller's list is untouched, so repeated calls stay idempotent. + assert messages == [{"role": "user", "content": "What's the weather like?"}] + def test_tool_choice_non_required_preserved(self): """Test that non-'required' tool_choice values are preserved""" config = MoonshotChatConfig() diff --git a/tests/test_litellm/llms/openai/transcriptions/test_transcription_duration_hidden.py b/tests/test_litellm/llms/openai/transcriptions/test_transcription_duration_hidden.py index 2b287e456a1..703fa13cbc9 100644 --- a/tests/test_litellm/llms/openai/transcriptions/test_transcription_duration_hidden.py +++ b/tests/test_litellm/llms/openai/transcriptions/test_transcription_duration_hidden.py @@ -13,7 +13,52 @@ from litellm.cost_calculator import completion_cost from litellm.litellm_core_utils.llm_response_utils.convert_dict_to_response import ( convert_to_model_response_object, ) -from litellm.types.utils import TranscriptionResponse +from litellm.types.utils import ( + TranscriptionResponse, + TranscriptionUsageDurationObject, +) + + +class TestDiarizedJsonUsageParsing: + """gpt-4o-transcribe / diarized_json returns a fractional `usage.seconds`.""" + + def test_fractional_duration_seconds_does_not_raise(self): + """ + A diarized_json response carries usage={"type": "duration", "seconds": }. + OpenAI specs `seconds` as a float, so a fractional value must parse cleanly + instead of raising and getting retried until the upstream rate-limits. + """ + response_object = { + "text": "speaker_1: Olá", + "task": "transcribe", + "duration": 295.8, + "segments": [ + { + "id": "seg_001", + "speaker": "speaker_1", + "start": 0.0, + "end": 1.0, + "text": "Olá", + "type": "transcript.text.segment", + } + ], + "usage": {"type": "duration", "seconds": 295.8}, + } + + result = convert_to_model_response_object( + response_object=response_object, + model_response_object=TranscriptionResponse(), + response_type="audio_transcription", + ) + + assert isinstance(result.usage, TranscriptionUsageDurationObject) + assert result.usage.seconds == 295.8 + + def test_usage_duration_object_accepts_float_seconds(self): + assert ( + TranscriptionUsageDurationObject(type="duration", seconds=295.8).seconds + == 295.8 + ) class TestTranscriptionDurationNotInResponseBody: diff --git a/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py b/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py index 6d0b4509b45..de5250fc5f3 100644 --- a/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py +++ b/tests/test_litellm/proxy/spend_tracking/test_spend_management_endpoints.py @@ -3787,3 +3787,298 @@ async def test_ui_view_spend_logs_metadata_invalid_json_falls_back_to_empty_dict assert body["data"][0]["metadata"] == {} finally: app.dependency_overrides.pop(ps.user_api_key_auth, None) + + +class _FakeColdStorageLogger: + """Injectable cold storage logger that records the object key it was asked for.""" + + def __init__(self, payload): + self._payload = payload + self.requested_object_keys = [] + + async def get_proxy_server_request_from_cold_storage_with_object_key( + self, object_key + ): + self.requested_object_keys.append(object_key) + return self._payload + + +def _cold_storage_handler(payload): + from litellm.proxy.spend_tracking.cold_storage_handler import ColdStorageHandler + + logger = _FakeColdStorageLogger(payload) + return ColdStorageHandler(cold_storage_logger=logger), logger + + +@pytest.mark.parametrize( + "value, expected", + [ + (None, False), + ("", False), + (" ", False), + ("{}", False), + ("[]", False), + ("null", False), + ('{"a": 1}', True), + ({}, False), + ({"a": 1}, True), + ([], False), + ([1], True), + (5, True), + ], +) +def test_spend_log_field_has_content(value, expected): + assert spend_management_endpoints._spend_log_field_has_content(value) is expected + + +@pytest.mark.parametrize( + "metadata, expected", + [ + (None, None), + ("{}", None), + ("not-json", None), + ({"cold_storage_object_key": ""}, None), + ({"cold_storage_object_key": "k/req-1.json"}, "k/req-1.json"), + ('{"cold_storage_object_key": "k/req-2.json"}', "k/req-2.json"), + ], +) +def test_cold_storage_object_key_from_metadata(metadata, expected): + assert ( + spend_management_endpoints._cold_storage_object_key_from_metadata(metadata) + == expected + ) + + +@pytest.mark.asyncio +async def test_resolve_payload_prefers_pg_and_skips_cold_storage(): + handler, logger = _cold_storage_handler({"messages": "X", "response": "Y"}) + row = { + "messages": "{}", + "response": '{"choices": [{"message": {"content": "hi"}}]}', + "proxy_server_request": "{}", + "metadata": {"cold_storage_object_key": "k/req.json"}, + } + + resolved = await spend_management_endpoints._resolve_request_response_payload( + row, cold_storage_handler=handler + ) + + assert resolved.response == '{"choices": [{"message": {"content": "hi"}}]}' + assert logger.requested_object_keys == [] + + +@pytest.mark.asyncio +async def test_resolve_payload_fetches_from_cold_storage_when_pg_empty(): + cold_payload = { + "messages": [{"role": "user", "content": "what is 2+2"}], + "response": {"choices": [{"message": {"content": "4"}}]}, + "proxy_server_request": {"body": {"model": "gpt-4o-mini"}}, + } + handler, logger = _cold_storage_handler(cold_payload) + row = { + "messages": "{}", + "response": "{}", + "proxy_server_request": "{}", + "metadata": {"cold_storage_object_key": "llm-gateway/prod/req-42.json"}, + } + + resolved = await spend_management_endpoints._resolve_request_response_payload( + row, cold_storage_handler=handler + ) + + assert logger.requested_object_keys == ["llm-gateway/prod/req-42.json"] + assert resolved.messages == cold_payload["messages"] + assert resolved.response == cold_payload["response"] + assert resolved.proxy_server_request == cold_payload["proxy_server_request"] + + +@pytest.mark.asyncio +async def test_resolve_payload_metadata_as_json_string(): + cold_payload = {"messages": "in", "response": "out", "proxy_server_request": None} + handler, logger = _cold_storage_handler(cold_payload) + row = { + "messages": "{}", + "response": "{}", + "proxy_server_request": "{}", + "metadata": json.dumps({"cold_storage_object_key": "k/str-meta.json"}), + } + + resolved = await spend_management_endpoints._resolve_request_response_payload( + row, cold_storage_handler=handler + ) + + assert logger.requested_object_keys == ["k/str-meta.json"] + assert resolved.response == "out" + + +@pytest.mark.asyncio +async def test_resolve_payload_no_object_key_returns_empty_without_fetch(): + handler, logger = _cold_storage_handler({"messages": "should-not-be-used"}) + row = { + "messages": "{}", + "response": "{}", + "proxy_server_request": "{}", + "metadata": {}, + } + + resolved = await spend_management_endpoints._resolve_request_response_payload( + row, cold_storage_handler=handler + ) + + assert logger.requested_object_keys == [] + assert resolved == spend_management_endpoints.RequestResponsePayload( + "{}", "{}", "{}" + ) + + +@pytest.mark.asyncio +async def test_resolve_payload_cold_storage_miss_falls_back_to_pg_values(): + handler, logger = _cold_storage_handler(None) + row = { + "messages": "{}", + "response": "{}", + "proxy_server_request": "{}", + "metadata": {"cold_storage_object_key": "k/missing.json"}, + } + + resolved = await spend_management_endpoints._resolve_request_response_payload( + row, cold_storage_handler=handler + ) + + assert logger.requested_object_keys == ["k/missing.json"] + assert resolved == spend_management_endpoints.RequestResponsePayload( + "{}", "{}", "{}" + ) + + +@pytest.mark.asyncio +async def test_resolve_payload_cold_storage_exception_falls_back_to_pg_values(): + """A backend error during fetch degrades to PG values instead of bubbling a 500.""" + + class _RaisingLogger: + async def get_proxy_server_request_from_cold_storage_with_object_key( + self, object_key + ): + raise RuntimeError("cold storage backend unavailable") + + from litellm.proxy.spend_tracking.cold_storage_handler import ColdStorageHandler + + handler = ColdStorageHandler(cold_storage_logger=_RaisingLogger()) + row = { + "messages": "{}", + "response": "{}", + "proxy_server_request": "{}", + "metadata": {"cold_storage_object_key": "k/boom.json"}, + } + + resolved = await spend_management_endpoints._resolve_request_response_payload( + row, cold_storage_handler=handler + ) + + assert resolved == spend_management_endpoints.RequestResponsePayload( + "{}", "{}", "{}" + ) + + +@pytest.mark.asyncio +async def test_cold_storage_handler_uses_injected_logger(): + from litellm.proxy.spend_tracking.cold_storage_handler import ColdStorageHandler + + logger = _FakeColdStorageLogger({"messages": "in", "response": "out"}) + handler = ColdStorageHandler(cold_storage_logger=logger) + + result = await handler.get_proxy_server_request_from_cold_storage_with_object_key( + object_key="k/req.json" + ) + + assert result == {"messages": "in", "response": "out"} + assert logger.requested_object_keys == ["k/req.json"] + + +@pytest.mark.asyncio +async def test_cold_storage_handler_returns_none_when_no_logger_configured(monkeypatch): + from litellm.proxy.spend_tracking.cold_storage_handler import ColdStorageHandler + + monkeypatch.setattr(litellm, "cold_storage_custom_logger", None, raising=False) + handler = ColdStorageHandler() + + result = await handler.get_proxy_server_request_from_cold_storage_with_object_key( + object_key="k/req.json" + ) + + assert result is None + + +@pytest.mark.asyncio +async def test_cold_storage_handler_resolves_configured_logger_from_registry(monkeypatch): + from litellm.proxy.spend_tracking.cold_storage_handler import ColdStorageHandler + + logger = _FakeColdStorageLogger({"messages": "from-registry"}) + monkeypatch.setattr(litellm, "cold_storage_custom_logger", "s3_v2", raising=False) + monkeypatch.setattr( + litellm.logging_callback_manager, + "get_active_custom_logger_for_callback_name", + lambda name: logger if name == "s3_v2" else None, + ) + handler = ColdStorageHandler() + + result = await handler.get_proxy_server_request_from_cold_storage_with_object_key( + object_key="k/req.json" + ) + + assert result == {"messages": "from-registry"} + assert logger.requested_object_keys == ["k/req.json"] + + +def test_ui_view_request_response_reads_from_cold_storage(client, monkeypatch): + """End-to-end: a placeholder row with a cold_storage_object_key is served from + cold storage through the detail endpoint.""" + from types import SimpleNamespace + + placeholder_row = { + "messages": "{}", + "response": "{}", + "proxy_server_request": "{}", + "metadata": {"cold_storage_object_key": "k/cold.json"}, + } + + async def _query_raw(_sql, *_args): + return [placeholder_row] + + fake_prisma = SimpleNamespace(db=SimpleNamespace(query_raw=_query_raw)) + monkeypatch.setattr("litellm.proxy.proxy_server.prisma_client", fake_prisma) + + cold_logger = _FakeColdStorageLogger( + { + "messages": [{"role": "user", "content": "hi"}], + "response": {"choices": [{"message": {"content": "hello"}}]}, + "proxy_server_request": None, + } + ) + monkeypatch.setattr(litellm, "cold_storage_custom_logger", "s3_v2", raising=False) + monkeypatch.setattr( + litellm.logging_callback_manager, + "get_active_additional_logging_utils_from_custom_logger", + lambda: [], + ) + monkeypatch.setattr( + litellm.logging_callback_manager, + "get_active_custom_logger_for_callback_name", + lambda name: cold_logger if name == "s3_v2" else None, + ) + + app.dependency_overrides[ps.user_api_key_auth] = lambda: UserAPIKeyAuth( + user_role=LitellmUserRoles.PROXY_ADMIN, user_id="admin_1" + ) + try: + response = client.get( + "/spend/logs/ui/req-cold", + headers={"Authorization": "Bearer sk-test"}, + ) + assert response.status_code == 200 + body = response.json() + assert body["messages"] == [{"role": "user", "content": "hi"}] + assert body["response"] == {"choices": [{"message": {"content": "hello"}}]} + assert cold_logger.requested_object_keys == ["k/cold.json"] + finally: + app.dependency_overrides.pop(ps.user_api_key_auth, None) diff --git a/tests/test_litellm/router_strategy/test_lar1_routing.py b/tests/test_litellm/router_strategy/test_lar1_routing.py new file mode 100644 index 00000000000..4710bbb6896 --- /dev/null +++ b/tests/test_litellm/router_strategy/test_lar1_routing.py @@ -0,0 +1,460 @@ +import pytest +from unittest.mock import AsyncMock + +from litellm import Router +from litellm.router_strategy.lar1_routing import ( + LAR1RoutingStrategy, + _normalize_thresholds, + _parse_lar1_metadata, + apply_lar1_routing_strategy, + lar1_thresholds_from_args, + DEFAULT_THRESHOLDS, +) + + +def _model_list(): + return [ + { + "model_name": "agent-router", + "litellm_params": { + "model": "openai/gpt-4o", + "api_key": "fake-key", + }, + "model_info": {"id": "cloud-smart", "type": "cloud-smart"}, + }, + { + "model_name": "agent-router", + "litellm_params": { + "model": "openai/gpt-4o-mini", + "api_key": "fake-key", + }, + "model_info": {"id": "cloud-fast", "type": "cloud-fast"}, + }, + { + "model_name": "agent-router", + "litellm_params": { + "model": "ollama/qwythos", + "api_key": "fake-key", + }, + "model_info": {"id": "local", "type": "local"}, + }, + { + "model_name": "agent-router", + "litellm_params": { + "model": "ollama/mythos", + "api_key": "fake-key", + }, + "model_info": {"id": "deep", "type": "deep"}, + }, + ] + + +def _create_test_router(): + return Router(model_list=_model_list()) + + +@pytest.mark.asyncio +async def test_low_confidence_routes_to_cloud_smart(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.2}}}, + ) + assert result["model_info"]["type"] == "cloud-smart" + + +@pytest.mark.asyncio +async def test_high_confidence_routes_to_deep(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.8}}}, + ) + assert result["model_info"]["type"] == "deep" + + +@pytest.mark.asyncio +async def test_unverified_evidence_fallback(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={ + "metadata": { + "lar1": {"confidence": 0.9, "evidence": ["UNVERIFIED"]}, + } + }, + ) + assert result["model_info"]["type"] == "cloud-smart" + + +@pytest.mark.asyncio +async def test_custom_thresholds(): + router = _create_test_router() + custom_strategy = LAR1RoutingStrategy( + router, + thresholds={"low": 0.1, "medium": 0.3, "high": 0.9}, + ) + + result = await custom_strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.85}}}, + ) + assert result["model_info"]["type"] == "local" + + +@pytest.mark.asyncio +async def test_mem_time_routes_to_cloud_fast(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={ + "metadata": { + "lar1": {"confidence": 0.9, "time": "MEM"}, + } + }, + ) + assert result["model_info"]["type"] == "cloud-fast" + + +@pytest.mark.asyncio +async def test_invalid_confidence_falls_back_to_local(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={ + "metadata": {"lar1": {"confidence": "not-a-number"}}, + }, + ) + assert result["model_info"]["type"] == "local" + + +@pytest.mark.asyncio +async def test_no_metadata_defaults_to_local(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={}, + ) + assert result["model_info"]["type"] == "local" + + +@pytest.mark.asyncio +async def test_router_init_with_lar1_routing_strategy(): + router = Router( + model_list=_model_list(), + routing_strategy="lar1", + routing_strategy_args={ + "confidence_threshold_low": 0.1, + "confidence_threshold_medium": 0.3, + "confidence_threshold_high": 0.9, + }, + ) + + result = await router.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.85}}}, + ) + assert result["model_info"]["type"] == "local" + assert router.routing_strategy == "lar1" + + +@pytest.mark.asyncio +async def test_router_init_lar1_default_thresholds(): + router = Router(model_list=_model_list(), routing_strategy="lar1") + + result = await router.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.4}}}, + ) + assert result["model_info"]["type"] == "cloud-fast" + assert router.routing_strategy == "lar1" + + +@pytest.mark.asyncio +async def test_mid_confidence_routes_to_cloud_fast(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.4}}}, + ) + assert result["model_info"]["type"] == "cloud-fast" + + +@pytest.mark.asyncio +async def test_no_router_returns_none(): + strategy = LAR1RoutingStrategy() + + result = await strategy.async_get_available_deployment(model="agent-router") + assert result is None + + +@pytest.mark.asyncio +async def test_request_kwargs_none_uses_defaults(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs=None, + ) + assert result["model_info"]["type"] == "local" + + +@pytest.mark.asyncio +async def test_invalid_lar1_metadata_type_uses_defaults(caplog): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + with caplog.at_level("WARNING"): + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": "not-a-dict"}}, + ) + + assert result["model_info"]["type"] == "local" + assert "Invalid lar1 metadata type" in caplog.text + + +@pytest.mark.asyncio +async def test_confirmed_evidence_routes_by_confidence(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={ + "metadata": { + "lar1": {"confidence": 0.8, "evidence": ["CONFIRMED"]}, + } + }, + ) + assert result["model_info"]["type"] == "deep" + + +@pytest.mark.asyncio +async def test_empty_healthy_deployments_returns_none(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + router.async_get_healthy_deployments = AsyncMock(return_value=[]) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.8}}}, + ) + assert result is None + + +@pytest.mark.asyncio +async def test_specific_deployment_dict_short_circuit(): + deployment = { + "model_info": {"type": "deep"}, + "litellm_params": {"model": "ollama/mythos"}, + } + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + router.async_get_healthy_deployments = AsyncMock(return_value=deployment) + + result = await strategy.async_get_available_deployment( + model="agent-router", + specific_deployment=True, + request_kwargs={}, + ) + assert result == deployment + + +@pytest.mark.asyncio +async def test_non_dict_healthy_deployment_returns_none(): + router = _create_test_router() + strategy = LAR1RoutingStrategy(router) + router.async_get_healthy_deployments = AsyncMock(return_value=[None]) + + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={}, + ) + assert result is None + + +def test_parse_lar1_metadata_defaults_when_missing(): + metadata = _parse_lar1_metadata({}) + assert metadata.confidence == 0.5 + assert metadata.time.value == "NOW" + + +def test_select_deployment_empty_list(): + strategy = LAR1RoutingStrategy() + selected, exact_match = strategy._select_deployment("local", []) + assert selected is None + assert exact_match is False + + +def test_select_deployment_skips_non_dict_entries(): + strategy = LAR1RoutingStrategy() + deployment = {"model_info": {"type": "local"}} + selected, exact_match = strategy._select_deployment( + "local", + ["skip-me", deployment], + ) + assert selected == deployment + assert exact_match is True + + +def test_select_deployment_fallback_uses_first_dict(): + strategy = LAR1RoutingStrategy() + deployment = {"model_info": {"type": "local"}} + selected, exact_match = strategy._select_deployment( + "cloud-smart", + ["skip-me", deployment], + ) + assert selected == deployment + assert exact_match is False + + +def test_select_deployment_all_non_dict_returns_none(): + strategy = LAR1RoutingStrategy() + selected, exact_match = strategy._select_deployment("local", ["a", None]) + assert selected is None + assert exact_match is False + + +def test_lar1_thresholds_from_args_uses_defaults(): + assert lar1_thresholds_from_args({}) == DEFAULT_THRESHOLDS + + +def test_lar1_thresholds_from_args_ignores_invalid_values(): + assert lar1_thresholds_from_args( + { + "confidence_threshold_low": "bad", + "confidence_threshold_medium": 0.4, + "confidence_threshold_high": 0.8, + } + ) == {"low": 0.3, "medium": 0.4, "high": 0.8} + + +@pytest.mark.asyncio +async def test_update_settings_switches_to_lar1_routing(): + router = Router(model_list=_model_list(), routing_strategy="simple-shuffle") + router.update_settings( + routing_strategy="lar1", + routing_strategy_args={ + "confidence_threshold_low": 0.1, + "confidence_threshold_medium": 0.3, + "confidence_threshold_high": 0.9, + }, + ) + + result = await router.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.85}}}, + ) + assert router.routing_strategy == "lar1" + assert result["model_info"]["type"] == "local" + + +def test_update_settings_switching_from_lar1_restores_default_selectors(): + router = Router(model_list=_model_list(), routing_strategy="lar1") + + router.update_settings(routing_strategy="simple-shuffle") + + assert router.routing_strategy == "simple-shuffle" + assert "get_available_deployment" not in router.__dict__ + assert "async_get_available_deployment" not in router.__dict__ + result = router.get_available_deployment(model="agent-router") + assert result["model_name"] == "agent-router" + + +@pytest.mark.asyncio +async def test_update_settings_routing_strategy_args_relinks_lar1(): + router = Router(model_list=_model_list(), routing_strategy="lar1") + + router.update_settings( + routing_strategy_args={ + "confidence_threshold_low": 0.1, + "confidence_threshold_medium": 0.3, + "confidence_threshold_high": 0.9, + }, + ) + + result = await router.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.85}}}, + ) + assert result["model_info"]["type"] == "local" + + +def test_apply_lar1_routing_strategy_wires_custom_selector(): + router = Router(model_list=_model_list(), routing_strategy="simple-shuffle") + apply_lar1_routing_strategy(router, {"confidence_threshold_high": 0.9}) + assert router.routing_strategy == "lar1" + with pytest.raises(NotImplementedError, match="async routing"): + router.get_available_deployment(model="agent-router") + + +def test_apply_lar1_invalid_thresholds_leaves_router_unchanged(): + router = Router(model_list=_model_list(), routing_strategy="simple-shuffle") + + with pytest.raises(ValueError, match="LAR-1 thresholds must satisfy"): + apply_lar1_routing_strategy( + router, + { + "confidence_threshold_low": 0.9, + "confidence_threshold_medium": 0.5, + "confidence_threshold_high": 0.7, + }, + ) + + assert router.routing_strategy == "simple-shuffle" + assert "async_get_available_deployment" not in router.__dict__ + result = router.get_available_deployment(model="agent-router") + assert result["model_name"] == "agent-router" + + +def test_invalid_threshold_order_raises(): + with pytest.raises(ValueError, match="LAR-1 thresholds must satisfy"): + _normalize_thresholds({"low": 0.5, "medium": 0.3, "high": 0.7}) + + +def test_get_available_deployment_raises_not_implemented(): + strategy = LAR1RoutingStrategy() + with pytest.raises(NotImplementedError, match="async routing"): + strategy.get_available_deployment(model="agent-router") + + +@pytest.mark.asyncio +async def test_missing_target_type_falls_back_with_warning(caplog): + router = Router( + model_list=[ + { + "model_name": "agent-router", + "litellm_params": { + "model": "openai/gpt-4o", + "api_key": "fake-key", + }, + "model_info": {"id": "only-local", "type": "local"}, + } + ] + ) + strategy = LAR1RoutingStrategy(router) + + with caplog.at_level("WARNING"): + result = await strategy.async_get_available_deployment( + model="agent-router", + request_kwargs={"metadata": {"lar1": {"confidence": 0.2}}}, + ) + + assert result["model_info"]["type"] == "local" + assert "No deployment for type 'cloud-smart'" in caplog.text diff --git a/tests/test_litellm/router_utils/test_fallback_event_handlers.py b/tests/test_litellm/router_utils/test_fallback_event_handlers.py index ca647bdce55..98a34de295c 100644 --- a/tests/test_litellm/router_utils/test_fallback_event_handlers.py +++ b/tests/test_litellm/router_utils/test_fallback_event_handlers.py @@ -2,7 +2,10 @@ import json import pytest -from litellm.router_utils.fallback_event_handlers import run_async_fallback +from litellm.router_utils.fallback_event_handlers import ( + get_fallback_model_group, + run_async_fallback, +) class StreamingWrapper: @@ -137,3 +140,16 @@ async def test_run_async_fallback_skips_original_model_group(): ) assert response._hidden_params["additional_headers"]["x-litellm-attempted-fallbacks"] == 1 + + +def test_get_fallback_model_group_does_not_mutate_fallbacks(): + """A string fallback must be resolved without mutating the caller's + fallbacks list, which is the live router config shared across requests.""" + fallbacks = [{"gpt-3.5-turbo": ["claude-3-haiku"]}, "gpt-4o-mini"] + + fallback_model_group, _ = get_fallback_model_group( + fallbacks=fallbacks, model_group="unmatched-model" + ) + + assert fallback_model_group == ["gpt-4o-mini"] + assert fallbacks == [{"gpt-3.5-turbo": ["claude-3-haiku"]}, "gpt-4o-mini"] diff --git a/tests/test_litellm/test_gpt_image_cost_calculator.py b/tests/test_litellm/test_gpt_image_cost_calculator.py index 6644b1389cf..c371f7442be 100644 --- a/tests/test_litellm/test_gpt_image_cost_calculator.py +++ b/tests/test_litellm/test_gpt_image_cost_calculator.py @@ -381,5 +381,93 @@ class TestCompletionCostIntegration: assert abs(cost - expected_cost) < 1e-6, f"Expected {expected_cost}, got {cost}" +class TestGPTImage2OutputImageTokensNoBreakdown: + """ + Regression test: the OpenAI Images endpoints (/v1/images/generations and + /v1/images/edits) return usage with NO output token breakdown — litellm's + ImageUsage has no ``output_tokens_details`` field. Before the fix, the + generated-image OUTPUT tokens were priced at the text rate + (``output_cost_per_token`` = $10/1M for gpt-image-2) instead of the image rate + (``output_cost_per_image_token`` = $30/1M), a ~3x undercount on the dominant + cost component. + """ + + def test_gpt_image_2_output_priced_as_image_when_no_breakdown(self): + from litellm.llms.openai.image_generation.cost_calculator import ( + cost_calculator, + ) + + # Mirrors a real gpt-image-2 /v1/images/edits response: input breakdown is + # present, but there is no usable output token breakdown. + usage = ImageUsage( + input_tokens=3987, + output_tokens=5488, + total_tokens=9475, + input_tokens_details=ImageUsageInputTokensDetails( + text_tokens=943, + image_tokens=3044, + ), + ) + + image_response = ImageResponse( + created=1234567890, + data=[ImageObject(b64_json="test")], + ) + image_response.usage = usage + image_response._hidden_params = {"custom_llm_provider": "openai"} + + cost = cost_calculator( + model="gpt-image-2", + image_response=image_response, + custom_llm_provider="openai", + ) + + # gpt-image-2 pricing: + # text input: 943 * $5/1M = 0.004715 + # image input: 3044 * $8/1M = 0.024352 + # image output: 5488 * $30/1M = 0.164640 (NOT text output $10/1M = 0.054880) + expected_cost = 943 * 5e-6 + 3044 * 8e-6 + 5488 * 3e-5 + assert abs(cost - expected_cost) < 1e-6, ( + f"Expected {expected_cost}, got {cost}. Generated image output tokens " + f"are likely being priced at the text output_cost_per_token rate." + ) + + def test_gpt_image_2_chat_usage_without_breakdown_is_costed_not_zero(self): + """A chat ``Usage`` with ``completion_tokens_details=None`` must still be + costed via ``generic_cost_per_token`` (output at the text rate) rather than + erroring or silently returning 0.0.""" + from litellm.llms.openai.image_generation.cost_calculator import ( + cost_calculator, + ) + + usage = Usage( + prompt_tokens=600, + completion_tokens=5000, + total_tokens=5600, + prompt_tokens_details=PromptTokensDetailsWrapper( + text_tokens=100, + image_tokens=500, + ), + ) + + image_response = ImageResponse( + created=1234567890, + data=[ImageObject(b64_json="test")], + ) + image_response.usage = usage + image_response._hidden_params = {"custom_llm_provider": "openai"} + + cost = cost_calculator( + model="gpt-image-2", + image_response=image_response, + custom_llm_provider="openai", + ) + + # No output breakdown -> output priced at the text rate (output_cost_per_token): + # text in 100*$5/1M + image in 500*$8/1M + output 5000*$10/1M + expected_cost = 100 * 5e-6 + 500 * 8e-6 + 5000 * 1e-5 + assert abs(cost - expected_cost) < 1e-6, f"Expected {expected_cost}, got {cost}" + + if __name__ == "__main__": pytest.main([__file__, "-v"]) diff --git a/ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx b/ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx index 4048c0f1c4a..a1fb3ec8bb4 100644 --- a/ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx +++ b/ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx @@ -1,6 +1,9 @@ export function valueFormatter(number: number) { - if (number >= 1000000) { - return (number / 1000000).toFixed(2) + "M"; + if (number >= 1_000_000_000) { + return (number / 1_000_000_000).toFixed(2) + "B"; + } + if (number >= 1_000_000) { + return (number / 1_000_000).toFixed(2) + "M"; } if (number >= 1000) { return number / 1000 + "k"; @@ -10,8 +13,11 @@ export function valueFormatter(number: number) { export function valueFormatterSpend(number: number) { if (number === 0) return "$0"; - if (number >= 1000000) { - return "$" + number / 1000000 + "M"; + if (number >= 1_000_000_000) { + return "$" + parseFloat((number / 1_000_000_000).toFixed(2)) + "B"; + } + if (number >= 1_000_000) { + return "$" + parseFloat((number / 1_000_000).toFixed(2)) + "M"; } if (number >= 1000) { return "$" + number / 1000 + "k"; diff --git a/ui/litellm-dashboard/src/components/VirtualKeysPage/VirtualKeysTable.tsx b/ui/litellm-dashboard/src/components/VirtualKeysPage/VirtualKeysTable.tsx index 299f8a05f71..0c95d5dcfb7 100644 --- a/ui/litellm-dashboard/src/components/VirtualKeysPage/VirtualKeysTable.tsx +++ b/ui/litellm-dashboard/src/components/VirtualKeysPage/VirtualKeysTable.tsx @@ -467,10 +467,15 @@ export function VirtualKeysTable({ teams, organizations, onSortChange, currentSo enableSorting: true, cell: (info) => { const maxBudget = info.getValue() as number | null; - if (maxBudget === null) { - return "Unlimited"; + if (maxBudget !== null) { + return `$${formatNumberWithCommas(maxBudget)}`; } - return `$${formatNumberWithCommas(maxBudget)}`; + const teamId = info.row.original.team_id; + const team = teams?.find((t) => t.team_id === teamId); + if (team?.max_budget != null) { + return `$${formatNumberWithCommas(team.max_budget)} (Team)`; + } + return "Unlimited"; }, }, { diff --git a/ui/litellm-dashboard/src/components/activity_metrics.tsx b/ui/litellm-dashboard/src/components/activity_metrics.tsx index bd16c2f7219..9c7b62d78d1 100644 --- a/ui/litellm-dashboard/src/components/activity_metrics.tsx +++ b/ui/litellm-dashboard/src/components/activity_metrics.tsx @@ -295,6 +295,7 @@ export const ActivityMetrics: React.FC = ({ modelMetrics, valueFormatter={valueFormatter} customTooltip={CustomTooltip} showLegend={false} + yAxisWidth={80} /> @@ -311,9 +312,10 @@ export const ActivityMetrics: React.FC = ({ modelMetrics, index="date" categories={["metrics.successful_requests", "metrics.failed_requests"]} colors={["emerald", "red"]} - valueFormatter={(number: number) => number.toLocaleString()} + valueFormatter={valueFormatter} customTooltip={CustomTooltip} showLegend={false} + yAxisWidth={80} /> diff --git a/ui/litellm-dashboard/src/components/templates/key_info_view.budget_display.test.tsx b/ui/litellm-dashboard/src/components/templates/key_info_view.budget_display.test.tsx index 77abde3d870..5407d37fcf1 100644 --- a/ui/litellm-dashboard/src/components/templates/key_info_view.budget_display.test.tsx +++ b/ui/litellm-dashboard/src/components/templates/key_info_view.budget_display.test.tsx @@ -1,7 +1,7 @@ import { renderWithProviders } from "../../../tests/test-utils"; import { screen, waitFor } from "@testing-library/react"; import { beforeEach, describe, expect, it, vi } from "vitest"; -import { KeyResponse } from "../key_team_helpers/key_list"; +import { KeyResponse, Team } from "../key_team_helpers/key_list"; import KeyInfoView from "./key_info_view"; import useAuthorized from "@/app/(dashboard)/hooks/useAuthorized"; import useTeams from "@/app/(dashboard)/hooks/useTeams"; @@ -103,8 +103,26 @@ const baseAuthorized = { userEmail: null, disabledPersonalKeyCreation: null, showSSOBanner: false, + isLoading: false, + isAuthorized: true, }; +const makeTeam = (overrides: Partial): Team => ({ + team_id: "team-default", + team_alias: "Default Team", + models: [], + max_budget: null, + budget_duration: null, + tpm_limit: null, + rpm_limit: null, + organization_id: "", + created_at: "2026-01-01T00:00:00Z", + keys: [], + members_with_roles: [], + spend: 0, + ...overrides, +}); + describe("KeyInfoView overview budget display (LIT-2845)", () => { beforeEach(() => { vi.mocked(useTeams).mockReturnValue({ teams: [], setTeams: vi.fn() }); @@ -151,7 +169,64 @@ describe("KeyInfoView overview budget display (LIT-2845)", () => { it("renders 'Unlimited' when max_budget is null", async () => { renderWithProviders( {}} + keyId={"test-key-id"} + onKeyDataUpdate={() => {}} + teams={[]} + />, + ); + await waitFor(() => { + expect(screen.getByText(/of Unlimited/)).toBeInTheDocument(); + }); + }); + + it("renders team budget with alias and duration when key has no own budget but team has one", async () => { + vi.mocked(useTeams).mockReturnValue({ + teams: [makeTeam({ team_id: "team-123", team_alias: "Test Budget", max_budget: 1200, budget_duration: "30d" })], + setTeams: vi.fn(), + }); + renderWithProviders( + {}} + keyId={"test-key-id"} + onKeyDataUpdate={() => {}} + teams={[]} + />, + ); + await waitFor(() => { + expect(screen.getByText(/of \$1,200\.00 \(Team: Test Budget \/ 30d\)/)).toBeInTheDocument(); + }); + }); + + it("renders team budget without duration when team has no budget_duration", async () => { + vi.mocked(useTeams).mockReturnValue({ + teams: [makeTeam({ team_id: "team-456", team_alias: "No Duration Team", max_budget: 500 })], + setTeams: vi.fn(), + }); + renderWithProviders( + {}} + keyId={"test-key-id"} + onKeyDataUpdate={() => {}} + teams={[]} + />, + ); + await waitFor(() => { + expect(screen.getByText(/of \$500\.00 \(Team: No Duration Team\)/)).toBeInTheDocument(); + }); + }); + + it("renders 'Unlimited' when key has no budget and team also has no budget", async () => { + vi.mocked(useTeams).mockReturnValue({ + teams: [makeTeam({ team_id: "team-789", team_alias: "Free Team" })], + setTeams: vi.fn(), + }); + renderWithProviders( + {}} keyId={"test-key-id"} onKeyDataUpdate={() => {}} diff --git a/ui/litellm-dashboard/src/components/templates/key_info_view.tsx b/ui/litellm-dashboard/src/components/templates/key_info_view.tsx index 018880b70aa..4244ae2d794 100644 --- a/ui/litellm-dashboard/src/components/templates/key_info_view.tsx +++ b/ui/litellm-dashboard/src/components/templates/key_info_view.tsx @@ -411,6 +411,15 @@ export default function KeyInfoView({ }); }; + const parentTeam = currentKeyData.team_id ? teamsData?.find((team) => team.team_id === currentKeyData.team_id) : null; + + const budgetDisplay = + currentKeyData.max_budget !== null + ? `$${formatNumberWithCommas(currentKeyData.max_budget, 2)}` + : parentTeam?.max_budget != null + ? `$${formatNumberWithCommas(parentTeam.max_budget, 2)} (Team: ${parentTeam.team_alias || parentTeam.team_id}${parentTeam.budget_duration ? ` / ${parentTeam.budget_duration}` : ""})` + : "Unlimited"; + return (
Spend
${formatNumberWithCommas(currentKeyData.spend, 4)} - - of{" "} - {currentKeyData.max_budget !== null - ? `$${formatNumberWithCommas(currentKeyData.max_budget, 2)}` - : "Unlimited"} - + of {budgetDisplay}
diff --git a/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/PrettyMessagesView.test.tsx b/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/PrettyMessagesView.test.tsx index b73dcafcdc3..e7295ed7a72 100644 --- a/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/PrettyMessagesView.test.tsx +++ b/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/PrettyMessagesView.test.tsx @@ -27,6 +27,17 @@ describe("PrettyMessagesView", () => { expect(screen.getByText("Hi there!")).toBeInTheDocument(); }); + it("renders input when request is a bare messages array (cold storage payload)", () => { + const request = [{ role: "user", content: "Write me a poem" }]; + const response = { + choices: [{ message: { role: "assistant", content: "A quiet moment." } }], + }; + + render(); + expect(screen.getByText("Write me a poem")).toBeInTheDocument(); + expect(screen.getByText("A quiet moment.")).toBeInTheDocument(); + }); + it("should render the realtime pretty view for realtime API responses", () => { const request = {}; const response = { diff --git a/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/prettyMessagesUtils.ts b/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/prettyMessagesUtils.ts index 32ae294b1ee..09b8f551c1d 100644 --- a/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/prettyMessagesUtils.ts +++ b/ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/prettyMessagesUtils.ts @@ -39,18 +39,24 @@ export const ROLE_STYLES: Record = { * Parse request messages and response message from log data */ export const parseMessages = (request: any, response: any): ParsedMessages => { - // Parse request messages + // Parse request messages. `request` is either the raw request body + // ({ messages: [...] }) or, when prompts come from cold storage, the bare + // messages array itself. const requestMessages: ParsedMessage[] = []; - if (request?.messages && Array.isArray(request.messages)) { - request.messages.forEach((msg: any) => { - requestMessages.push({ - role: msg.role || "user", - content: parseMessageContent(msg.content), - toolCallId: msg.tool_call_id, - }); + const requestMessageList = Array.isArray(request) + ? request + : Array.isArray(request?.messages) + ? request.messages + : []; + + requestMessageList.forEach((msg: any) => { + requestMessages.push({ + role: msg.role || "user", + content: parseMessageContent(msg.content), + toolCallId: msg.tool_call_id, }); - } + }); // Parse response message let responseMessage: ParsedMessage | null = null;