scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-07 07:58:09 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 150

Merge remote-tracking branch 'origin/main' into litellm_fallback_write_reads_router_settings_from_db

Browse source
This commit is contained in:
mateo-berri 2026-10-28 23:57:42 -07:00
commit 4908728464
160 changed files with 9950 additions and 1043 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
.github/PULL_REQUEST_TEMPLATE/general.md vendored
View file

@ -84,9 +84,10 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
## Screenshots / Proof of Fix
<!-- Include screenshots, screen recordings, or command (e.g., curl) + output demonstrating that your changes work as expected
The proof must be completely e2e with no mocks, using actual LLM calls costing real $$$ if applicable. `pytest` commands are not enough
Show ONLY the latest run: capture Before at the merge base and After at the PR's current tip, and when new commits change behavior, replace this whole section with the fresh run instead of stacking it on top of older ones. The run must be up to date. As soon as a new commit is made and it makes this PR description's after sha stale (it's no longer tip of PR), you must re-run the QA
Structure the section exactly as below: Before and After one heading level below this section, each naming the commit hash it was captured at, one lower-level heading per case inside each, the same case names in the same order on both sides, and numbered steps (command, observed output) under every case, never loose prose; shared setup (config, payloads) goes above Before, and with a single case, drop the case headings and number the steps directly
Simple PR exception (2026-10-28, #49030). A mechanical change fully proven by focused checks may skip before/after e2e. Follow AGENTS.md's eligibility rule, trace callers, and verify bug-fix regression sensitivity. Record E2E exception at <tip SHA>, scope, rationale, commands and results here. Tests/static checks suffice only for this exception. Keep CI and bot gates, reassess changed diffs, and honor explicit e2e requests. Omit the Before/After scaffold for an exception
Otherwise proof must be fully e2e, no mocks, with real provider calls where applicable. Pytest alone is insufficient. The following format applies to live proof
Show only the latest proof. Before uses the merge base and After the tested tip. Rerun affected QA after behavior changes. Verified neutral diffs may carry proof with the tested SHA, current tip, equivalence reason and focused checks, following AGENTS.md and /qa
Use matching Before/After case headings with numbered commands and observed output. Put shared setup above them. For one case, omit case headings
### Before (<hash>)

9
.grype.yaml
View file

@ -11,3 +11,12 @@ ignore:
package:
name: zlib
type: apk
- vulnerability: GHSA-v4x8-3wr5-3wq8
reason: >-
filelock ReadWriteLock advisory, and no package in the image takes ReadWriteLock;
the fix (4.2.1) is blocked by litellm's own filelock<4.0 cap until #48635 or #48802 lifts it,
and the image then moves off 3.32.7, which retires this version-scoped rule
package:
name: filelock
version: 3.32.7
type: python

6
AGENTS.md
View file

@ -31,6 +31,8 @@ A test must only fail when litellm code changes. Never pin facts we don't own (a
End-to-end tests belong in `tests/e2e/`. A rule about one directory's files, like its harness or which CI runs it, goes in that directory's `AGENTS.md`, never only in a README or here. Read a directory's `AGENTS.md` before editing files in it
Before you build a new way to move data between the endpoints, the provider transformations, and cost tracking or logging, read the architecture decision records in `docs/adrs/`, which say why those layers are shaped the way they are
When creating PRs, target the repository's current default branch for both internal and external / OSS contributions. Check it with `python3 scripts/default_branch.py --branch` instead of assuming a branch name or relying on cached `origin/HEAD`
When writing a PR body, treat the comments and imperative instructions inside .github/PULL_REQUEST_TEMPLATE/general.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/PULL_REQUEST_TEMPLATE/general.md from disk before writing a PR body to make sure you see every comment rule. A section you have nothing to put in (Relevant issues, Affected release, Linear ticket, Caveats, QA runbook, and so on) is removed entirely, heading included, never left as an empty title
@ -39,7 +41,9 @@ Same applies for filing bug reports and feature requests, with .github/ISSUE_TEM
If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just drop the section
Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`. The Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into. I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, or the main use case runs through a headful agentic coding tool like Claude Code or Codex, drive that surface yourself and embed your own before and after screenshots of it in the PR (the Admin UI page, or what the coding tool shows), next to an ordered list of the URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, and what fields to fill out so a reviewer can reproduce it
Simple PRs may skip before/after e2e when the full diff is mechanical and focused checks completely prove it, such as #49030 restoring an existing helper name. Trace affected callers and run relevant tests/static checks at the tip. For a bug fix, establish that the regression catches the original defect. Size alone never qualifies. New behavior, API/provider semantics, auth, billing, persistence, concurrency or uncertain integration risk need live e2e. In Screenshots / Proof of Fix, record `E2E exception at <tip SHA>`, scope, rationale, commands and results. Tests count as proof only for this exception. Reassess changed diffs, retain CI/bot gates, and honor explicit e2e requests (2026-10-28, unnecessary live QA for #49030)
For other PRs, proof must show the end user's real flow before and after, with commands and output against a live proxy and real provider calls where applicable. Pytest alone is insufficient. For UI or interactive coding-tool flows, drive that surface and embed before/after screenshots, with URLs, clicks and input fields a reviewer can reproduce
If you ever write any human-facing text (pull requests, issues, commit messages, discussion posts, github comments, release notes, docs, etc.), always follow these guidelines to sound less AI-y:
- don't use emojis

2
ARCHITECTURE.md
View file

@ -2,6 +2,8 @@
This document helps contributors understand where to make changes in LiteLLM.
For why a layer is shaped the way it is, read the architecture decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) (Python SDK and proxy) and [`litellm-rust/docs/adrs/`](litellm-rust/docs/adrs/AGENTS.md) (Rust core)
---
## How It Works

36
CONTRIBUTING.md
View file

@ -8,6 +8,7 @@ Here are the core requirements for any PR submitted to LiteLLM:
- [ ] **Sign the Contributor License Agreement (CLA)** - [see details](#contributor-license-agreement-cla)
- [ ] **Keep scope isolated** - Your changes should address 1 specific problem at a time
- [ ] **Prove it works end to end** - If your change affects what the proxy or SDK does, paste the commands you ran against a live proxy and their output in the PR - [see details](#proving-your-change-works-end-to-end)
#### Proxy (Backend) PRs
@ -248,6 +249,41 @@ If `make test-unit` fails:
- **Test edge cases**: Don't just test the happy path
- **Update documentation**: If you change APIs, update docs
## Proving Your Change Works End to End
Tests are required, but they do not prove that your change works for a user. A PR that changes what the proxy or SDK does shows a real run in its "Screenshots / Proof of Fix" section, meaning the exact commands you sent to a live proxy and the output that came back, against real provider APIs and not mocks. `pytest` output does not count as proof, because a passing test only shows that the code does what its mocks expect. A PR that changes only docs or CI shows the rendered page or the CI run instead
A good proof looks like this:
1. Install the proxy dependencies with `make install-proxy-dev`, then start the proxy from your branch with a config that has the model you changed, `uv run litellm --config your_config.yaml --detailed_debug`
2. Send the request a user would send with `curl`, and paste both the command and the response. When cost or routing is involved, include the response headers, since `x-litellm-response-cost` is what the user sees
3. For a bug fix, do this twice, once at the merge base with BerriAI/litellm's main (`git merge-base origin/main HEAD`, with your remote for BerriAI/litellm in place of `origin` in a fork clone) to show the bug and once at your branch's latest commit to show the fix. Name both commit hashes
4. If your change is reachable from more than one endpoint (`/v1/chat/completions`, `/v1/responses`, `/v1/messages`), show each one
5. If the request can stream, show a `"stream": true` run too, because streaming builds usage and cost on a different path. A stream sends its headers before its usage exists, so show the `usage.cost` on the last chunk, which `/v1/chat/completions` sends when the request has `"stream_options": {"include_usage": true}` and other endpoints send with `include_cost_in_streaming_usage: true` under `litellm_settings`, or, when the proxy has a database, the request's row on the Admin UI's Logs page or `GET /spend/logs`
6. For UI changes, add before and after screenshots and name the page
7. If users reach your change through a coding tool such as Claude Code or Codex, drive that tool the way a user does and add before and after screenshots of it, because a `curl` or a headless run alone does not show what the user sees
```bash
curl -sD - http://localhost:4000/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "your-model", "messages": [{"role": "user", "content": "hi"}]}'
```
When the provider reports its own usage or spend numbers, compare the proxy's numbers against them in the proof
## Contributing an LLM Provider Integration
Provider work (a new provider, a new parameter, a new usage or cost field) is the most common kind of contribution, and it is where PRs get reworked most often. Follow these rules
**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths that move data between the endpoints, the provider transformations, and cost tracking. A second, hand-rolled path for the same data is a common reason a provider PR gets rewritten before merge. [`ARCHITECTURE.md`](ARCHITECTURE.md) shows where the layers live, and the decision records in [`docs/adrs/`](docs/adrs/AGENTS.md) explain why they are shaped that way. For example, [the ADR on provider usage fields](docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md) covers how provider-specific usage fields reach cost tracking. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before you build one
**Bill a new provider usage field from the usage object.** Leave the field on the usage object as an extra and read it in that provider's `cost_calculator.py`. A provider without one gets a new `cost_calculator.py` and an arm for it in `cost_per_token` (`litellm/cost_calculator.py`), which picks the calculator by provider. If the provider's transform builds `Usage` from named fields, as the Anthropic, Gemini, and Bedrock Converse ones do, pass the new field there too. Streamed requests rebuild usage in `ChunkProcessor.calculate_usage` (`litellm/litellm_core_utils/streaming_chunk_builder_utils.py`), which does not copy extras yet the way the Responses-to-chat bridge does (minus a short exclusion list), so until it does, bill the field on streams by naming it there, as Anthropic's `inference_geo` is. A provider that reports its own charge for the request, as OpenRouter does in `usage.cost`, is billed at that charge from `_hidden_params` before any calculator runs, unless the deployment sets its own prices. When the field counts a built-in tool such as web search, add the provider to `get_cost_for_web_search_request` in `litellm/llms/__init__.py`, and add a gate to `litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py` only when no existing gate fires. Then show the field's cost in your proof on every endpoint it reaches, streaming included
**Do not change what a caller sees to make internals easier.** Each endpoint keeps the schema of the API it emulates. A `/v1/responses` caller gets `input_tokens` and a `/v1/chat/completions` caller gets `prompt_tokens`, whatever the provider sent and whatever cost tracking needs. A provider usage field that shows up as an extra key is fine, since it only adds a key, but renaming, removing, or reshaping a key the endpoint defines is not. Reshaping a public response to feed an internal consumer breaks every user of that endpoint
If your change makes an architecture decision, such as a new cross-cutting mechanism or a deliberate difference from a provider's API shape, add an ADR to [`docs/adrs/`](docs/adrs/AGENTS.md) in the same PR
## Building and Running Locally
### LiteLLM Proxy Server

5
docs/adrs/AGENTS.md Normal file
View file

@ -0,0 +1,5 @@
# Architecture decision records for the Python SDK and proxy
This folder records decisions about the Python SDK and proxy the way [`litellm-rust/docs/adrs/`](../../litellm-rust/docs/adrs/AGENTS.md) records them for the Rust core, and that file's process applies here unchanged: one short `adr_NNN_short_title.md` per decision that is hard to reverse or that a new contributor would reasonably question, with Context, Decision, Alternatives Considered, and Consequences, superseded by a new ADR instead of edited once accepted. Each folder numbers its own ADRs, so a reference to an ADR in the other folder names its path
Read the ADRs here before you build a new way to move data between the endpoints, the provider transformations, and cost tracking or logging. A PR that makes a decision of that kind adds its ADR in the same PR

39
docs/adrs/adr_001_provider_usage_fields_ride_as_usage_extras.md Normal file
View file

@ -0,0 +1,39 @@
# ADR 001: Provider usage fields ride as extras on the normalized usage object
Status: Accepted
## Context
Providers report usage fields that no other provider has, such as xAI's `usage.server_side_tool_usage_details.web_search_calls` or Anthropic's `usage.inference_geo`, and cost tracking has to read them or that spend never reaches `LiteLLM_SpendLogs`. One provider response can be served on `/v1/chat/completions`, `/v1/responses`, and `/v1/messages`, streaming or not, so a field that only one of those paths carries is billed wrong on the others
[#30817](https://github.com/BerriAI/litellm/pull/30817) fixed unbilled xAI web searches. One draft did the copy in the xAI Responses transform to keep provider code out of shared files, and that swapped `/v1/responses` usage to the chat shape, so callers of that endpoint lost the Responses schema. The merged version keeps the schema, copies unknown usage fields in the shared Responses-to-chat bridge for every provider, and gates on the reported field without naming the provider
In October 2026, `Usage` and `ResponseAPIUsage` kept unknown usage keys as pydantic extras, and that bridge copied them. Paths that build usage from named fields kept only the fields they name: the Anthropic, Gemini, and Bedrock Converse transforms, the stream builder, and the bridges from chat usage to `/v1/responses` and `/v1/messages`
## Decision
Provider-only usage fields stay on the usage object as pydantic extras, and cost tracking reads them there, in that provider's cost calculator or in a shared gate that checks the field instead of the provider. An extra reaches callers as an added usage key. No transform renames, removes, or reshapes a usage field the endpoint's schema defines to feed cost tracking, though it may fill one LiteLLM's usage object already defines, such as `server_tool_use`
A shared path that rebuilds usage, such as the stream builder or a bridge between endpoint shapes, copies the extras instead of naming provider fields. A provider transform that builds usage from named fields names the new field. Neither is bypassed with a side channel. Older code that differs, such as the Azure branches in the shared tracker and the Gemini pricing tier read from `_hidden_params`, predates this ADR
## Alternatives Considered
**Reshape usage in the provider transform**, as a draft of #30817 did. It changes the schema that callers of that endpoint rely on, and every provider and endpoint needs its own copy
**A typed field on `ResponseAPIUsage` and `Usage` for each provider's quirk**. The types grow without bound and every bridge needs an update for each one. A field several providers report, like `server_tool_use`, does get one
**A side channel such as `_hidden_params`**. Cost calculators receive `Usage`, so a side channel makes a second source of truth for the same number
**Provider branches or provider field names in shared code**. They put one provider's logic in a file every provider shares, which is why #30817 replaced its xAI-only gate with one that checks the reported field
## Consequences
Benefits:
- Billing a new provider usage field touches that provider's calculator (plus a `cost_per_token` arm when the calculator is new), its transform when that names fields, and for a built-in tool at most one gate and one `get_cost_for_web_search_request` arm
Costs:
- Extras are untyped, so a reader validates them before use, as `_usage_reports_server_side_web_search_calls` does
- The stream builder and the bridges from chat usage did not copy extras when this was written, so until they do, a field billed on those paths is named in them, and tests and proof runs cover each endpoint and `"stream": true`
- This covers usage fields LiteLLM prices, not request-side params or a provider's own charge for the request, which OpenRouter and opencode Go report and `response_cost_calculator` reads from `_hidden_params` first

10
enterprise/enterprise_hooks/google_text_moderation.py
View file

@ -15,7 +15,10 @@ import litellm
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.guardrails._content_utils import iter_message_text
from litellm.proxy.guardrails._content_utils import (
is_non_conversational_request,
iter_message_text,
)
from litellm.types.utils import CallTypesLiteral
_THRESHOLD_VALUE: Final = TypeAdapter(object)
@ -94,7 +97,7 @@ class ENTERPRISE_GoogleTextModeration(CustomLogger):
async def async_moderation_hook(
self,
data: dict,
data: dict[str, object],
user_api_key_dict: UserAPIKeyAuth,
call_type: CallTypesLiteral,
):
@ -102,6 +105,9 @@ class ENTERPRISE_GoogleTextModeration(CustomLogger):
- Calls Google's Text Moderation API
- Rejects request if it fails safety check
"""
if is_non_conversational_request(call_type, data):
verbose_proxy_logger.debug("Google text moderation: skipping non-conversational call type %s", call_type)
return
# Covers multimodal list content + Responses-API input.
text = "".join(iter_message_text(data))
if text:

3
enterprise/litellm_enterprise/enterprise_callbacks/callback_controls.py
View file

@ -33,9 +33,8 @@ class EnterpriseCallbackControls:
try:
disabled_callbacks = EnterpriseCallbackControls.get_disabled_callbacks(litellm_params, standard_callback_dynamic_params)
verbose_logger.debug(f"Dynamically disabled callbacks from {X_LITELLM_DISABLE_CALLBACKS}: {disabled_callbacks}")
verbose_logger.debug(f"Checking if {callback} is disabled via headers. Disable callbacks from headers: {disabled_callbacks}")
if disabled_callbacks is not None:
verbose_logger.debug(f"Checking if {callback} is in dynamically disabled callbacks: {disabled_callbacks}")
#########################################################
# premium user check
#########################################################

133
litellm-rust/crates/inference-ocr/tests/ocr/azure_document_intelligence.rs
View file

@ -12,6 +12,7 @@ use rstest::rstest;
use super::*;
const MODEL: &str = "azure_ai/doc-intelligence/prebuilt-read";
const LAYOUT_MODEL: &str = "azure_ai/doc-intelligence/prebuilt-layout";
fn read_request(base: &str, options: Value) -> LiteLLMOcrRequest {
ocr_request(MODEL, base, options)
@ -70,6 +71,7 @@ async fn pages_features_and_extra_options_map_to_the_analyze_call() {
#[case(json!({"pages": "1&&features=bad"}), Error::Pages(PagesFault::Range))]
#[case(json!({"features": "languages&pages=1"}), Error::Features)]
#[case(json!({"req_format": "azure"}), Error::RequestFormat)]
#[case(json!({"outputContentFormat": "html"}), Error::OutputContentFormat)]
#[tokio::test]
async fn invalid_pages_features_and_format_are_rejected_before_sending(
#[case] options: Value,
@ -130,6 +132,137 @@ async fn an_inline_document_is_sent_as_base64_and_only_page_text_is_kept(#[case]
assert_eq!(sent.json(), json!({"base64Source": "YWJj"}));
}
#[rstest]
#[case::lower_case("markdown")]
#[case::capitalized("Markdown")]
#[tokio::test]
async fn markdown_output_is_requested_on_the_analyze_call_and_split_into_pages(
#[case] output_content_format: &str,
) {
let upstream = upstream([json_response(json!({
"status": "succeeded",
"analyzeResult": {
"content": "# Tïtle\n\n<!-- PageBreak -->\n\n| a |",
"contentFormat": "markdown",
"stringIndexType": "unicodeCodePoint",
"pages": [
{"pageNumber": 1, "spans": [{"offset": 0, "length": 29}], "lines": [{"content": "Tïtle"}]},
{"pageNumber": 2, "spans": [{"offset": 29, "length": 5}], "lines": [{"content": "a"}]}
]
}
}))])
.await;
let response = perform(ocr_request(
LAYOUT_MODEL,
&upstream.uri(),
json!({"outputContentFormat": output_content_format, "pages": [0, 1]}),
))
.await
.unwrap();
let sent = only_request(&upstream).await;
assert_eq!(
sent.query("outputContentFormat").as_deref(),
Some("markdown")
);
assert_eq!(
sent.query("stringIndexType").as_deref(),
Some("unicodeCodePoint")
);
assert_eq!(sent.query("pages").as_deref(), Some("1,2"));
assert_eq!(sent.json(), json!({"base64Source": "YWJj"}));
assert_eq!(
response.content.as_deref(),
Some("# Tïtle\n\n<!-- PageBreak -->\n\n| a |")
);
let pages = response
.pages
.iter()
.map(|page| page.markdown.as_str())
.collect::<Vec<_>>();
assert_eq!(pages, ["# Tïtle\n\n<!-- PageBreak -->\n\n", "| a |"]);
assert_eq!(
response.extra_fields.get("stringIndexType"),
Some(&json!("unicodeCodePoint"))
);
assert_eq!(
response.usage_info.as_ref().unwrap().pages_processed,
Some(2)
);
}
#[rstest]
#[case::native_markdown(LAYOUT_MODEL, json!({"outputContentFormat": "markdown", "req_format": "native"}), Some("markdown"))]
#[case::markdown_on_read(MODEL, json!({"outputContentFormat": "markdown"}), Some("markdown"))]
#[case::text(LAYOUT_MODEL, json!({"outputContentFormat": "text"}), Some("text"))]
#[case::omitted(LAYOUT_MODEL, json!({}), None)]
#[tokio::test]
async fn only_layout_markdown_in_litellm_format_asks_for_code_point_offsets(
#[case] model: &str,
#[case] options: Value,
#[case] output_content_format: Option<&str>,
) {
let upstream = upstream([json_response(json!({
"status": "succeeded",
"analyzeResult": {
"contentFormat": "text",
"stringIndexType": "textElements",
"pages": [{"pageNumber": 1, "lines": [{"content": "hello"}]}]
}
}))])
.await;
let response = perform(ocr_request(model, &upstream.uri(), options))
.await
.unwrap();
let sent = only_request(&upstream).await;
assert_eq!(
sent.query("outputContentFormat").as_deref(),
output_content_format
);
assert_eq!(sent.query("stringIndexType"), None);
assert_eq!(sent.json(), json!({"base64Source": "YWJj"}));
assert_eq!(response.pages[0].markdown, "hello");
assert_eq!(response.extra_fields.get("stringIndexType"), None);
}
#[rstest]
#[case::snake_case(json!({"output_content_format": "markdown"}), Some("markdown"))]
#[case::padded(json!({"outputContentFormat": " markdown "}), Some("markdown"))]
#[case::empty(json!({"outputContentFormat": ""}), None)]
#[case::empty_camel_case_falls_back_to_snake_case(
json!({"outputContentFormat": "", "output_content_format": "markdown"}),
Some("markdown")
)]
#[case::dropped(json!({"outputContentFormat": "html", "drop_params": true}), None)]
#[tokio::test]
async fn the_requested_output_content_format_reaches_azure_in_its_query_spelling(
#[case] options: Value,
#[case] output_content_format: Option<&str>,
) {
let upstream = upstream([json_response(json!({
"status": "succeeded",
"analyzeResult": {"pages": [{"pageNumber": 1, "lines": [{"content": "hello"}]}]}
}))])
.await;
let response = perform(ocr_request(MODEL, &upstream.uri(), options))
.await
.unwrap();
let sent = only_request(&upstream).await;
assert_eq!(
sent.query("outputContentFormat").as_deref(),
output_content_format
);
assert_eq!(sent.query("output_content_format"), None);
assert_eq!(sent.query("drop_params"), None);
assert_eq!(sent.json(), json!({"base64Source": "YWJj"}));
assert_eq!(response.pages[0].markdown, "hello");
}
#[tokio::test]
async fn native_format_normalizes_pages_and_keeps_the_provider_response() {
let operation = json!({

23
litellm-rust/crates/llms-types/src/formats/audio_transcription/response.rs
View file

@ -1,14 +1,31 @@
use serde_json::Value;
#[macro_rules_attribute::apply(crate::wire_type)]
pub struct AudioTranscriptionUsage {
pub input_tokens: u64,
pub output_tokens: u64,
pub total_tokens: u64,
}
#[macro_rules_attribute::apply(crate::wire_type)]
pub struct AudioTranscriptionResponseData {
pub text: String,
pub usage: Option<AudioTranscriptionUsage>,
}
impl AudioTranscriptionResponseData {
pub fn into_json(self) -> Value {
serde_json::json!({
"text": self.text,
})
match self.usage {
None => serde_json::json!({"text": self.text}),
Some(usage) => serde_json::json!({
"text": self.text,
"usage": {
"type": "tokens",
"input_tokens": usage.input_tokens,
"output_tokens": usage.output_tokens,
"total_tokens": usage.total_tokens,
},
}),
}
}
}

426
litellm-rust/crates/llms/src/azure_ai/ocr/document_intelligence/transformation.rs
View file

@ -1,4 +1,7 @@
use std::{collections::BTreeSet, time::Duration};
use std::{
collections::{BTreeMap, BTreeSet},
time::Duration,
};
use base64::{Engine, engine::general_purpose::STANDARD};
use litellm_auth::{InputSource, Sourced};
@ -8,7 +11,7 @@ use litellm_llms_types::serde_compat::{FiniteF64, LaxI64};
use reqwest::Url;
use serde::{Deserialize, Serialize};
use serde_json::{Map, Value};
use serde_with::serde_as;
use serde_with::{DefaultOnError, serde_as};
use tokio::time::Instant;
use crate::base_llm::ocr::{
@ -17,9 +20,10 @@ use crate::base_llm::ocr::{
handler::{CallHooks, OcrClient, read_json_response},
settings::OcrSettings,
transformation::{
BaseOcrConfig, DecodedOcrResponse, OCR_INLINE_MAX_BYTES, OCR_POLL_RETRY_SECS,
OcrConnection, OcrCredentialInputs, OcrResponseContext, PreparedOcrRequest,
ResolvedOcrCredentials, decode_and_normalize_response, decode_response,
BaseOcrConfig, DROP_PARAMS_PARAM, DecodedOcrResponse, OCR_INLINE_MAX_BYTES,
OCR_POLL_RETRY_SECS, OcrConnection, OcrCredentialInputs, OcrResponseContext,
PreparedOcrRequest, ResolvedOcrCredentials, decode_and_normalize_response, decode_response,
drops_unsupported_params, response_format,
},
};
use litellm_llms_types::formats::ocr::{
@ -32,6 +36,30 @@ const AZURE_DI_DEFAULT_HEIGHT: f64 = 11.0;
const AZURE_DI_API_KEY_ENV: &str = "AZURE_DOCUMENT_INTELLIGENCE_API_KEY";
const AZURE_DI_ENDPOINT_ENV: &str = "AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT";
const OUTPUT_CONTENT_FORMAT_PARAM: &str = "outputContentFormat";
const OUTPUT_CONTENT_FORMAT_ALIAS: &str = "output_content_format";
const CODE_POINT_STRING_INDEX: &str = "unicodeCodePoint";
const STRING_INDEX_TYPE_FIELD: &str = "stringIndexType";
const MARKDOWN_MODEL_ID: &str = "prebuilt-layout";
#[derive(
Clone,
Copy,
Debug,
PartialEq,
strum::EnumString,
strum::IntoStaticStr,
strum::Display,
strum::VariantArray,
serde_with::SerializeDisplay,
)]
#[strum(ascii_case_insensitive)]
pub enum OutputContentFormat {
#[strum(serialize = "text")]
Text,
#[strum(serialize = "markdown")]
Markdown,
}
#[derive(Clone, Debug, PartialEq, Serialize)]
pub struct DocumentIntelligenceParams {
@ -39,6 +67,13 @@ pub struct DocumentIntelligenceParams {
pub pages: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub features: Option<String>,
#[serde(
rename = "outputContentFormat",
skip_serializing_if = "Option::is_none"
)]
pub output_content_format: Option<OutputContentFormat>,
#[serde(rename = "stringIndexType", skip_serializing_if = "Option::is_none")]
pub string_index_type: Option<&'static str>,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
@ -80,6 +115,10 @@ pub struct AzureDocumentIntelligenceOperation {
#[derive(Clone, Debug, Default, Deserialize)]
struct AzureDocumentIntelligenceAnalyzeResult {
pub content: Option<String>,
#[serde(rename = "contentFormat")]
pub content_format: Option<String>,
#[serde(rename = "stringIndexType")]
pub string_index_type: Option<String>,
#[serde(default)]
pub pages: Vec<AzureDocumentIntelligencePage>,
pub tables: Option<Vec<Map<String, Value>>>,
@ -100,6 +139,9 @@ struct AzureDocumentIntelligencePage {
pub unit: Option<String>,
#[serde(default)]
pub lines: Vec<AzureDocumentIntelligenceLine>,
#[serde(default)]
#[serde_as(deserialize_as = "DefaultOnError")]
pub spans: Option<Vec<AzureDocumentIntelligenceSpan>>,
}
#[derive(Clone, Debug, Deserialize)]
@ -107,6 +149,18 @@ struct AzureDocumentIntelligenceLine {
pub content: Option<String>,
}
#[derive(Clone, Debug, Deserialize)]
struct AzureDocumentIntelligenceSpan {
pub offset: usize,
pub length: usize,
}
impl AzureDocumentIntelligenceSpan {
fn end(&self) -> usize {
self.offset.saturating_add(self.length)
}
}
#[derive(Clone, Debug)]
pub struct AzureDocumentIntelligenceOcrConfig;
@ -116,7 +170,14 @@ impl BaseOcrConfig for AzureDocumentIntelligenceOcrConfig {
type Environment = Vec<(String, String)>;
fn get_supported_ocr_params(&self, _model: &str) -> &'static [&'static str] {
&["pages", "features", "req_format"]
&[
"pages",
"features",
OUTPUT_CONTENT_FORMAT_PARAM,
OUTPUT_CONTENT_FORMAT_ALIAS,
"req_format",
DROP_PARAMS_PARAM,
]
}
fn get_api_key_env_var(&self) -> Option<&'static str> {
@ -154,11 +215,17 @@ impl BaseOcrConfig for AzureDocumentIntelligenceOcrConfig {
fn map_ocr_params(
&self,
non_default_params: &CallArguments,
_model: &str,
model: &str,
) -> Result<DocumentIntelligenceParams, Error> {
let output_content_format = requested_output_content_format(non_default_params)?;
let slices_markdown_pages = output_content_format == Some(OutputContentFormat::Markdown)
&& response_format(non_default_params)? != OcrResponseFormat::Native
&& model_id(model).is_ok_and(|id| id == MARKDOWN_MODEL_ID);
Ok(DocumentIntelligenceParams {
pages: normalize_pages_param(non_default_params.get("pages"))?,
features: normalize_features_param(non_default_params.get("features"))?,
output_content_format,
string_index_type: slices_markdown_pages.then_some(CODE_POINT_STRING_INDEX),
})
}
@ -337,6 +404,36 @@ fn normalize_features_param(features: Option<&Value>) -> Result<Option<String>,
Ok(Some(normalized.join(",")))
}
fn requested_output_content_format(
params: &CallArguments,
) -> Result<Option<OutputContentFormat>, Error> {
let requested = [OUTPUT_CONTENT_FORMAT_PARAM, OUTPUT_CONTENT_FORMAT_ALIAS]
.into_iter()
.filter_map(|name| params.get(name))
.find(|format| !is_unset(format));
match requested.map(parse_output_content_format).transpose() {
Err(Error::OutputContentFormat) if drops_unsupported_params(params) => Ok(None),
parsed => parsed,
}
}
fn is_unset(format: &Value) -> bool {
match format {
Value::Null => true,
Value::String(format) => format.trim().is_empty(),
_ => false,
}
}
fn parse_output_content_format(format: &Value) -> Result<OutputContentFormat, Error> {
format
.as_str()
.ok_or(Error::OutputContentFormat)?
.trim()
.parse()
.map_err(|_| Error::OutputContentFormat)
}
fn build_request(document: OcrDocument) -> Result<DocumentIntelligenceRequest, Error> {
let source = document.source();
if source.is_empty() {
@ -367,10 +464,27 @@ fn transform_completed_response(
));
}
let result = response.analyze_result.unwrap_or_default();
let markdown_content = result
.content
.as_deref()
.filter(|_| {
result.content_format.as_deref() == Some(<&str>::from(OutputContentFormat::Markdown))
&& result.string_index_type.as_deref() == Some(CODE_POINT_STRING_INDEX)
&& result.pages.iter().all(|page| page.spans.is_some())
})
.map(|content| {
CodePointText::new(
content,
result
.pages
.iter()
.flat_map(|page| page.spans.iter().flatten()),
)
});
let pages = result
.pages
.into_iter()
.map(|page| transform_azure_page(page, dpi))
.map(|page| transform_azure_page(page, markdown_content.as_ref(), dpi))
.collect::<Result<Vec<_>, _>>()?;
let pages_processed = i64::try_from(pages.len()).map_err(|_| Error::NumericRange("pages"))?;
Ok(LiteLLMOcrResponse {
@ -381,11 +495,20 @@ fn transform_completed_response(
pages_processed: Some(pages_processed),
..Default::default()
}),
extra_fields: result
.string_index_type
.filter(|unit| unit == CODE_POINT_STRING_INDEX)
.map(|unit| Map::from_iter([(STRING_INDEX_TYPE_FIELD.to_owned(), Value::String(unit))]))
.unwrap_or_default(),
..LiteLLMOcrResponse::new(model, pages)
})
}
fn transform_azure_page(page: AzureDocumentIntelligencePage, dpi: i64) -> Result<OcrPage, Error> {
fn transform_azure_page(
page: AzureDocumentIntelligencePage,
markdown_content: Option<&CodePointText<'_>>,
dpi: i64,
) -> Result<OcrPage, Error> {
let index = page
.page_number
.unwrap_or(1)
@ -397,12 +520,20 @@ fn transform_azure_page(page: AzureDocumentIntelligencePage, dpi: i64) -> Result
page.unit.as_deref().unwrap_or("inch"),
dpi,
)?;
let markdown = page
.lines
.iter()
.map(|line| line.content.as_deref().unwrap_or_default())
.collect::<Vec<_>>()
.join("\n");
let markdown = match markdown_content {
Some(content) => page
.spans
.iter()
.flatten()
.map(|span| content.slice(span))
.collect::<String>(),
None => page
.lines
.iter()
.map(|line| line.content.as_deref().unwrap_or_default())
.collect::<Vec<_>>()
.join("\n"),
};
Ok(OcrPage {
index,
markdown,
@ -411,6 +542,43 @@ fn transform_azure_page(page: AzureDocumentIntelligencePage, dpi: i64) -> Result
})
}
struct CodePointText<'a> {
text: &'a str,
byte_offsets: BTreeMap<usize, usize>,
}
impl<'a> CodePointText<'a> {
fn new<'s>(
text: &'a str,
spans: impl Iterator<Item = &'s AzureDocumentIntelligenceSpan>,
) -> Self {
let span_bounds = spans
.flat_map(|span| [span.offset, span.end()])
.collect::<BTreeSet<_>>();
Self {
text,
byte_offsets: text
.char_indices()
.map(|(byte_offset, _)| byte_offset)
.enumerate()
.filter(|(code_point, _)| span_bounds.contains(code_point))
.collect(),
}
}
fn slice(&self, span: &AzureDocumentIntelligenceSpan) -> &'a str {
let byte_offset = |code_point: usize| {
self.byte_offsets
.get(&code_point)
.copied()
.unwrap_or(self.text.len())
};
self.text
.get(byte_offset(span.offset)..byte_offset(span.end()))
.unwrap_or_default()
}
}
fn convert_dimensions(
width: f64,
height: f64,
@ -578,6 +746,16 @@ impl AzureDocumentIntelligenceOcrConfig {
.features
.iter()
.map(|features| ("features", features.as_str())),
)
.chain(
params
.output_content_format
.map(|format| (OUTPUT_CONTENT_FORMAT_PARAM, format.into())),
)
.chain(
params
.string_index_type
.map(|index_type| ("stringIndexType", index_type)),
),
)
.into_string()
@ -697,8 +875,12 @@ mod tests {
}
fn map(value: Value) -> Result<DocumentIntelligenceParams, Error> {
map_for_model(value, "azure_ai/doc-intelligence/prebuilt-layout")
}
fn map_for_model(value: Value, model: &str) -> Result<DocumentIntelligenceParams, Error> {
let arguments = serde_json::from_value(value).unwrap();
AzureDocumentIntelligenceOcrConfig.map_ocr_params(&arguments, "model")
AzureDocumentIntelligenceOcrConfig.map_ocr_params(&arguments, model)
}
#[test]
@ -828,6 +1010,218 @@ mod tests {
assert_eq!(map(json!({"features": []})).unwrap().features, None);
}
#[rstest]
#[case::markdown(
json!({"outputContentFormat": "markdown"}),
Some(OutputContentFormat::Markdown),
Some("unicodeCodePoint")
)]
#[case::markdown_litellm_format(
json!({"outputContentFormat": "markdown", "req_format": "litellm"}),
Some(OutputContentFormat::Markdown),
Some("unicodeCodePoint")
)]
#[case::markdown_native_format(
json!({"outputContentFormat": "markdown", "req_format": "native"}),
Some(OutputContentFormat::Markdown),
None
)]
#[case::capitalized_markdown(
json!({"outputContentFormat": "Markdown"}),
Some(OutputContentFormat::Markdown),
Some("unicodeCodePoint")
)]
#[case::text(json!({"outputContentFormat": "text"}), Some(OutputContentFormat::Text), None)]
#[case::upper_case_text(json!({"outputContentFormat": "TEXT"}), Some(OutputContentFormat::Text), None)]
#[case::padded_markdown(
json!({"outputContentFormat": " markdown\n"}),
Some(OutputContentFormat::Markdown),
Some("unicodeCodePoint")
)]
#[case::snake_case_markdown(
json!({"output_content_format": "markdown"}),
Some(OutputContentFormat::Markdown),
Some("unicodeCodePoint")
)]
#[case::camel_case_wins_over_snake_case(
json!({"outputContentFormat": "text", "output_content_format": "markdown"}),
Some(OutputContentFormat::Text),
None
)]
#[case::empty_camel_case_falls_back_to_snake_case(
json!({"outputContentFormat": "", "output_content_format": "markdown"}),
Some(OutputContentFormat::Markdown),
Some("unicodeCodePoint")
)]
#[case::null_camel_case_falls_back_to_snake_case(
json!({"outputContentFormat": null, "output_content_format": "markdown"}),
Some(OutputContentFormat::Markdown),
Some("unicodeCodePoint")
)]
#[case::empty(json!({"outputContentFormat": ""}), None, None)]
#[case::blank(json!({"outputContentFormat": " "}), None, None)]
#[case::null(json!({"outputContentFormat": null}), None, None)]
#[case::omitted(json!({}), None, None)]
fn output_content_format_maps_to_query_fields(
#[case] options: Value,
#[case] format: Option<OutputContentFormat>,
#[case] string_index_type: Option<&str>,
) {
let mapped = map(options).unwrap();
assert_eq!(mapped.output_content_format, format);
assert_eq!(mapped.string_index_type, string_index_type);
}
#[rstest]
#[case::layout("azure_ai/doc-intelligence/prebuilt-layout", Some("unicodeCodePoint"))]
#[case::bare_layout("prebuilt-layout", Some("unicodeCodePoint"))]
#[case::read("azure_ai/doc-intelligence/prebuilt-read", None)]
#[case::invoice("azure_ai/doc-intelligence/prebuilt-invoice", None)]
#[case::custom("azure_ai/doc-intelligence/my-extraction-model", None)]
fn only_the_layout_model_asks_for_code_point_offsets(
#[case] model: &str,
#[case] string_index_type: Option<&str>,
) {
let mapped = map_for_model(json!({"outputContentFormat": "markdown"}), model).unwrap();
assert_eq!(
mapped.output_content_format,
Some(OutputContentFormat::Markdown)
);
assert_eq!(mapped.string_index_type, string_index_type);
}
#[rstest]
#[case::unknown(json!({"outputContentFormat": "html"}))]
#[case::number(json!({"outputContentFormat": 1}))]
#[case::boolean(json!({"outputContentFormat": true}))]
#[case::list(json!({"outputContentFormat": ["markdown"]}))]
#[case::snake_case_unknown(json!({"output_content_format": "html"}))]
#[case::drop_params_off(json!({"outputContentFormat": "html", "drop_params": false}))]
fn invalid_output_content_format_is_a_request_error(#[case] options: Value) {
let error = map(options).unwrap_err();
assert!(matches!(error, Error::OutputContentFormat), "{error:?}");
assert_eq!(error.http_status_code(), Some(400));
}
#[rstest]
#[case::unknown(json!({"outputContentFormat": "html", "drop_params": true, "pages": [0]}))]
#[case::number(json!({"outputContentFormat": 1, "drop_params": true, "pages": [0]}))]
#[case::snake_case_unknown(
json!({"output_content_format": "html", "drop_params": true, "pages": [0]})
)]
fn drop_params_drops_an_invalid_output_content_format(#[case] options: Value) {
let mapped = map(options).unwrap();
assert_eq!(serde_json::to_value(mapped).unwrap(), json!({"pages": "1"}));
}
#[rstest]
#[case::camel_case(json!({"outputContentFormat": "markdown", "drop_params": true}))]
#[case::snake_case(json!({"output_content_format": "markdown", "drop_params": true}))]
fn drop_params_keeps_a_valid_output_content_format(#[case] options: Value) {
let mapped = map(options).unwrap();
assert_eq!(
mapped.output_content_format,
Some(OutputContentFormat::Markdown)
);
}
fn markdown_pages(analyze_result: Value) -> Vec<String> {
let raw =
serde_json::to_vec(&json!({"status": "succeeded", "analyzeResult": analyze_result}))
.unwrap();
AzureDocumentIntelligenceOcrConfig
.transform_ocr_response("model", &raw, OcrResponseFormat::Litellm)
.unwrap()
.pages
.into_iter()
.map(|page| page.markdown)
.collect()
}
#[rstest]
fn markdown_pages_are_code_point_slices_of_the_content() {
let content = "# Ünïcode क्ष\n\n<!-- PageBreak -->\n\n## Två 🙂\n\n|a|\n\n> tail";
let second_page_start = "# Ünïcode क्ष\n\n<!-- PageBreak -->\n\n".chars().count();
let pages = markdown_pages(json!({
"content": content,
"contentFormat": "markdown",
"stringIndexType": "unicodeCodePoint",
"pages": [
{"pageNumber": 1, "spans": [{"offset": 0, "length": second_page_start}],
"lines": [{"content": "Ünïcode क्ष"}]},
{"pageNumber": 2, "spans": [
{"offset": second_page_start, "length": 8},
{"offset": second_page_start + 10, "length": 3}
], "lines": [{"content": "Två"}]}
]
}));
assert_eq!(
pages,
["# Ünïcode क्ष\n\n<!-- PageBreak -->\n\n", "## Två 🙂|a|"]
);
}
#[rstest]
fn spans_past_the_content_end_are_clamped() {
let pages = markdown_pages(json!({
"content": "ab",
"contentFormat": "markdown",
"stringIndexType": "unicodeCodePoint",
"pages": [{"spans": [{"offset": 1, "length": 9}, {"offset": 7, "length": 1}]}]
}));
assert_eq!(pages, ["b"]);
}
#[rstest]
#[case::text_content(json!("text"), json!("unicodeCodePoint"))]
#[case::grapheme_offsets(json!("markdown"), json!("textElements"))]
#[case::no_index_type(json!("markdown"), Value::Null)]
#[case::no_content_format(Value::Null, json!("unicodeCodePoint"))]
fn other_responses_keep_line_joined_pages(
#[case] content_format: Value,
#[case] string_index_type: Value,
) {
let pages = markdown_pages(json!({
"content": "# A\n\nB",
"contentFormat": content_format,
"stringIndexType": string_index_type,
"pages": [{"spans": [{"offset": 0, "length": 6}],
"lines": [{"content": "A"}, {"content": "B"}]}]
}));
assert_eq!(pages, ["A\nB"]);
}
#[rstest]
#[case::null_spans(json!("markdown"), json!(null))]
#[case::fractional_offset(json!("markdown"), json!([{"offset": 0.5, "length": 1}]))]
#[case::missing_length(json!("markdown"), json!([{"offset": 0}]))]
#[case::negative_offset(json!("markdown"), json!([{"offset": -1, "length": 1}]))]
#[case::text_content(json!("text"), json!([{"offset": "0", "length": 1}]))]
fn unusable_page_spans_keep_line_joined_pages(
#[case] content_format: Value,
#[case] broken_spans: Value,
) {
let pages = markdown_pages(json!({
"content": "# A\n\nB",
"contentFormat": content_format,
"stringIndexType": "unicodeCodePoint",
"pages": [
{"spans": [{"offset": 0, "length": 5}], "lines": [{"content": "A"}]},
{"spans": broken_spans, "lines": [{"content": "B"}]}
]
}));
assert_eq!(pages, ["A", "B"]);
}
#[tokio::test]
async fn request_endpoint_cannot_receive_environment_key() {
let connection = OcrConnection {

8
litellm-rust/crates/llms/src/base_llm/ocr/error.rs
View file

@ -59,6 +59,10 @@ pub enum Error {
Pages(PagesFault),
#[error("invalid OCR features")]
Features,
#[error(
"invalid OCR outputContentFormat, expected text or markdown. To drop unsupported params, set `litellm.drop_params = True`."
)]
OutputContentFormat,
#[error("OCR model cannot be a dot segment")]
DotModel,
#[error("OCR response exceeds the size limit of {limit} bytes")]
@ -200,6 +204,7 @@ impl Error {
| Self::InlineDocumentTooLarge
| Self::Pages(_)
| Self::Features
| Self::OutputContentFormat
| Self::DotModel
| Self::TooLarge { .. }
| Self::ResponseField { .. }
@ -242,6 +247,7 @@ impl Error {
| Self::InlineDocumentTooLarge
| Self::Pages(_)
| Self::Features
| Self::OutputContentFormat
| Self::DotModel
| Self::TooLarge { .. }
| Self::ResponseField { .. }
@ -291,6 +297,7 @@ impl Error {
| Self::InlineDocumentTooLarge
| Self::Pages(_)
| Self::Features
| Self::OutputContentFormat
| Self::DotModel
| Self::InvalidProvider(_)
| Self::InvalidModel { .. }
@ -345,6 +352,7 @@ impl Error {
| Self::InlineDocumentTooLarge
| Self::Pages(_)
| Self::Features
| Self::OutputContentFormat
| Self::DotModel
| Self::InvalidProvider(_)
| Self::InvalidModel { .. }

5
litellm-rust/crates/llms/src/base_llm/ocr/transformation.rs
View file

@ -21,6 +21,7 @@ pub const OCR_RESPONSE_MAX_BYTES: usize = 64 * 1024 * 1024;
pub const OCR_INLINE_MAX_BYTES: usize = 50 * 1024 * 1024;
pub const OCR_MAX_FETCH_REDIRECTS: usize = 10;
pub const OCR_POLL_RETRY_SECS: u64 = 2;
pub const DROP_PARAMS_PARAM: &str = "drop_params";
#[derive(Clone, Default)]
pub struct OcrCredentialInputs {
@ -185,6 +186,10 @@ pub fn response_format(optional_params: &CallArguments) -> Result<OcrResponseFor
.map(|format| format.unwrap_or_default())
}
pub fn drops_unsupported_params(optional_params: &CallArguments) -> bool {
optional_params.get(DROP_PARAMS_PARAM) == Some(&Value::Bool(true))
}
#[derive(Debug)]
pub struct DecodedOcrResponse<T> {
pub data: T,

83
litellm-rust/crates/llms/src/bedrock/audio_transcription/mod.rs
View file

@ -6,8 +6,8 @@ use litellm_auth_aws::{
};
use litellm_core_utils::core_helpers::json_type_name;
use litellm_llms_types::{
formats::audio_transcription::AudioTranscriptionResponseData,
providers::bedrock::ConverseResponse,
formats::audio_transcription::{AudioTranscriptionResponseData, AudioTranscriptionUsage},
providers::bedrock::{ConverseResponse, ConverseUsage},
};
use serde::Deserialize;
use serde_json::{Map, Value, json};
@ -101,10 +101,10 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig {
if let Some(prompt) = optional_string(&optional_params, "prompt") {
instruction.push_str(&format!(" Additional context: {prompt}"));
}
let mut inference_config = Map::from_iter([("maxTokens".to_string(), json!(4096))]);
if let Some(temperature) = optional_params.get("temperature") {
inference_config.insert("temperature".to_string(), temperature.clone());
}
let temperature = optional_params
.get("temperature")
.cloned()
.unwrap_or(json!(0));
Ok(AudioTranscriptionRequestData {
body: json!({
"messages": [{
@ -114,8 +114,7 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig {
{"text": instruction}
]
}],
"system": [{"text": "You are a transcription assistant."}],
"inferenceConfig": inference_config,
"inferenceConfig": {"maxTokens": 4096, "temperature": temperature},
}),
})
}
@ -125,6 +124,7 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig {
_model: &str,
response_json: Value,
) -> Result<AudioTranscriptionResponseData, Error> {
let reports_usage = response_json.get("usage").is_some();
let response: ConverseResponse =
serde_json::from_value(response_json).map_err(|error| {
Error::InvalidResponse(ErrorDetail::invalid("Converse response", error))
@ -137,6 +137,7 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig {
}
Ok(AudioTranscriptionResponseData {
text: response.content_text(),
usage: reports_usage.then(|| transcription_usage(&response.usage)),
})
}
@ -191,22 +192,41 @@ impl BaseAudioTranscriptionConfig for BedrockAudioTranscriptionConfig {
}
}
fn transcription_usage(usage: &ConverseUsage) -> AudioTranscriptionUsage {
AudioTranscriptionUsage {
input_tokens: usage.input_tokens,
output_tokens: usage.output_tokens,
total_tokens: usage
.total_tokens
.unwrap_or(usage.input_tokens + usage.output_tokens),
}
}
#[cfg(test)]
mod tests {
use super::*;
use rstest::rstest;
fn no_env(_: &str) -> Option<String> {
None
}
#[test]
fn request_matches_python_shape() {
let params = Map::from_iter([
("language".to_string(), json!("en")),
("prompt".to_string(), json!("Speaker names")),
("temperature".to_string(), json!(0)),
("timestamp_granularities".to_string(), json!(["word"])),
]);
#[rstest]
#[case::caller_temperature(Some(json!(0.4)), json!(0.4))]
#[case::default_temperature(None, json!(0))]
fn request_sends_audio_and_instruction_without_system_block(
#[case] temperature: Option<Value>,
#[case] expected_temperature: Value,
) {
let params = Map::from_iter(
[
("language".to_string(), json!("en")),
("prompt".to_string(), json!("Speaker names")),
("timestamp_granularities".to_string(), json!(["word"])),
]
.into_iter()
.chain(temperature.map(|value| ("temperature".to_string(), value))),
);
let params = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG.map_transcription_params(&params);
let result = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG
.transform_audio_transcription_request(
@ -225,22 +245,39 @@ mod tests {
{"text": "Transcribe the audio. Respond with only the transcript. The audio language is en. Additional context: Speaker names"}
]
}],
"system": [{"text": "You are a transcription assistant."}],
"inferenceConfig": {"maxTokens": 4096, "temperature": 0}
"inferenceConfig": {"maxTokens": 4096, "temperature": expected_temperature}
})
);
}
#[test]
fn response_concatenates_content_blocks() {
#[rstest]
#[case::reported_total(
json!({"inputTokens": 390, "outputTokens": 60, "totalTokens": 450}),
json!({"text": "hello world", "usage": {"type": "tokens", "input_tokens": 390, "output_tokens": 60, "total_tokens": 450}})
)]
#[case::derived_total(
json!({"inputTokens": 1, "outputTokens": 2}),
json!({"text": "hello world", "usage": {"type": "tokens", "input_tokens": 1, "output_tokens": 2, "total_tokens": 3}})
)]
fn response_carries_text_and_converse_usage(#[case] usage: Value, #[case] expected: Value) {
let result = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG
.transform_audio_transcription_response(
"model",
json!({"output": {"message": {"content": [{"text": "hello "}, {"text": "world"}]}}, "usage": {"inputTokens": 1, "outputTokens": 2}}),
json!({"output": {"message": {"content": [{"text": "hello "}, {"text": "world"}]}}, "usage": usage}),
)
.expect("response");
assert_eq!(result.text, "hello world");
assert_eq!(result.into_json(), json!({"text": "hello world"}));
assert_eq!(result.into_json(), expected);
}
#[rstest]
fn response_without_usage_reports_none() {
let result = BEDROCK_AUDIO_TRANSCRIPTION_CONFIG
.transform_audio_transcription_response(
"model",
json!({"output": {"message": {"content": [{"text": "hello"}]}}}),
)
.expect("response");
assert_eq!(result.into_json(), json!({"text": "hello"}));
}
#[test]

6
litellm-rust/crates/python-bridge/src/routes/ocr/host.rs
View file

@ -13,7 +13,7 @@ use pyo3::{
use super::{
errors::route_failure,
project::{OcrHostHandles, SETTINGS_ERROR_MARKER, project_request},
project::{OcrHostHandles, ROUTE_HOST_MODULE, SETTINGS_ERROR_MARKER, project_request},
};
use crate::{marshal::public_response, routes::RequestView};
@ -77,7 +77,7 @@ impl OcrPythonHost {
_ => "",
};
let mapped = py
.import("litellm.rust_bridge.host.routes.ocr")
.import(ROUTE_HOST_MODULE)
.and_then(|module| module.getattr("map_failure"))
.and_then(|map| map.call1((error.value(py), self.view.request(py)?, provider)))
.and_then(|mapped| mapped.extract::<Py<PyBaseException>>().map_err(PyErr::from));
@ -115,7 +115,7 @@ impl PythonBinding for OcrPythonHost {
py: Python<'_>,
response: LiteLLMOcrResponse,
) -> PyResult<Py<PyAny>> {
public_response(py, "litellm.rust_bridge.host.routes.ocr", &response)
public_response(py, ROUTE_HOST_MODULE, &response)
}
fn encode_stream_head(

30
litellm-rust/crates/python-bridge/src/routes/ocr/project.rs
View file

@ -5,7 +5,7 @@ use litellm_inference_ocr::{
types::{LiteLLMOcrRequest, OcrDocumentInput},
wire::{OcrWireRequest, consumed_optional_params, decode_document, decode_request_input},
};
use litellm_llms::base_llm::ocr::error::Error;
use litellm_llms::base_llm::ocr::{error::Error, transformation::DROP_PARAMS_PARAM};
use pyo3::{prelude::*, types::PyDict};
use serde_json::{Map, Value};
@ -19,6 +19,8 @@ use crate::{
python_settings::{PythonSettings, Snapshot},
};
pub(super) const ROUTE_HOST_MODULE: &str = "litellm.rust_bridge.host.routes.ocr";
/// What the host keeps after projection: the caller's token callable that answers the
/// token operation, and the provider name the failure mapping reports.
pub(super) struct OcrHostHandles {
@ -139,6 +141,15 @@ pub(super) fn project_request(
Some(defaults) => fold_module_globals(bound.py(), optional_params, &globals, &defaults)?,
None => optional_params,
};
let optional_params = if names.contains(&DROP_PARAMS_PARAM) {
fold_drop_params(
bound.py(),
optional_params,
present(kwargs, bound, DROP_PARAMS_PARAM)?,
)?
} else {
optional_params
};
let input_sources = request_input_sources(
kwargs,
names
@ -216,6 +227,23 @@ fn fold_module_globals(
.collect())
}
fn fold_drop_params(
py: Python<'_>,
optional_params: Map<String, Value>,
requested: Option<Bound<'_, PyAny>>,
) -> PyResult<Map<String, Value>> {
let drops: bool = py
.import(ROUTE_HOST_MODULE)?
.getattr("drops_unsupported_params")?
.call1((requested,))?
.extract()?;
Ok(optional_params
.into_iter()
.filter(|(name, _)| name != DROP_PARAMS_PARAM)
.chain(drops.then(|| (DROP_PARAMS_PARAM.to_string(), Value::Bool(true))))
.collect())
}
/// A misconfigured litellm setting is the operator's error, not the request's or the
/// provider's, so the host raises it as is instead of mapping it onto a provider failure.
pub(super) const SETTINGS_ERROR_MARKER: &str = "native_settings_error";

5
litellm/__init__.py
View file

@ -681,6 +681,7 @@ moonshot_models: Set = set()
publicai_models: Set = set()
darkbloom_models: Set = set()
openinfer_models: Final[Set[str]] = set()
streamlake_models: Final[Set[str]] = set()
zerogpu_models: Final[Set[str]] = set()
v0_models: Set = set()
morph_models: Set = set()
@ -970,6 +971,8 @@ def _populate_provider_model_sets(model_cost_map: Mapping[str, Mapping[str, obje
darkbloom_models.add(key)
elif value.get("litellm_provider") == "openinfer":
openinfer_models.add(key)
elif value.get("litellm_provider") == "streamlake":
streamlake_models.add(key)
elif value.get("litellm_provider") == "zerogpu":
zerogpu_models.add(key)
elif value.get("litellm_provider") == "v0":
@ -1145,6 +1148,7 @@ model_list = list(
| publicai_models
| darkbloom_models
| openinfer_models
| streamlake_models
| zerogpu_models
| v0_models
| morph_models
@ -1265,6 +1269,7 @@ def _build_models_by_provider() -> dict:
"publicai": publicai_models,
"darkbloom": darkbloom_models,
"openinfer": openinfer_models,
"streamlake": streamlake_models,
"zerogpu": zerogpu_models,
"v0": v0_models,
"morph": morph_models,

2
litellm/constants.py
View file

@ -1064,6 +1064,7 @@ openai_compatible_endpoints: Final[list[str]] = [
"https://api.reka.ai/v1",
"https://gigachat.devices.sberbank.ru/api/v1",
"https://api.openinfer.ai/v1",
"https://vanchin.streamlake.ai/api/gateway/v1/endpoints",
]
@ -1145,6 +1146,7 @@ openai_compatible_providers: Final[list[str]] = [
"prism",
"sail",
"reka",
"streamlake",
]
OPENAI_AUDIO_TRANSCRIPTION_PROVIDERS: Final = frozenset({"openai"} | frozenset(openai_compatible_providers))

2
litellm/cost_calculator.py
View file

@ -688,7 +688,7 @@ def cost_per_token(
elif call_type == "atranscription" or call_type == "transcription":
if _transcription_usage_has_token_details(usage_block):
return generic_cost_per_token(
model=model_without_prefix,
model=model,
usage=usage_block,
custom_llm_provider=custom_llm_provider,
service_tier=service_tier,

7
litellm/google_genai/adapters/transformation.py
View file

@ -198,6 +198,11 @@ def _genai_message_part(part: object) -> _GenAIMessagePart | None:
return ChatCompletionTextObject(type="text", text=text) if isinstance(text, str) else _genai_media_part(fields)
def _genai_content_parts(value: object) -> Sequence[object]:
single_part: Final = _validated(_CONFIG_FIELDS, value)
return (single_part,) if single_part is not None else _validated(_GENAI_CONTENT_PARTS, value) or ()
def _genai_message_parts(parts: object) -> tuple[_GenAIMessagePart, ...]:
items: Final = _validated(_GENAI_CONTENT_PARTS, parts) or ()
return tuple(converted for part in items if (converted := _genai_message_part(part)) is not None)
@ -248,7 +253,7 @@ def _genai_content_messages(content: object) -> tuple[AllMessageValues, ...]:
fields: Final = _validated(_CONFIG_FIELDS, content)
if fields is None:
return ()
parts: Final = _validated(_GENAI_CONTENT_PARTS, fields.get("parts", ())) or ()
parts: Final = _genai_content_parts(fields.get("parts", ()))
converted: Final = _genai_message_parts(parts)
role: Final = fields.get("role", "user")
if role == "user":

1
litellm/litellm_core_utils/get_litellm_params.py
View file

@ -77,6 +77,7 @@ OPTIONAL_KWARGS_KEYS: Final = (
"otpm",
"tpd",
"rpd",
"daily_reset_time",
"use_xai_oauth",
"chatgpt_auth_file",
"github_copilot_auth_type",

37
litellm/litellm_core_utils/litellm_logging.py
View file

@ -1130,9 +1130,6 @@ class Logging(LiteLLMLoggingBaseClass):
}
self.litellm_request_debug = litellm_params.get("litellm_request_debug", False)
self.logger_fn = litellm_params.get("logger_fn", None)
if is_debugging_on() or self.litellm_request_debug:
verbose_logger.debug("self.optional_params: %s", self.optional_params)
self.model_call_details.update(
{
"model": self.model,
@ -5055,6 +5052,23 @@ class Logging(LiteLLMLoggingBaseClass):
return result_copy
_SENSITIVE_KEYWORDS: Final = (
"authorization",
"token",
"key",
"secret",
"vertex_credentials",
"credentials",
"password",
"passwd",
)
def has_sensitive_keyword(key: str) -> bool:
lowered: Final = key.lower()
return any(keyword in lowered for keyword in _SENSITIVE_KEYWORDS)
def get_masked_values(
sensitive_object: Mapping[str, object],
ignore_sensitive_values: bool = False,
@ -5073,16 +5087,6 @@ def get_masked_values(
masked_length: Optional length for the masked portion (number of *). If set, will use exactly this many *
regardless of original string length. The total length will be unmasked_length + masked_length.
"""
sensitive_keywords: Final = (
"authorization",
"token",
"key",
"secret",
"vertex_credentials",
"credentials",
"password",
"passwd",
)
def _mask_value(v: object) -> object:
if isinstance(v, dict):
@ -5109,12 +5113,7 @@ def get_masked_values(
return v[: unmasked_length // 2] + "*" * (len(v) - unmasked_length) + v[-unmasked_length // 2 :]
return {
k: (
v
if ignore_sensitive_values
or not any(sensitive_keyword in k.lower() for sensitive_keyword in sensitive_keywords)
else _mask_value(v)
)
k: (v if ignore_sensitive_values or not has_sensitive_keyword(k) else _mask_value(v))
for k, v in sensitive_object.items()
}

11
litellm/litellm_core_utils/llm_cost_calc/usage_object_transformation.py
View file

@ -27,13 +27,18 @@ class TranscriptionUsageObjectTransformation:
if isinstance(usage_object, TranscriptionUsageDurationObject):
return None
elif isinstance(usage_object, TranscriptionUsageTokensObject):
input_token_details: Final = usage_object.input_token_details
return Usage(
prompt_tokens=usage_object.input_tokens,
completion_tokens=usage_object.output_tokens,
total_tokens=usage_object.total_tokens,
prompt_tokens_details=PromptTokensDetailsWrapper(
text_tokens=usage_object.input_token_details.text_tokens,
audio_tokens=usage_object.input_token_details.audio_tokens,
prompt_tokens_details=(
None
if input_token_details is None
else PromptTokensDetailsWrapper(
text_tokens=input_token_details.text_tokens,
audio_tokens=input_token_details.audio_tokens,
)
),
)
return None

14
litellm/litellm_core_utils/streaming_handler.py
View file

@ -92,6 +92,18 @@ def _next_sync_or_exhausted(it: Iterator[object]) -> object:
return _SYNC_ITER_EXHAUSTED
def _stamp_usage_cost(
usage_chunk: ModelResponse | TextCompletionResponse | ModelResponseStream,
complete_response: ModelResponse | TextCompletionResponse | ModelResponseStream,
) -> None:
bill: Final = (get_hidden_params(complete_response) or {}).get("response_cost")
usage: Final[object] = getattr(usage_chunk, "usage", None)
if usage is None or isinstance(bill, bool) or not isinstance(bill, (int, float)):
return
if bill > 0 or isinstance(getattr(usage, "cost", None), (int, float)):
setattr(usage, "cost", float(bill)) # noqa: B010 # cost is an optional extra on the usage model
def _stamp_served_service_tier(response: ModelResponseStream, complete_streaming_response: ModelResponse) -> None:
served_tier: Final = complete_streaming_response.model_dump().get("service_tier")
if isinstance(served_tier, str) and served_tier:
@ -2141,6 +2153,7 @@ class CustomStreamWrapper:
getattr(complete_streaming_response, "usage"),
)
_stamp_served_service_tier(response, complete_streaming_response)
_stamp_usage_cost(response, complete_streaming_response)
try:
_cache_copy = complete_streaming_response.model_copy(deep=True)
_log_copy = complete_streaming_response.model_copy(deep=True)
@ -2361,6 +2374,7 @@ class CustomStreamWrapper:
getattr(complete_streaming_response, "usage"),
)
_stamp_served_service_tier(response, complete_streaming_response)
_stamp_usage_cost(response, complete_streaming_response)
try:
_copy = complete_streaming_response.model_copy(deep=True)
except RuntimeError:

40
litellm/llms/anthropic/messages/context_management/editors/clear_tool_uses.py
View file

@ -31,6 +31,7 @@ from ..constants import (
)
from ..errors import AnthropicContextManagementError
from ..placeholders import build_cleared_tool_result_content
from .compact import count_system_tokens, history_after_client_compaction
if TYPE_CHECKING:
from litellm.types.llms.openai import ChatCompletionToolParam
@ -203,9 +204,10 @@ def _trigger_met(
model: str,
messages: list[dict[str, object]],
tools: list[dict[str, object]] | None,
system: str | Sequence[Mapping[str, object]] | None,
counted_tool_uses: int,
) -> tuple[bool, int | None]:
"""Return (trigger_met, input_tokens if counted for reuse)."""
"""Return (trigger_met, message and tool tokens for reuse); like the API, input_tokens adds the system prompt."""
trigger_type: Final = trigger.get("type", "input_tokens")
threshold = trigger.get("value")
@ -216,14 +218,17 @@ def _trigger_met(
if not isinstance(threshold, int):
threshold = DEFAULT_INPUT_TOKENS_TRIGGER
current_tokens: Final = litellm.token_counter(
message_tokens: Final = litellm.token_counter(
model=model,
messages=messages,
tools=cast("list[ChatCompletionToolParam] | None", tools),
)
if message_tokens > threshold:
return True, message_tokens
current_tokens: Final = message_tokens + count_system_tokens(model, system)
verbose_logger.debug("context_management polyfill: current_tokens: %s", current_tokens)
verbose_logger.debug("context_management polyfill: threshold: %s", threshold)
return current_tokens > threshold, current_tokens
return current_tokens > threshold, message_tokens
def _resolve_keep_count(keep: dict[str, object]) -> int:
@ -237,7 +242,7 @@ def _resolve_keep_count(keep: dict[str, object]) -> int:
def _last_completed_tool_use_id(
messages: list[dict[str, object]],
messages: Sequence[Mapping[str, object]],
) -> str | None:
"""Latest completed tool_result id; never cleared."""
last_id: str | None = None
@ -253,7 +258,7 @@ def _last_completed_tool_use_id(
def _clear_tool_results(
messages: list[dict[str, object]], ids_to_clear: set[str]
messages: list[dict[str, object]], ids_to_clear: Set[str]
) -> tuple[list[dict[str, object]], int]:
"""Clear matching tool_result content; return (messages, cleared_count)."""
cleared = 0
@ -311,7 +316,11 @@ def apply_clear_tool_uses_20250919(
system: str | list[dict[str, object]] | None,
edit_spec: ClearToolUsesEditSpec,
) -> tuple[list[dict[str, object]], AppliedEdit | None]:
"""Apply clear_tool_uses to an edit ``check_clear_tool_uses_spec`` returned; return (messages, AppliedEdit or None)."""
"""Apply clear_tool_uses to an edit ``check_clear_tool_uses_spec`` returned; return (messages, AppliedEdit or None).
Like the API, it reads only the history after a client compaction block and counts the block's summary as system.
"""
kept, counted_system = history_after_client_compaction(messages, system)
fields: Final = _ClearToolUsesFields.model_validate(edit_spec)
clear_at_least: Final = 0 if fields.clear_at_least is None else fields.clear_at_least.value
exclude_tools: Final = frozenset(fields.exclude_tools or ())
@ -326,7 +335,7 @@ def apply_clear_tool_uses_20250919(
"value": DEFAULT_KEEP_TOOL_USES,
}
blocks: Final = _all_blocks(messages)
blocks: Final = _all_blocks(kept)
results: Final = _blocks_by_id(blocks, "tool_result", "tool_use_id")
uses: Final = _blocks_by_id(blocks, "tool_use", "id")
tiny: Final = _tiny_result_ids(results)
@ -335,15 +344,15 @@ def apply_clear_tool_uses_20250919(
for tool_use_id, name in _tool_uses_in_order(blocks)
if name not in exclude_tools and (tool_use_id not in tiny or _clears_input(name, input_tools))
)
met, counted_tokens = _trigger_met(trigger, model, messages, tools, len(clearable))
if not met:
return messages, None
keep_count: Final = _resolve_keep_count(keep)
if len(clearable) <= keep_count:
return messages, None
last_completed_id: Final = _last_completed_tool_use_id(messages)
met, counted_tokens = _trigger_met(trigger, model, kept, tools, counted_system, len(clearable))
if not met:
return messages, None
last_completed_id: Final = _last_completed_tool_use_id(kept)
candidates: Final = tuple(
(tool_use_id, name)
for tool_use_id, name in clearable[: len(clearable) - keep_count]
@ -358,7 +367,7 @@ def apply_clear_tool_uses_20250919(
_select_in_batches(candidate_ids, tokens, clear_at_least) if clear_at_least > 0 else candidate_ids
)
edited, cleared_count = _clear_tool_results(messages, set(ids_to_clear))
edited, cleared_count = _clear_tool_results(messages, ids_to_clear)
verbose_logger.debug("context_management polyfill: edited: %s", edited)
if cleared_count == 0:
return messages, None
@ -370,11 +379,12 @@ def apply_clear_tool_uses_20250919(
counted_tokens
if counted_tokens is not None
else litellm.token_counter(
model=model, messages=messages, tools=cast("list[ChatCompletionToolParam] | None", tools)
model=model, messages=kept, tools=cast("list[ChatCompletionToolParam] | None", tools)
)
)
kept_after, _ = history_after_client_compaction(final_messages, system)
tokens_after: Final = litellm.token_counter(
model=model, messages=final_messages, tools=cast("list[ChatCompletionToolParam] | None", tools)
model=model, messages=kept_after, tools=cast("list[ChatCompletionToolParam] | None", tools)
)
cleared_input_tokens: Final = max(tokens_before - tokens_after, 0)

27
litellm/llms/anthropic/messages/context_management/editors/compact.py
View file

@ -189,6 +189,21 @@ def _strip_compaction_blocks(
return cleaned
def history_after_client_compaction(
messages: Sequence[dict[str, object]],
system: str | Sequence[Mapping[str, object]] | None,
) -> tuple[list[dict[str, object]], str | Sequence[Mapping[str, object]] | None]:
"""Return the history the API keeps after the latest compaction block, without it, and the system with its summary."""
sliced, block = _slice_around_compaction_block(messages)
if block is None:
return list(messages), system
tail: Final = _strip_compaction_blocks(sliced)
summary: Final = block.get("content")
if not isinstance(summary, str) or not summary:
return tail, system
return tail, _augment_system_with_summary(_system_to_text(system) or None, summary)
def _augment_system_with_summary(
system: str | list[dict[str, object]] | None,
summary_text: str,
@ -320,14 +335,16 @@ def _count_effective_tokens(
content: Final = compaction_block.get("content") or ""
if content:
total += litellm.token_counter(model=model, text=content)
return total + count_system_tokens(model, system)
def count_system_tokens(model: str, system: str | Sequence[Mapping[str, object]] | None) -> int:
system_text: Final = _system_to_text(system)
if system_text:
total += litellm.token_counter(model=model, text=system_text)
return total
return litellm.token_counter(model=model, text=system_text) if system_text else 0
def _system_to_text(
system: str | list[dict[str, object]] | None,
system: str | Sequence[Mapping[str, object]] | None,
) -> str:
"""Flatten an Anthropic-style ``system`` value into a single string for
token counting. Returns ``""`` when ``system`` carries no text."""
@ -338,7 +355,7 @@ def _system_to_text(
return "\n".join(
text
for block in system
if isinstance(block, dict)
if isinstance(block, Mapping)
and block.get("type") == "text"
and isinstance(text := block.get("text"), str)
and text

8
litellm/llms/azure/common_utils.py
View file

@ -621,7 +621,7 @@ class BaseAzureLLM(BaseOpenAILLM):
# For Azure v1 API, use standard OpenAI client instead of AzureOpenAI
# See: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#api-specs
if self._is_azure_v1_api_version(api_version):
if self.is_azure_v1_api_version(api_version):
# Extract only params that OpenAI client accepts
# Always use /openai/v1/ regardless of whether user passed "v1", "latest", or "preview"
# The OpenAI client accepts a callable for `api_key` and re-invokes it
@ -922,7 +922,7 @@ class BaseAzureLLM(BaseOpenAILLM):
else:
new_url = api_base
if BaseAzureLLM._is_azure_v1_api_version(api_version):
if BaseAzureLLM.is_azure_v1_api_version(api_version):
# ensure the request go to /openai/v1 and not just /openai
if "/openai/v1" not in new_url:
parsed_url: Final = httpx.URL(new_url)
@ -952,7 +952,7 @@ class BaseAzureLLM(BaseOpenAILLM):
Returns None when ``api_version`` is a dated one, which still uses the deployment route.
"""
if not BaseAzureLLM._is_azure_v1_api_version(api_version):
if not BaseAzureLLM.is_azure_v1_api_version(api_version):
return None
base_url: Final = httpx.URL(api_base)
@ -970,7 +970,7 @@ class BaseAzureLLM(BaseOpenAILLM):
)
@staticmethod
def _is_azure_v1_api_version(api_version: str | None) -> bool:
def is_azure_v1_api_version(api_version: str | None) -> bool:
if api_version is None:
return False
return api_version in {"preview", "latest", "v1"}

34
litellm/llms/azure/responses/transformation.py
View file

@ -54,6 +54,16 @@ def _resolve_responses_api_version(api_base: str | None, litellm_params: Mapping
return api_version
def _v1_responses_url(api_base: str) -> httpx.URL:
parsed_url: Final = httpx.URL(api_base)
path: Final = parsed_url.path.rstrip("/")
root: Final = next(
(path.removesuffix(route) for route in ("/openai/v1/responses", "/openai/responses") if path.endswith(route)),
path,
)
return parsed_url.copy_with(path=f"{root}/openai/v1/responses", query=None)
def _with_responses_api_version(api_base: str | None, litellm_params: Mapping[str, object]) -> Mapping[str, object]:
return MappingProxyType({**litellm_params, "api_version": _resolve_responses_api_version(api_base, litellm_params)})
@ -132,7 +142,9 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
return filtered_item
return item
def _validate_input_param(self, input: str | ResponseInputParam) -> str | ResponseInputParam:
def _validate_input_param(
self, input: str | ResponseInputParam, *, drops_set_created_by: bool = False
) -> str | ResponseInputParam:
"""
Override parent method to also filter out 'status' field from message items.
Azure OpenAI API does not accept 'status' field in input messages.
@ -140,7 +152,7 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
from typing import cast
# First call parent's validation
validated_input: Final = super()._validate_input_param(input)
validated_input: Final = super()._validate_input_param(input, drops_set_created_by=drops_set_created_by)
# Then filter out status from message items
if isinstance(validated_input, list):
@ -247,20 +259,20 @@ class AzureOpenAIResponsesAPIConfig(OpenAIResponsesAPIConfig):
if api_base is None:
raise ValueError("api_base is required for Azure WebSocket")
parsed_url: Final = httpx.URL(api_base)
path = parsed_url.path.rstrip("/")
# Strip existing /openai/responses path if the api_base already contains it
for suffix in ("/openai/v1/responses", "/openai/responses"):
if path.endswith(suffix):
path = path[: -len(suffix)]
break
scheme: Final = "wss" if parsed_url.scheme == "https" else "ws"
return str(parsed_url.copy_with(scheme=scheme, path=f"{path}/openai/v1/responses", query=None))
v1_url: Final = _v1_responses_url(api_base)
return str(v1_url.copy_with(scheme="wss" if v1_url.scheme == "https" else "ws"))
def model_in_websocket_url(self) -> bool:
# Azure sends the model in the response.create body, not the URL
return False
def compact_url(self, api_base: str) -> str:
params: Final = httpx.URL(api_base).params
v1_params: Final = (
params if BaseAzureLLM.is_azure_v1_api_version(params.get("api-version")) else params.remove("api-version")
)
return super().compact_url(str(_v1_responses_url(api_base).copy_with(params=v1_params)))
#########################################################
########## DELETE RESPONSE API TRANSFORMATION ##############
#########################################################

9
litellm/llms/azure_ai/common_utils.py
View file

@ -140,6 +140,15 @@ def get_azure_ai_auth_headers(
AZURE_MODEL_ROUTER_SELECTED_MODEL_KEY: Final = "azure_model_router_selected_model"
FOUNDRY_CLASSIC_AGENT_ID_PREFIX: Final = "asst_"
def is_foundry_project_agent(model: str | None) -> bool:
if model is None or AzureFoundryModelInfo.get_azure_ai_route(model) != "agents":
return False
return not model.split("agents/", 1)[1].startswith(FOUNDRY_CLASSIC_AGENT_ID_PREFIX)
def azure_ai_supports_native_responses(model: str | None, api_base: str | None) -> bool:
resolved_base: Final = AzureFoundryModelInfo.get_api_base(api_base)
if resolved_base is not None and not is_azure_openai_v1_host(resolved_base):

263
litellm/llms/azure_ai/responses/transformation.py
View file

@ -1,26 +1,64 @@
from typing import Final
import os
from collections.abc import Collection, Mapping, Sequence
from typing import TYPE_CHECKING, Final, NoReturn
import httpx
from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
import litellm
from litellm.exceptions import UnsupportedParamsError
from litellm.llms.azure.responses.transformation import AzureOpenAIResponsesAPIConfig
from litellm.llms.azure_ai.chat.transformation import AzureAIGPT5Config
from litellm.llms.azure_ai.common_utils import (
AZURE_AI_AGENTS_SCOPE,
AzureFoundryModelInfo,
api_key_header_for_base,
get_azure_ai_auth_headers,
)
from litellm.types.llms.azure_ai import FoundryAgentReference
from litellm.types.llms.openai import (
ResponseInputParam,
ResponsesAPIOptionalRequestParams,
ResponsesAPIResponse,
ResponsesAPIStreamingResponse,
)
from litellm.types.router import GenericLiteLLMParams
from litellm.types.utils import LlmProviders
if TYPE_CHECKING:
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
_PROJECT_PATH_PREFIX: Final = ("api", "projects")
_RESPONSES_PATH: Final = ("openai", "v1", "responses")
_AGENT_OWNED_PARAMS: Final = frozenset({"temperature", "top_p", "tools", "text", "reasoning"})
_CALLER_AGENT_KEYS: Final = ("agent", "agent_reference")
_RESPONSE_FIELDS: Final = TypeAdapter(dict[str, object])
def _project_root(base_url: httpx.URL) -> tuple[str, ...]:
segments: Final = tuple(segment for segment in base_url.path.split("/") if segment)
return segments[:3] if segments[:2] == _PROJECT_PATH_PREFIX else ()
def _responses_url(api_base: str) -> str:
base_url: Final = httpx.URL(api_base)
segments: Final = tuple(segment for segment in base_url.path.split("/") if segment)
project_root: Final = segments[:3] if segments[:2] == _PROJECT_PATH_PREFIX else ()
return str(base_url.copy_with(path="/" + "/".join((*project_root, *_RESPONSES_PATH)), query=None))
return str(base_url.copy_with(path="/" + "/".join((*_project_root(base_url), *_RESPONSES_PATH)), query=None))
def _without_agent_instructions(response: ResponsesAPIResponse) -> ResponsesAPIResponse:
if (response.model_extra or {}).get("agent_reference") is None:
return response
return response.model_copy(update={"instructions": None})
def _event_without_agent_instructions(event: Mapping[str, object]) -> Mapping[str, object]:
response: Final = event.get("response")
if not isinstance(response, Mapping):
return event
fields: Final = _RESPONSE_FIELDS.validate_python(response)
if fields.get("agent_reference") is None:
return event
return {**event, "response": {**fields, "instructions": None}}
class AzureAIResponsesAPIConfig(AzureOpenAIResponsesAPIConfig):
@ -42,10 +80,19 @@ class AzureAIResponsesAPIConfig(AzureOpenAIResponsesAPIConfig):
def validate_environment(self, headers: dict, model: str, litellm_params: GenericLiteLLMParams | None) -> dict:
params: Final = litellm_params or GenericLiteLLMParams()
api_base: Final = AzureFoundryModelInfo.get_api_base(params.api_base)
on_project: Final = api_base is not None and bool(_project_root(httpx.URL(api_base)))
auth_params: Final = (
params.model_copy(
update={"azure_scope": params.azure_scope or os.getenv("AZURE_SCOPE") or AZURE_AI_AGENTS_SCOPE}
)
if on_project
else params
)
auth_headers: Final = get_azure_ai_auth_headers(
api_key=AzureFoundryModelInfo.get_api_key(params.api_key),
litellm_params=params.model_dump(),
api_key_header=api_key_header_for_base(AzureFoundryModelInfo.get_api_base(params.api_base)),
litellm_params=auth_params.model_dump(),
api_key_header="Authorization" if on_project else api_key_header_for_base(api_base),
)
return {
**headers,
@ -64,3 +111,207 @@ class AzureAIResponsesAPIConfig(AzureOpenAIResponsesAPIConfig):
"Set the api_base parameter or the AZURE_AI_API_BASE environment variable."
)
return _responses_url(resolved_base)
def transform_response_api_response(
self, model: str, raw_response: httpx.Response, logging_obj: "LiteLLMLoggingObj"
) -> ResponsesAPIResponse:
return _without_agent_instructions(super().transform_response_api_response(model, raw_response, logging_obj))
def transform_streaming_response(
self,
model: str,
parsed_chunk: Mapping[str, object],
logging_obj: "LiteLLMLoggingObj",
) -> ResponsesAPIStreamingResponse:
return super().transform_streaming_response( # pyright: ignore[reportUnknownMemberType] # parent types parsed_chunk as a bare dict
model=model, parsed_chunk=dict(_event_without_agent_instructions(parsed_chunk)), logging_obj=logging_obj
)
def transform_get_response_api_response(
self, raw_response: httpx.Response, logging_obj: "LiteLLMLoggingObj"
) -> ResponsesAPIResponse:
return _without_agent_instructions(super().transform_get_response_api_response(raw_response, logging_obj))
def transform_cancel_response_api_response(
self, raw_response: httpx.Response, logging_obj: "LiteLLMLoggingObj"
) -> ResponsesAPIResponse:
return _without_agent_instructions(super().transform_cancel_response_api_response(raw_response, logging_obj))
def _agent_reference(model: str) -> FoundryAgentReference:
name, _, version = model.split("agents/", 1)[1].partition(":")
if version:
return FoundryAgentReference(type="agent_reference", name=name, version=version)
return FoundryAgentReference(type="agent_reference", name=name)
class _CallerAgentReference(BaseModel):
model_config = ConfigDict(frozen=True, coerce_numbers_to_str=True)
name: str
version: str | None = None
def _caller_agent_reference(value: object) -> _CallerAgentReference | None:
try:
return _CallerAgentReference.model_validate(value)
except ValidationError:
return None
def _requested_agent(
deployment_agent: FoundryAgentReference, caller_agents: Sequence[_CallerAgentReference | None]
) -> FoundryAgentReference:
requested_versions: Final = frozenset(
agent.version
for agent in caller_agents
if agent is not None and agent.name == deployment_agent["name"] and agent.version is not None
)
if "version" in deployment_agent or len(requested_versions) != 1:
return deployment_agent
return FoundryAgentReference(
type="agent_reference", name=deployment_agent["name"], version=next(iter(requested_versions))
)
def _caller_agent_fields(fields: Mapping[str, object]) -> tuple[tuple[str, object], ...]:
return tuple((key, fields[key]) for key in _CALLER_AGENT_KEYS if key in fields)
def _deployment_agent(
model: str, caller_fields: Sequence[tuple[str, object]], litellm_params: GenericLiteLLMParams
) -> FoundryAgentReference:
caller_keys: Final = tuple(key for key, _ in caller_fields)
caller_agents: Final = tuple(_caller_agent_reference(value) for _, value in caller_fields)
agent: Final = _requested_agent(_agent_reference(model), caller_agents)
other_agents: Final = tuple(
key
for key, caller_agent in zip(caller_keys, caller_agents)
if caller_agent is None
or caller_agent.name != agent["name"]
or caller_agent.version not in (None, agent.get("version"))
)
if other_agents and not (litellm.drop_params is True or litellm_params.drop_params is True):
raise UnsupportedParamsError(
message=(
f"This model entry always calls Foundry agent {agent['name']!r}, so `{other_agents[0]}` "
"in the request cannot pick another agent or version. Remove it, call the model entry for that "
"agent, or set `drop_params: true` to drop it"
),
llm_provider=LlmProviders.AZURE_AI.value,
model=model,
)
return agent
_INPUT_ITEMS: Final = TypeAdapter(list[object])
def _with_system_prompt(instructions: str, request_input: object) -> Sequence[object]:
conversation: Final = (
[{"type": "message", "role": "user", "content": request_input}]
if isinstance(request_input, str)
else _INPUT_ITEMS.validate_python(request_input)
)
return [{"type": "message", "role": "system", "content": instructions}, *conversation]
class AzureAIAgentResponsesAPIConfig(AzureAIResponsesAPIConfig):
def should_fake_stream(
self, model: str | None, stream: bool | None, custom_llm_provider: str | None = None
) -> bool:
return False
def get_supported_openai_params(self, model: str) -> list[str]: # mutable-ok: override contract
inherited: Final[list[str]] = super().get_supported_openai_params(model) # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType] # parent returns a bare list # mutable-ok: same
return [param for param in inherited if param not in _AGENT_OWNED_PARAMS]
def map_openai_params_for_deployment(
self,
response_api_optional_params: ResponsesAPIOptionalRequestParams,
model: str,
drop_params: bool,
*,
declared_reasoning_efforts: Collection[str],
) -> Mapping[str, object]:
mapped: Final = super().map_openai_params_for_deployment(
response_api_optional_params, model, drop_params, declared_reasoning_efforts=declared_reasoning_efforts
)
resends_instructions: Final = bool(mapped.get("instructions")) and bool(mapped.get("previous_response_id"))
if resends_instructions and not drop_params:
raise UnsupportedParamsError(
message=(
"azure_ai Foundry agents store a turn's system message with the conversation, so `instructions` "
"with `previous_response_id` would add another copy each turn. Send `instructions` on the first "
"turn only, or set `drop_params: true` to drop it"
),
llm_provider=LlmProviders.AZURE_AI.value,
model=model,
)
dropped: Final = _AGENT_OWNED_PARAMS | (
frozenset({"instructions"}) if resends_instructions else frozenset[str]()
)
return {key: value for key, value in mapped.items() if key not in dropped}
def transform_responses_api_request(
self,
model: str,
input: str | ResponseInputParam,
response_api_optional_request_params: dict[str, object], # mutable-ok: signature fixed by the override contract
litellm_params: GenericLiteLLMParams,
headers: dict[str, str], # mutable-ok: same
) -> dict[str, object]: # mutable-ok: same
request: Final[dict[str, object]] = super().transform_responses_api_request( # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType] # parent returns a bare dict # mutable-ok: same
model=model,
input=input,
response_api_optional_request_params=response_api_optional_request_params,
litellm_params=litellm_params,
headers=headers,
)
instructions: Final = request.get("instructions")
return {
**{key: value for key, value in request.items() if key not in ("model", "instructions")},
**(
{"input": _with_system_prompt(instructions, request.get("input"))}
if isinstance(instructions, str) and instructions
else {}
),
"agent_reference": _deployment_agent(
model, _caller_agent_fields(litellm_params.model_extra or {}), litellm_params
),
}
def transform_extra_body(
self,
extra_body: Mapping[str, object],
request: Mapping[str, object],
model: str,
litellm_params: GenericLiteLLMParams,
) -> Mapping[str, object]:
passthrough: Final = {key: value for key, value in extra_body.items() if key not in _CALLER_AGENT_KEYS}
if len(passthrough) == len(extra_body):
return passthrough
return {
**passthrough,
"agent_reference": _deployment_agent(
model, (*_caller_agent_fields(request), *_caller_agent_fields(extra_body)), litellm_params
),
}
def transform_compact_response_api_request(
self,
model: str,
input: str | ResponseInputParam,
response_api_optional_request_params: dict[str, object], # mutable-ok: signature fixed by the override contract
api_base: str,
litellm_params: GenericLiteLLMParams,
headers: dict[str, str], # mutable-ok: same
) -> NoReturn:
raise UnsupportedParamsError(
message=(
"azure_ai Foundry agents cannot be compacted, because the project's /responses/compact endpoint "
"takes a model deployment, not an agent. Call compact on a model entry for that deployment instead"
),
llm_provider=LlmProviders.AZURE_AI.value,
model=model,
)

75
litellm/llms/bedrock/audio_transcription/__init__.py
View file

@ -1,7 +1,11 @@
import base64
import math
from collections.abc import Mapping
from dataclasses import dataclass
from typing import Final
import httpx
from pydantic import BaseModel, ConfigDict
from litellm.litellm_core_utils.audio_utils.utils import process_audio_file
from litellm.rust_bridge import runtime
@ -14,7 +18,72 @@ from litellm.rust_bridge.transcription.native import (
RustAtranscription,
RustTranscription,
)
from litellm.types.utils import FileTypes, TranscriptionResponse
from litellm.secret_managers.main import get_secret_str
from litellm.types.utils import FileTypes, TranscriptionResponse, TranscriptionUsageTokensObject
SUPPORTED_OPENAI_TRANSCRIPTION_PARAMS: Final = ("language", "prompt", "temperature", "response_format")
TEXT_RESPONSE_FORMATS: Final = frozenset({"json", "text"})
def _as_float(value: object) -> float | None:
if isinstance(value, bool) or not isinstance(value, (int, float, str)):
return None
try:
number: Final = float(value)
except ValueError:
return None
return number if math.isfinite(number) else None
def _rejection(key: str, value: object) -> str | None:
if key not in SUPPORTED_OPENAI_TRANSCRIPTION_PARAMS:
return f"bedrock transcription does not support {key}."
if key == "response_format" and value not in TEXT_RESPONSE_FORMATS:
return f"bedrock transcription returns only the transcript text, so response_format={value!r} is not supported."
if key == "temperature" and _as_float(value) is None:
return f"bedrock transcription needs a number for temperature, got {value!r}."
return None
@dataclass(frozen=True, slots=True)
class UnsupportedTranscriptionParams:
message: str
def map_openai_transcription_params(
non_default_params: Mapping[str, object], drop_params: bool
) -> Mapping[str, object] | UnsupportedTranscriptionParams:
rejections: Final = {
key: reason for key, value in non_default_params.items() if (reason := _rejection(key, value)) is not None
}
if rejections and not drop_params:
return UnsupportedTranscriptionParams(
message=f"{' '.join(rejections.values())} To drop it from the call, set `litellm.drop_params = True`."
)
return {
key: _as_float(value) if key == "temperature" else value
for key, value in non_default_params.items()
if key not in rejections
}
class _RustTranscriptionResult(BaseModel):
model_config = ConfigDict(frozen=True)
text: str
usage: TranscriptionUsageTokensObject | None = None
def _transcription_response(
result: Mapping[str, object], optional_params: Mapping[str, object]
) -> TranscriptionResponse:
parsed: Final = _RustTranscriptionResult.model_validate(result)
response: Final = TranscriptionResponse(text=parsed.text).model_copy(update={"usage": parsed.usage})
region_name: Final = (
optional_params.get("aws_region_name") or get_secret_str("AWS_REGION_NAME") or get_secret_str("AWS_REGION")
)
response.hidden_params = {**response.hidden_params, "region_name": region_name}
return response
class BedrockAudioTranscriptionRustDispatch:
@ -64,7 +133,7 @@ class BedrockAudioTranscriptionRustDispatch:
"timeout_seconds": timeout_to_seconds(timeout),
}
call: Final = NativeCall(args=(), kwargs=fields, base={})
return TranscriptionResponse(**rust(call))
return _transcription_response(rust(call), optional_params)
return runtime.run(
RouteContext(Route.TRANSCRIPTION, provider=custom_llm_provider, model=model),
@ -97,7 +166,7 @@ class BedrockAudioTranscriptionRustDispatch:
"timeout_seconds": timeout_to_seconds(timeout),
}
call: Final = NativeCall(args=(), kwargs=fields, base={})
return TranscriptionResponse(**await rust(call))
return _transcription_response(await rust(call), optional_params)
return await runtime.arun(
RouteContext(Route.TRANSCRIPTION, provider=custom_llm_provider, model=model),

13
litellm/llms/openai/responses/count_tokens/transformation.py
View file

@ -22,7 +22,7 @@ from litellm.llms.anthropic.messages.responses_adapters.transformation import (
)
from litellm.responses.utils import ResponsesAPIRequestUtils
from litellm.types.llms.anthropic import AnthropicMessagesRequest, AnthropicOutputSchema
from litellm.types.llms.openai import ResponsesAPIRequestParams
from litellm.types.llms.openai import ResponsesAPIRequestParams, replayable_responses_input
from litellm.types.utils import all_litellm_params
@ -387,10 +387,13 @@ def _forwards(field: str, policy: DropPolicy) -> bool:
return field in _COUNT_REQUEST_FIELDS or (not policy.drop_params and field in _INFERENCE_REQUEST_FIELDS)
def _restored(field: str, value: object) -> object:
def _restored(field: str, value: object, policy: DropPolicy) -> object:
match field, value:
case "input", _:
return ResponsesAPIRequestUtils.restore_encrypted_content_item_ids_in_input(copy.deepcopy(value))
return replayable_responses_input(
ResponsesAPIRequestUtils.restore_encrypted_content_item_ids_in_input(copy.deepcopy(value)),
drops_set_created_by=policy.drop_params,
)
case "previous_response_id", str():
return ResponsesAPIRequestUtils.decode_previous_response_id_to_original_previous_response_id(value)
case _:
@ -398,5 +401,7 @@ def _restored(field: str, value: object) -> object:
def native_count_request(request: Mapping[str, object], model: str, policy: DropPolicy) -> Mapping[str, object]:
forwarded: Final = {field: _restored(field, value) for field, value in request.items() if _forwards(field, policy)}
forwarded: Final = {
field: _restored(field, value, policy) for field, value in request.items() if _forwards(field, policy)
}
return {**forwarded, "model": model}

192
litellm/llms/openai/responses/guardrail_translation/handler.py
View file

@ -32,11 +32,11 @@ import copy
import json
import time
import uuid
from collections.abc import Mapping, MutableSequence, Sequence
from collections.abc import Iterable, Mapping, MutableSequence, Sequence
from dataclasses import dataclass
from itertools import accumulate, chain, repeat
from itertools import accumulate, chain, groupby, repeat
from types import MappingProxyType
from typing import TYPE_CHECKING, Any, Final, NamedTuple, Protocol, Union, cast
from typing import TYPE_CHECKING, Any, Final, NamedTuple, Protocol, TypeVar, Union, cast
from pydantic import BaseModel, ConfigDict, TypeAdapter, ValidationError
from typing_extensions import ReadOnly, TypedDict
@ -55,7 +55,9 @@ from litellm.llms.base_llm.guardrail_translation.base_translation import (
)
from litellm.llms.base_llm.guardrail_translation.utils import (
blocked_responses_stream_usage,
effective_scan_only_tool_results_for_guardrail,
effective_skip_system_message_for_guardrail,
effective_skip_tool_message_for_guardrail,
list_field,
merge_guardrailed_scoped_messages,
role_out_of_guardrail_scope,
@ -65,7 +67,10 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
stream_item_items,
unappliable_request_rewrite,
)
from litellm.llms.openai.responses.guardrail_translation.tool_merge import merge_guardrailed_tools
from litellm.llms.openai.responses.guardrail_translation.tool_merge import (
append_returned_tools,
merge_guardrailed_tools,
)
from litellm.responses.litellm_completion_transformation.compaction_items import (
compaction_summary,
decode_compaction_item,
@ -74,7 +79,9 @@ from litellm.responses.litellm_completion_transformation.compaction_items import
from litellm.responses.litellm_completion_transformation.custom_tools import without_malformed_tool_flags
from litellm.responses.litellm_completion_transformation.tool_search import is_server_tool_search_tool
from litellm.responses.litellm_completion_transformation.transformation import (
TOOL_CALL_OUTPUT_ITEM_TYPES,
LiteLLMCompletionResponsesConfig,
tool_output_to_chat_content,
)
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.types.llms.openai import (
@ -270,6 +277,7 @@ _TOOL_CALL_PAYLOAD_EVENT_TYPES: Final = _TOOL_CALL_PAYLOAD_DELTA_EVENT_TYPES | f
)
_OUTPUT_ITEM_EVENT_TYPES: Final = frozenset({"response.output_item.added", "response.output_item.done"})
_OUTPUT_TEXT_EVENT_TYPES: Final = frozenset({"response.output_text.delta", "response.output_text.done"})
_TOOL_OUTPUT_TEXT_PART_TYPES: Final = frozenset({"input_text", "output_text", "text"})
_PATCHABLE_ITEM_FIELDS: Final[Mapping[str, str]] = MappingProxyType(
{"function_call_output": "output", "custom_tool_call_output": "output", "message": "content"}
)
@ -519,10 +527,109 @@ def scannable_instructions(data: Mapping[str, object], *, skip_system: bool = Fa
def _input_item_role(item: object) -> str:
if stream_item_field(item, "type") in TOOL_CALL_OUTPUT_ITEM_TYPES:
return "tool"
role: Final = item.get("role") if isinstance(item, Mapping) else None
return role.lower() if isinstance(role, str) else ""
def _scanned_text_field(item: Mapping[str, object]) -> str:
return "output" if item.get("type") in TOOL_CALL_OUTPUT_ITEM_TYPES else "content"
def _tool_output_part_text(part: object) -> str | None:
if not isinstance(part, dict):
return None
fields: Final = _item_fields_adapter.validate_python(part)
text: Final = fields.get("text")
return text if fields.get("type") in _TOOL_OUTPUT_TEXT_PART_TYPES and isinstance(text, str) else None
def tool_output_texts(item: object) -> tuple[tuple[int | None, str], ...]:
if not stream_item_field(item, "call_id") or stream_item_field(item, "type") == "computer_call_output":
return ()
parts: Final = stream_item_items(item, "output")
part_texts: Final = tuple(
(index, text) for index, part in enumerate(parts) if (text := _tool_output_part_text(part))
)
if part_texts:
return part_texts
tool_content: Final = tool_output_to_chat_content(stream_item_field(item, "output"))
return ((None, tool_content),) if isinstance(tool_content, str) and tool_content else ()
_Grouped: Final = TypeVar("_Grouped")
def _pair_call_id(pair: tuple[str, object]) -> str:
return pair[0]
def _grouped_by_call_id(pairs: Iterable[tuple[str, _Grouped]]) -> Mapping[str, tuple[_Grouped, ...]]:
ordered: Final = sorted(pairs, key=_pair_call_id)
return MappingProxyType(
{call_id: tuple(value for _, value in group) for call_id, group in groupby(ordered, key=_pair_call_id)}
)
def _texts_by_tool_row(
messages: Sequence[Mapping[str, object]], input_data: "str | ResponseInputParam"
) -> Mapping[int, tuple[str, ...]]:
if isinstance(input_data, str):
return MappingProxyType({})
output_texts_by_call_id: Final = _grouped_by_call_id(
(str(stream_item_field(item, "call_id")), tuple(text for _, text in tool_output_texts(item)))
for item in input_data
if stream_item_field(item, "type") in TOOL_CALL_OUTPUT_ITEM_TYPES
)
row_indices_by_call_id: Final = _grouped_by_call_id(
(str(message.get("tool_call_id")), index)
for index, message in enumerate(messages)
if message.get("role") == "tool"
)
return MappingProxyType(
dict(
chain.from_iterable(
zip(indices, output_texts_by_call_id[call_id], strict=True)
for call_id, indices in row_indices_by_call_id.items()
if len(indices) == len(output_texts_by_call_id.get(call_id, ()))
)
)
)
def _is_string_tool_row(message: Mapping[str, object]) -> bool:
return message.get("role") == "tool" and isinstance(message.get("content"), str)
def _split_into_text_parts(message: AllMessageValues, texts: tuple[str, ...]) -> AllMessageValues:
if not _is_string_tool_row(message) or len(texts) < 2:
return message
return cast( # cast-ok: a tool row whose content is its text parts as chat text blocks
"AllMessageValues", {**message, "content": [{"type": "text", "text": text} for text in texts]}
)
def _with_split_rows_restored(
rows: Sequence[AllMessageValues], split_rows: Sequence[tuple[AllMessageValues, AllMessageValues]]
) -> Sequence[AllMessageValues]:
if not isinstance(rows, list):
return rows
return [next((original for split, original in split_rows if row == split), row) for row in rows]
def _is_textless_tool_row(message: Mapping[str, object], texts: tuple[str, ...] | None) -> bool:
return _is_string_tool_row(message) and (not message.get("content") or texts == ())
def _rewritten_whole_text(item: object, field: str, guardrail_response: str) -> object | None:
if isinstance(stream_item_field(item, field), str):
return guardrail_response
if tool_output_texts(item) == ((None, guardrail_response),):
return None
return guardrail_response
def _patched_request_fields(
raw_input: object,
instructions: object,
@ -636,17 +743,28 @@ class OpenAIResponsesHandler(BaseTranslation):
Handles both string input and list of message objects.
"""
input_data: Final[str | ResponseInputParam | None] = data.get("input")
if not isinstance(input_data, (str, list)):
raw_input: Final[str | ResponseInputParam | None] = data.get("input")
if not isinstance(raw_input, (str, list)):
return data
input_data: Final = cast("str | ResponseInputParam", raw_input) # cast-ok: fields read via stream_item_field
skip_system: Final = effective_skip_system_message_for_guardrail(guardrail_to_apply)
skip_tool: Final = effective_skip_tool_message_for_guardrail(guardrail_to_apply)
scan_only: Final = effective_scan_only_tool_results_for_guardrail(guardrail_to_apply)
structured_messages: Final = self.get_structured_messages(data)
scoped_indices: Final = scoped_structured_message_indices(
structured_messages or [], scan_only_tool_results=False, skip_system=skip_system, skip_tool=False
all_messages: Final = structured_messages or []
texts_by_tool_row: Final = _texts_by_tool_row(all_messages, input_data)
scoped_indices: Final = tuple(
index
for index in scoped_structured_message_indices(
all_messages, scan_only_tool_results=scan_only, skip_system=skip_system, skip_tool=skip_tool
)
if not _is_textless_tool_row(all_messages[index], texts_by_tool_row.get(index))
)
scoped_structured_messages: Final = (
[structured_messages[index] for index in scoped_indices] if structured_messages else None
split_rows: Final = tuple(
(_split_into_text_parts(all_messages[index], texts_by_tool_row.get(index, ())), all_messages[index])
for index in scoped_indices
)
scoped_structured_messages: Final = [split for split, _ in split_rows] if structured_messages else None
raw_tools: Final = data.get("tools")
original_tools: Final[tuple[Mapping[str, object], ...]] = (
tuple(raw_tools) if isinstance(raw_tools, list) else ()
@ -661,7 +779,12 @@ class OpenAIResponsesHandler(BaseTranslation):
)
)
extracted: Final = self._extract_guardrail_inputs(
data, input_data, flattened_tool_groups, skip_system=skip_system
data,
input_data,
() if scan_only else flattened_tool_groups,
skip_system=skip_system,
skip_tool=skip_tool,
scan_only=scan_only,
)
if not extracted.inputs.get("texts"):
return data
@ -674,13 +797,14 @@ class OpenAIResponsesHandler(BaseTranslation):
logging_obj=litellm_logging_obj,
)
self._apply_guardrailed_tools_to_data(
data, original_tools, flattened_tool_groups, guardrailed_inputs.get("tools")
data, original_tools, flattened_tool_groups, guardrailed_inputs.get("tools"), scan_only=scan_only
)
written_back: Final = self._written_back_request_fields(
data,
structured_messages or (),
scoped_indices,
scoped_structured_messages,
tuple((split, original) for split, original in split_rows if split is not original),
guardrail_to_apply,
guardrailed_inputs,
)
@ -730,8 +854,10 @@ class OpenAIResponsesHandler(BaseTranslation):
flattened_tool_groups: Sequence[Sequence[Mapping[str, object]]],
*,
skip_system: bool = False,
skip_tool: bool = False,
scan_only: bool = False,
) -> _ExtractedInputs:
instructions: Final = scannable_instructions(data, skip_system=skip_system)
instructions: Final = scannable_instructions(data, skip_system=skip_system or scan_only)
texts_to_check: Final[list[str]] = [] if instructions is None else [instructions]
images_to_check: Final[list[str]] = []
task_mappings: Final[list[tuple[int, int | None]]] = []
@ -745,11 +871,15 @@ class OpenAIResponsesHandler(BaseTranslation):
)
)
if isinstance(input_data, str):
texts_to_check.append(input_data)
if not scan_only:
texts_to_check.append(input_data)
else:
for msg_idx, message in enumerate(input_data):
if role_out_of_guardrail_scope(
_input_item_role(message), skip_system_message=skip_system, skip_tool_message=False
_input_item_role(message),
skip_system_message=skip_system,
skip_tool_message=skip_tool,
scan_only_tool_results=scan_only,
):
continue
self._extract_input_text_and_images(
@ -775,12 +905,14 @@ class OpenAIResponsesHandler(BaseTranslation):
structured_messages: Sequence[AllMessageValues],
scoped_indices: Sequence[int],
scoped_structured_messages: Sequence[AllMessageValues] | None,
split_rows: Sequence[tuple[AllMessageValues, AllMessageValues]],
guardrail_to_apply: "CustomGuardrail",
guardrailed_inputs: GenericGuardrailAPIInputs,
) -> _RequestFields | None:
guardrailed: Final = guardrailed_inputs.get("structured_messages")
if guardrailed is None or guardrailed is scoped_structured_messages:
returned: Final = guardrailed_inputs.get("structured_messages")
if returned is None or returned is scoped_structured_messages:
return None
guardrailed: Final = _with_split_rows_restored(returned, split_rows)
covers_full_request: Final = len(scoped_indices) == len(structured_messages) or (
guardrail_to_apply.structured_messages_cover_full_request() and len(guardrailed) == len(structured_messages)
)
@ -814,11 +946,14 @@ class OpenAIResponsesHandler(BaseTranslation):
original_tools: Sequence[Mapping[str, object]],
flattened_tool_groups: Sequence[Sequence[Mapping[str, object]]],
guardrailed_tools: Sequence[ChatCompletionToolParam] | None,
*,
scan_only: bool,
) -> None:
if guardrailed_tools is None:
return
merge: Final = append_returned_tools if scan_only else merge_guardrailed_tools
data["tools"] = list( # rebind-ok: in-place request rewrite
merge_guardrailed_tools(original_tools, flattened_tool_groups, guardrailed_tools)
merge(original_tools, flattened_tool_groups, guardrailed_tools)
)
def _extract_input_text_and_images(
@ -839,6 +974,11 @@ class OpenAIResponsesHandler(BaseTranslation):
texts_to_check.append(summary)
task_mappings.append((msg_idx, None))
return
if message.get("type") in TOOL_CALL_OUTPUT_ITEM_TYPES:
for content_idx, text in tool_output_texts(message):
texts_to_check.append(text)
task_mappings.append((msg_idx, content_idx))
return
content: Final = message.get("content", None)
if content is None:
return
@ -890,18 +1030,20 @@ class OpenAIResponsesHandler(BaseTranslation):
)
continue
content = messages[msg_idx].get("content", None)
field = _scanned_text_field(messages[msg_idx])
content = messages[msg_idx].get(field, None)
if content is None:
continue
if isinstance(content, str) and content_idx_optional is None:
# Replace string content with guardrail response
messages[msg_idx]["content"] = guardrail_response
if content_idx_optional is None:
rewritten = _rewritten_whole_text(messages[msg_idx], field, guardrail_response)
if rewritten is not None:
messages[msg_idx][field] = rewritten
elif isinstance(content, list) and content_idx_optional is not None:
elif isinstance(content, list):
# Replace specific text item in list content
if isinstance(messages[msg_idx]["content"][content_idx_optional], dict):
messages[msg_idx]["content"][content_idx_optional]["text"] = guardrail_response
if isinstance(messages[msg_idx][field][content_idx_optional], dict):
messages[msg_idx][field][content_idx_optional]["text"] = guardrail_response
async def process_output_response(
self,

16
litellm/llms/openai/responses/guardrail_translation/tool_merge.py
View file

@ -6,6 +6,7 @@ from typing import Final, TypeAlias
from pydantic import BaseModel, TypeAdapter, ValidationError
from litellm._logging import verbose_logger
from litellm.llms.base_llm.guardrail_translation.utils import merge_returned_tools_into_request_tools, openai_tool_name
from litellm.responses.litellm_completion_transformation.custom_tools import custom_tool_grammar_suffix
from litellm.responses.litellm_completion_transformation.transformation import (
NAMESPACE_DESCRIPTION_SEPARATOR,
@ -183,3 +184,18 @@ def merge_guardrailed_tools(
tuple(tool for key, tool in zip(guardrailed_keys, guardrailed) if key not in owned_keys)
)
return tuple(chain(merged_originals, appended))
def append_returned_tools(
original_tools: Sequence[Tool],
flattened_groups: Sequence[Sequence[Tool]],
returned_tools: Iterable[object],
) -> tuple[Tool, ...]:
request_chat_tools: Final = tuple(chain.from_iterable(flattened_groups))
additions: Final = merge_returned_tools_into_request_tools(
request_tools=request_chat_tools, returned_tools=_validated_tools(returned_tools), tool_name=openai_tool_name
)[len(request_chat_tools) :]
return (
*original_tools,
*LiteLLMCompletionResponsesConfig.transform_chat_completion_tool_params_to_responses_api_tools(additions),
)

89
litellm/llms/openai/responses/transformation.py
View file

@ -66,6 +66,29 @@ _PROVIDERS_VALIDATING_TOOL_CALL_ITEM_IDS: Final = frozenset({LlmProviders.AZURE,
_PROVIDERS_REPLAYING_ONLY_THEIR_OWN_REASONING: Final = _PROVIDERS_VALIDATING_TOOL_CALL_ITEM_IDS
_ROLES_READING_PROMPT_CACHE_BREAKPOINTS: Final = frozenset({"system", "developer", "user"})
_PART_TYPES_READING_PROMPT_CACHE_BREAKPOINTS: Final = frozenset({"input_text", "input_image", "input_file"})
_COMPACT_TRACKING_PARAMS: Final = frozenset({"metadata", "user"})
_COMPACT_REJECTED_PARAMS: Final = frozenset(
{
"background",
"context_management",
"max_output_tokens",
"max_tool_calls",
"moderation",
"partial_images",
"prompt",
"safety_identifier",
"store",
"stream_options",
"temperature",
"tool_choice",
"top_logprobs",
"top_p",
"truncation",
}
)
_PROVIDERS_WITH_STRICT_COMPACT_PARAMS: Final = frozenset(
{LlmProviders.OPENAI, LlmProviders.AZURE, LlmProviders.AZURE_AI}
)
_INPUT_ITEM: Final[TypeAdapter[Mapping[str, object]]] = TypeAdapter(Mapping[str, object])
_CONTENT_PARTS: Final[TypeAdapter[Sequence[Mapping[str, object]]]] = TypeAdapter(Sequence[Mapping[str, object]])
@ -85,7 +108,7 @@ class _DropParamsSetting(BaseModel):
drop_params: object = None
def _drops_params(litellm_params: GenericLiteLLMParams) -> bool:
def drops_params(litellm_params: GenericLiteLLMParams) -> bool:
setting: Final = _DropParamsSetting.model_validate(litellm_params)
return litellm.drop_params is True or normalize_drop_params(setting.drop_params) is True
@ -518,7 +541,9 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
litellm_params: GenericLiteLLMParams,
prompt_cache_options: object = None,
) -> tuple[str | ResponseInputParam, Sequence[ALL_RESPONSES_API_TOOL_PARAMS] | None]:
validated_input: Final = self._validate_input_param(self._drop_bridge_minted_reasoning_items(input))
validated_input: Final = self._validate_input_param(
self._drop_bridge_minted_reasoning_items(input), drops_set_created_by=drops_params(litellm_params)
)
marked_input: Final = (
self._input_with_prompt_cache_breakpoints(validated_input)
if self._reads_prompt_cache_breakpoints(
@ -559,7 +584,7 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
if tools is None or not any(is_anthropic_tool_search_chat_tool(tool) for tool in tools):
return tools
callers_own_search: Final = any(is_openai_tool_search_tool(tool) for tool in tools)
if callers_own_search or not _drops_params(litellm_params) or _runs_tool_search(model, tools):
if callers_own_search or not drops_params(litellm_params) or _runs_tool_search(model, tools):
return tools_with_anthropic_tool_search_mapped(tools)
verbose_logger.warning(
"Dropping tool search under drop_params: OpenAI model '%s' cannot run it or no tool is deferred, so every "
@ -752,7 +777,9 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
]
return tools if all(new is old for new, old in zip(sanitized, tools, strict=True)) else sanitized
def _validate_input_param(self, input: str | ResponseInputParam) -> str | ResponseInputParam:
def _validate_input_param(
self, input: str | ResponseInputParam, *, drops_set_created_by: bool = False
) -> str | ResponseInputParam:
"""
Ensure all input fields if pydantic are converted to dict
@ -764,18 +791,20 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
for item in input:
# if it's pydantic, convert to dict
if isinstance(item, BaseModel):
validated_input.append(item.model_dump(exclude_none=True))
validated_input.append(
replayable_responses_item(
item.model_dump(exclude_none=True), drops_set_created_by=drops_set_created_by
)
)
elif isinstance(item, dict):
# Handle reasoning items specifically to filter out status=None
if item.get("type") == "reasoning":
verbose_logger.debug("Handling reasoning item: %s", item)
# Type assertion since we know it's a dict at this point
dict_item = cast(dict[str, object], item)
filtered_item = self._handle_reasoning_item(dict_item)
else:
# For other dict items, just pass through
filtered_item = cast(dict[str, object], item)
validated_input.append(filtered_item)
validated_input.append(
replayable_responses_item(
self._handle_reasoning_item(cast(dict[str, object], item))
if item.get("type") == "reasoning"
else item,
drops_set_created_by=drops_set_created_by,
)
)
else:
validated_input.append(item)
return validated_input
@ -1182,6 +1211,17 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
#########################################################
########## COMPACT RESPONSE API TRANSFORMATION ##########
#########################################################
def compact_url(self, api_base: str) -> str:
parsed_url: Final = httpx.URL(api_base)
return str(parsed_url.copy_with(path=parsed_url.path.rstrip("/") + "/compact"))
def compact_dropped_params(self, litellm_params: GenericLiteLLMParams) -> frozenset[str]:
if self.custom_llm_provider not in _PROVIDERS_WITH_STRICT_COMPACT_PARAMS:
return frozenset()
if drops_params(litellm_params):
return _COMPACT_TRACKING_PARAMS | _COMPACT_REJECTED_PARAMS
return _COMPACT_TRACKING_PARAMS
def transform_compact_response_api_request(
self,
model: str,
@ -1197,10 +1237,8 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
OpenAI API expects the following request
- POST /v1/responses/compact
"""
# Preserve query params (e.g., api-version) while appending /compact.
parsed_url: Final = httpx.URL(api_base)
compact_path: Final = parsed_url.path.rstrip("/") + "/compact"
url: Final = str(parsed_url.copy_with(path=compact_path))
url: Final = self.compact_url(api_base)
dropped_params: Final = self.compact_dropped_params(litellm_params)
request_params: Final = cast( # cast-ok: the base signature leaves the params bag untyped
"Mapping[str, object]", response_api_optional_request_params
@ -1214,14 +1252,11 @@ class OpenAIResponsesAPIConfig(BaseResponsesAPIConfig):
)
if sanitized_tools is not None:
response_api_optional_request_params["tools"] = sanitized_tools
data: Final = cast( # cast-ok: typed dict spread is inferred as Unknown by basedpyright
dict[str, object],
{
"model": model,
"input": replay_safe_input,
**response_api_optional_request_params,
},
)
data: Final = {
"model": model,
"input": replay_safe_input,
**{key: value for key, value in request_params.items() if key not in dropped_params},
}
return url, data

11
litellm/llms/openai_like/dynamic_config.py
View file

@ -55,6 +55,15 @@ def create_config_class(provider: SimpleProviderConfig):
else:
return super()._transform_messages(messages=messages, model=model, is_async=False)
def _should_preserve_cache_control_for_endpoint(
self,
custom_llm_provider: str | None,
api_base: str | None,
) -> bool:
return provider.special_handling.get(
"preserve_cache_control", False
) or super()._should_preserve_cache_control_for_endpoint(custom_llm_provider, api_base)
def _get_openai_compatible_provider_info(
self, api_base: str | None, api_key: str | None
) -> tuple[str | None, str | None]:
@ -177,7 +186,7 @@ def create_config_class(provider: SimpleProviderConfig):
return JSONProviderConfig
_responses_config_cache: Final[dict] = {}
_responses_config_cache: Final[dict[str, "type[OpenAILikeResponsesConfig]"]] = {}
def create_responses_config_class(provider: SimpleProviderConfig) -> "type[OpenAILikeResponsesConfig]":

10
litellm/llms/openai_like/json_loader.py
View file

@ -3,6 +3,7 @@ JSON-based provider configuration loader for OpenAI-compatible providers.
"""
import json
from collections.abc import Mapping
from pathlib import Path
from typing import Final
@ -12,6 +13,7 @@ from litellm._logging import verbose_logger
from litellm.types.utils import ProviderSpecificModelInfo
_PROVIDER_INFO_ADAPTER: Final = TypeAdapter(ProviderSpecificModelInfo)
_SPECIAL_HANDLING_ADAPTER: Final = TypeAdapter(Mapping[str, bool])
class SimpleProviderConfig:
@ -20,12 +22,14 @@ class SimpleProviderConfig:
def __init__(self, slug: str, data: dict):
self.slug = slug
self.base_url = data["base_url"]
self.api_key_env = data["api_key_env"]
self.api_base_env = data.get("api_base_env")
self.api_key_env: str = data["api_key_env"]
self.api_base_env: str | None = data.get("api_base_env")
self.base_class = data.get("base_class", "openai_gpt")
self.param_mappings = data.get("param_mappings", {})
self.constraints = data.get("constraints", {})
self.special_handling = data.get("special_handling", {})
self.special_handling: Mapping[str, bool] = _SPECIAL_HANDLING_ADAPTER.validate_python(
data.get("special_handling", {})
)
self.supported_endpoints = data.get("supported_endpoints", [])
self.provider_info: ProviderSpecificModelInfo = (
_PROVIDER_INFO_ADAPTER.validate_python(data["provider_info"]) if "provider_info" in data else {}

16
litellm/llms/openai_like/providers.json
View file

@ -269,5 +269,21 @@
"/v1/chat/completions",
"/v1/responses"
]
},
"streamlake": {
"base_url": "https://vanchin.streamlake.ai/api/gateway/v1/endpoints",
"api_key_env": "STREAMLAKE_API_KEY",
"api_base_env": "STREAMLAKE_API_BASE",
"param_mappings": {
"max_tokens": "max_completion_tokens"
},
"provider_info": {
"supports_function_calling": true,
"supports_reasoning": true
},
"special_handling": {
"preserve_cache_control": true
},
"supported_endpoints": ["/v1/chat/completions"]
}
}

4
litellm/llms/perplexity/responses/transformation.py
View file

@ -17,7 +17,7 @@ from pydantic import TypeAdapter
from litellm.litellm_core_utils.litellm_logging import Logging as LiteLLMLoggingObj
from litellm.litellm_core_utils.url_utils import join_api_base
from litellm.llms.base_llm.chat.transformation import BaseLLMException
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig
from litellm.llms.openai.responses.transformation import OpenAIResponsesAPIConfig, drops_params
from litellm.secret_managers.main import get_secret_str
from litellm.types.llms.openai import ResponseInputParam, ResponsesAPIResponse
from litellm.types.router import GenericLiteLLMParams
@ -84,7 +84,7 @@ class PerplexityResponsesConfig(OpenAIResponsesAPIConfig):
"""Handle preset/ model prefix: send as {"preset": name} instead of {"model": name}."""
input = self._ensure_message_type(input)
if model.startswith("preset/"):
input = self._validate_input_param(input)
input = self._validate_input_param(input, drops_set_created_by=drops_params(litellm_params))
data: Final[dict] = {
"preset": model[len("preset/") :],
"input": input,

481
litellm/model_prices_and_context_window_backup.json
View file

@ -6101,6 +6101,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6133,6 +6134,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6164,6 +6166,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6230,6 +6233,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6293,6 +6297,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6369,6 +6374,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -41933,6 +41939,120 @@
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-southeast-2/mistral.voxtral-mini-3b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 4.12e-08,
"input_cost_per_token_batches": 2.06e-08,
"input_cost_per_token_flex": 2.06e-08,
"input_cost_per_token_priority": 7.21e-08,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4.12e-08,
"output_cost_per_token_batches": 2.06e-08,
"output_cost_per_token_flex": 2.06e-08,
"output_cost_per_token_priority": 7.21e-08,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-southeast-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-southeast-2/mistral.voxtral-small-24b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 1.03e-07,
"input_cost_per_token_batches": 5.15e-08,
"input_cost_per_token_flex": 5.15e-08,
"input_cost_per_token_priority": 1.8025e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 3.09e-07,
"output_cost_per_token_batches": 1.545e-07,
"output_cost_per_token_flex": 1.545e-07,
"output_cost_per_token_priority": 5.4075e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-southeast-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-northeast-1/mistral.voxtral-mini-3b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 5e-08,
"input_cost_per_token_batches": 3e-08,
"input_cost_per_token_flex": 3e-08,
"input_cost_per_token_priority": 9e-08,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 5e-08,
"output_cost_per_token_batches": 3e-08,
"output_cost_per_token_flex": 3e-08,
"output_cost_per_token_priority": 9e-08,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-northeast-1/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-northeast-1/mistral.voxtral-small-24b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 1.2e-07,
"input_cost_per_token_batches": 6e-08,
"input_cost_per_token_flex": 6e-08,
"input_cost_per_token_priority": 2.1e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 3.6e-07,
"output_cost_per_token_batches": 1.8e-07,
"output_cost_per_token_flex": 1.8e-07,
"output_cost_per_token_priority": 6.3e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-northeast-1/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/eu-west-2/mistral.voxtral-mini-3b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 6e-08,
"input_cost_per_token_batches": 3e-08,
"input_cost_per_token_flex": 3e-08,
"input_cost_per_token_priority": 1.1e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 6e-08,
"output_cost_per_token_batches": 3e-08,
"output_cost_per_token_flex": 3e-08,
"output_cost_per_token_priority": 1.1e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/eu-west-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/eu-west-2/mistral.voxtral-small-24b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 1.6e-07,
"input_cost_per_token_batches": 8e-08,
"input_cost_per_token_flex": 8e-08,
"input_cost_per_token_priority": 2.8e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4.7e-07,
"output_cost_per_token_batches": 2.3e-07,
"output_cost_per_token_flex": 2.3e-07,
"output_cost_per_token_priority": 8.2e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/eu-west-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"mistral/codestral-2508": {
"cache_read_input_token_cost": 3e-08,
"input_cost_per_token": 3e-07,
@ -61269,6 +61389,359 @@
"supports_tool_choice": true,
"supports_vision": true
},
"streamlake/deepseek-v4-flash-0731": {
"cache_read_input_token_cost": 1.4e-08,
"input_cost_per_token": 4.4e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 1.32e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/DeepSeek-V4-Pro-0813": {
"cache_read_input_token_cost": 4.4e-08,
"input_cost_per_token": 1.32e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048566,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 3.96e-06,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/DeepSeek-V3": {
"cache_read_input_token_cost": 1.14e-07,
"input_cost_per_token": 2.86e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 56000,
"max_output_tokens": 8000,
"mode": "chat",
"output_cost_per_token": 1.143e-06,
"supports_function_calling": true,
"supports_reasoning": false,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/GLM-5.3": {
"cache_read_input_token_cost": 2.6e-07,
"input_cost_per_token": 1.4e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 4.4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_assistant_prefill": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/GLM-5.3-Flash": {
"cache_read_input_token_cost": 3e-08,
"input_cost_per_token": 1.5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_vision": true,
"supports_assistant_prefill": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/Kimi-K2-Instruct-0905": {
"cache_read_input_token_cost": 1.43e-07,
"input_cost_per_token": 5.71e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 224000,
"max_output_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 2.286e-06,
"supports_function_calling": true,
"supports_reasoning": false,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-r1": {
"cache_read_input_token_cost": 2.29e-07,
"input_cost_per_token": 5.71e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 98304,
"max_output_tokens": 16000,
"mode": "chat",
"output_cost_per_token": 2.286e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v3.1": {
"cache_read_input_token_cost": 2.29e-07,
"input_cost_per_token": 5.71e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 98304,
"max_output_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.714e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4-flash": {
"cache_read_input_token_cost": 2.8e-08,
"input_cost_per_token": 1.4e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 2.8e-07,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4-pro": {
"cache_read_input_token_cost": 1.45e-07,
"input_cost_per_token": 1.74e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 3.48e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4.1-flash-1017": {
"cache_read_input_token_cost": 6e-09,
"input_cost_per_token": 3e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4.1-flash-260910": {
"cache_read_input_token_cost": 6e-09,
"input_cost_per_token": 3e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048568,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/glm-5": {
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 202752,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/glm-5.1": {
"cache_read_input_token_cost": 2.6e-07,
"input_cost_per_token": 1.4e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 202752,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 4.4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/glm-5.2": {
"cache_read_input_token_cost": 2.6e-07,
"input_cost_per_token": 1.4e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048566,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 4.4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/kimi-k2.6": {
"cache_read_input_token_cost": 1.6e-07,
"input_cost_per_token": 9.5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 262144,
"max_output_tokens": 256000,
"mode": "chat",
"output_cost_per_token": 4e-06,
"supports_function_calling": true,
"supports_vision": true,
"supports_reasoning": false,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/kimi-k2.7-code": {
"cache_read_input_token_cost": 1.9e-07,
"input_cost_per_token": 9.5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 262144,
"max_output_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/mimo-v2.6-flash": {
"cache_read_input_token_cost": 2.8e-09,
"input_cost_per_token": 1.4e-07,
"litellm_provider": "streamlake",
"mode": "chat",
"output_cost_per_token": 2.8e-07,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/mimo-v2.6-pro": {
"cache_read_input_token_cost": 3.6e-09,
"input_cost_per_token": 4.35e-07,
"litellm_provider": "streamlake",
"mode": "chat",
"output_cost_per_token": 8.7e-07,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/minimax-m2.5": {
"cache_creation_input_token_cost": 3.75e-07,
"cache_read_input_token_cost": 3e-08,
"input_cost_per_token": 3e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 196608,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/minimax-m3": {
"cache_read_input_token_cost": 1.2e-07,
"cache_read_input_token_cost_above_512k_tokens": 2.4e-07,
"input_cost_per_token": 6e-07,
"input_cost_per_token_above_512k_tokens": 1.2e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1000000,
"max_output_tokens": 512000,
"mode": "chat",
"output_cost_per_token": 2.4e-06,
"output_cost_per_token_above_512k_tokens": 4.8e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.5-397b-a17b": {
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 6e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 258048,
"max_output_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 3.6e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.6-27b": {
"input_cost_per_token": 6e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 229376,
"max_output_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 3.6e-06,
"supports_function_calling": true,
"supports_vision": true,
"supports_reasoning": false,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.6-35b-a3b": {
"input_cost_per_token": 3.75e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 229376,
"max_output_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.25e-06,
"supports_function_calling": true,
"supports_vision": true,
"supports_reasoning": false,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.8-2.4t-a95b": {
"cache_read_input_token_cost": 2.5e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 983616,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"cache_creation_input_token_cost": 2.5e-06,
"explicit_cache_read_input_token_cost": 1.7e-07,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.8-27b": {
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 983616,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"cache_creation_input_token_cost": 6.25e-07,
"explicit_cache_read_input_token_cost": 5e-08,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"deepseek-v4-flash": {
"cache_creation_input_token_cost": 0.0,
"cache_read_input_token_cost": 6e-09,
@ -62584,6 +63057,14 @@
"litellm_provider": "anthropic"
}
},
{
"name": "typesafe-jev-ids",
"pattern": "^jev-(?:latest|preview|1\\.13\\.0|1\\.14\\.0)$",
"description": "A bare Jev id that has a typesafe/ price entry (jev-latest, jev-preview, jev-1.13.0, jev-1.14.0), the name the /typesafe pass-through forwards to TypeSafe. Listed by name, matched case-insensitively like every rule here, so foo/jev-latest or a model group named like a Jev id TypeSafe does not serve, such as jev-1.0 or jev-internal-router, never resolves to typesafe. A new Jev version goes here together with its price entry. Provider inference feeds the proxy's provider/* wildcard access checks, so this rule is what lets typesafe/* on a key, team or project allowlist cover a jev-latest request.",
"model_info": {
"litellm_provider": "typesafe"
}
},
{
"name": "claude-family-baseline",
"pattern": "claude-[a-z]+-\\d+(?:[-.]\\d+)?(?:-\\d{8})?",

26
litellm/provider_endpoints_support_backup.json
View file

@ -3785,6 +3785,32 @@
"generateContent": true
}
},
"streamlake": {
"display_name": "StreamLake (`streamlake`)",
"url": "https://docs.litellm.ai/docs/providers/openai_compatible",
"endpoints": {
"chat_completions": true,
"responses": true,
"text_completion": true,
"embeddings": false,
"image_generations": false,
"image_edits": false,
"audio_transcriptions": false,
"audio_speech": false,
"rerank": false,
"ocr": false,
"search": false,
"moderations": false,
"realtime": false,
"video_generations": false,
"batches": false,
"messages": true,
"interactions": true,
"systemone": false,
"a2a": true,
"generateContent": true
}
},
"tencent": {
"display_name": "Tencent TokenHub (`tencent`)",
"url": "https://docs.litellm.ai/docs/providers/tencent",

18
litellm/proxy/_types.py
View file

@ -3816,6 +3816,16 @@ class ManagementEndpointLoggingPayload(LiteLLMPydanticObjectBase):
end_time: datetime | None = None
def proxy_exception_status_code(message: str, code: str) -> str:
# Litellm router.py returns "No healthy deployment available" when there are no deployments available
# Should map to 429 errors https://github.com/BerriAI/litellm/issues/2487
if "No healthy deployment available" in message or "No deployments available" in message:
return "429"
if RouterErrors.no_deployments_with_tag_routing.value in message:
return "401"
return code
class ProxyException(Exception):
# NOTE: DO NOT MODIFY THIS
# This is used to map exactly to OPENAI Exceptions
@ -3844,13 +3854,7 @@ class ProxyException(Exception):
headers[k] = str(v)
self.headers = headers or {}
self.provider_specific_fields = provider_specific_fields
# rules for proxyExceptions
# Litellm router.py returns "No healthy deployment available" when there are no deployments available
# Should map to 429 errors https://github.com/BerriAI/litellm/issues/2487
if "No healthy deployment available" in self.message or "No deployments available" in self.message:
self.code = "429"
elif RouterErrors.no_deployments_with_tag_routing.value in self.message:
self.code = "401"
self.code = proxy_exception_status_code(self.message, self.code)
def to_dict(self) -> dict:
"""Converts the ProxyException instance to a dictionary."""

122
litellm/proxy/auth/auth_checks.py
View file

@ -46,6 +46,11 @@ from litellm.constants import (
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.get_llm_provider_logic import get_llm_provider
from litellm.litellm_core_utils.safe_json_loads import safe_json_loads
from litellm.litellm_core_utils.tool_fields import DropPolicy
from litellm.llms.base_llm.managed_resources.utils import (
extract_provider_resource_id_from_unified_id,
is_base64_encoded_unified_id,
)
from litellm.models.project import LiteLLM_ProjectTable
from litellm.proxy._types import (
MODEL_LIST_ROUTES,
@ -165,6 +170,7 @@ from litellm.types.proxy.auth.auth_checks import UserNotFoundError
from litellm.types.proxy.model_access_group_budget import ModelAccessGroupBudget
from litellm.types.router import DeploymentTypedDict
from litellm.utils import get_utc_datetime
from litellm.vector_stores.vector_store_registry import listed_vector_store_ids, vector_store_ids_in_tool
from .auth_checks_organization import (
add_team_org_context_to_request_body,
@ -1063,6 +1069,15 @@ def is_mcp_discovery_request(route: str, request_body: Mapping[str, object]) ->
return isinstance(method, str) and method in MCP_ZERO_SPEND_JSONRPC_METHODS
class ModelMaxBudgetHolder(Protocol):
@property
def model_max_budget(self) -> Mapping[str, object] | None: ...
def model_max_budget_of(holder: ModelMaxBudgetHolder) -> Mapping[str, object] | None:
return holder.model_max_budget
def route_skips_budget_checks(route: str) -> bool:
return route not in BUDGET_ENFORCED_SIDE_EFFECT_ROUTES and (
route in MODEL_DISCOVERY_ROUTES or not RouteChecks.is_llm_api_route(route=route)
@ -1505,6 +1520,7 @@ async def common_checks(
valid_token=valid_token,
user_object=user_object,
deny_by_default=_vector_store_deny_by_default(_typed_request_body(general_settings)),
llm_router=llm_router,
)
# 12. [OPTIONAL] Tool allowlist - key/team allowed_tools (no DB in hot path)
@ -7976,6 +7992,57 @@ def _strict_requested_vector_store_ids(request_body: Mapping[str, object]) -> tu
)
@dataclass(frozen=True, slots=True)
class _UnreadableVectorStoreIds:
pass
def _top_level_vector_store_ids(
vector_store_ids: object, drop_policies: Callable[[], tuple[DropPolicy, ...]]
) -> tuple[str, ...] | _UnreadableVectorStoreIds:
listed: Final = listed_vector_store_ids(vector_store_ids)
if vector_store_ids is None or vector_store_ids == [*listed]:
return listed
if any(policy.drops("vector_store_ids") for policy in drop_policies()):
return listed
if isinstance(vector_store_ids, str):
return (vector_store_ids,)
return _UnreadableVectorStoreIds()
def _legacy_requested_vector_store_ids(
request_body: Mapping[str, object], drop_policies: Callable[[], tuple[DropPolicy, ...]]
) -> tuple[str, ...] | _UnreadableVectorStoreIds:
"""
Reads tools the way VectorStoreRegistry does and leaves a malformed tool to the proxy's tool
validation. A top-level vector_store_ids string names one store, since a query parameter always
arrives as a string. Any other value that is not a list of strings names no readable stores unless a
drop policy the request can resolve to drops it, and then only the string entries of a list name
stores, as in registry retrieval.
"""
top_level: Final = _top_level_vector_store_ids(request_body.get("vector_store_ids"), drop_policies)
if isinstance(top_level, _UnreadableVectorStoreIds):
return top_level
return tuple(
chain(
top_level,
chain.from_iterable(vector_store_ids_in_tool(tool) for tool in _tools(request_body.get("tools"))),
)
)
def _restricts_vector_stores(grants: Sequence[tuple[GrantLayer, LiteLLM_ObjectPermissionTable | None]]) -> bool:
return any(grant is not None and bool(grant.vector_stores) for _, grant in grants)
def _request_drop_policies(
request_body: Mapping[str, object], llm_router: Router | None, valid_token: UserAPIKeyAuth | None
) -> tuple[DropPolicy, ...]:
from litellm.proxy.common_request_processing import counted_drop_policies
return counted_drop_policies(request_body, llm_router, valid_token.team_id if valid_token is not None else None)
def _require_vector_store_grant(
object_type: GrantLayer,
vector_store_ids_to_run: Sequence[str],
@ -8023,6 +8090,7 @@ async def vector_store_access_check(
*,
user_object: LiteLLM_UserTable | None = None,
deny_by_default: bool = False,
llm_router: Router | None = None,
):
"""
Checks whether the caller may use every vector store the request names.
@ -8031,11 +8099,19 @@ async def vector_store_access_check(
`retrieval_config.vector_store_id`. Grants come from each identity's
`object_permission.vector_stores`.
With `deny_by_default=False` (legacy), a key or team only restricts access when its list is
nonempty, and stores are only read from the request when a vector store registry is loaded.
Both modes read the same request fields, with or without a vector store registry, and answer 400
when the top-level `vector_store_ids` is not a list of strings, which legacy mode does only when the
key or team has an allowlist and the value is not a string, which it checks as one store id. A
LiteLLM-managed store id is compared by the provider store id it decodes to, on both sides. Legacy mode skips that value instead when the request, `litellm_settings`, the router defaults or a deployment of the requested
model drops it (`drop_params`, or `additional_drop_params` naming it), the same settings the proxy's
tool validation reads, and reads tools the way the registry does, where strict mode answers 400 on a
malformed one.
With `deny_by_default=True` (`general_settings.vector_store_deny_by_default`), stores are read
even without a registry, and a missing record, `null` or `[]` grants nothing:
With `deny_by_default=False` (legacy), a key or team only restricts access when its list is
nonempty.
With `deny_by_default=True` (`general_settings.vector_store_deny_by_default`), a missing record,
`null` or `[]` grants nothing:
- virtual key: the key must grant every store, and so must its team when it has one, even if
the team failed to load
@ -8047,8 +8123,8 @@ async def vector_store_access_check(
rescue nor restrict a key or team request.
Raises ProxyException (401, `{key,team,user}_vector_store_access_denied`) on the first identity
that does not grant a requested store, and with the flag on, ProxyException (400,
`invalid_request_error`) when a `vector_store_ids` field is not a list of strings.
that does not grant a requested store, and ProxyException (400, `invalid_request_error`) on a
malformed `vector_store_ids` the request does not drop and, in legacy mode, an allowlist applies to.
"""
from litellm.proxy.proxy_server import prisma_client, user_api_key_cache
@ -8059,23 +8135,24 @@ async def vector_store_access_check(
verbose_proxy_logger.debug("Prisma client not found, skipping vector store access check")
return True
registry_ids: Final = (
_strict_requested_vector_store_ids(_typed_request_body(request_body))
typed_request_body: Final = _typed_request_body(request_body)
requested_ids: Final = (
_strict_requested_vector_store_ids(typed_request_body)
if deny_by_default
else (
litellm.vector_store_registry.get_vector_store_ids_to_run(
non_default_params=request_body, tools=request_body.get("tools", None)
)
if litellm.vector_store_registry is not None
else None
else _legacy_requested_vector_store_ids(
typed_request_body, lambda: _request_drop_policies(typed_request_body, llm_router, valid_token)
)
or ()
)
rag_vector_store_id: Final = _get_rag_query_vector_store_id(_typed_request_body(request_body))
if isinstance(requested_ids, _UnreadableVectorStoreIds):
if _restricts_vector_stores(
await _identity_grants(valid_token, team_object, None, prisma_client, user_api_key_cache)
):
raise _malformed_vector_store_ids()
return True
rag_vector_store_id: Final = _get_rag_query_vector_store_id(typed_request_body)
rag_ids: Final = (rag_vector_store_id,) if rag_vector_store_id is not None else ()
vector_store_ids_to_run: Final = tuple(dict.fromkeys((*registry_ids, *rag_ids)))
vector_store_ids_to_run: Final = tuple(dict.fromkeys((*requested_ids, *rag_ids)))
if not vector_store_ids_to_run:
verbose_proxy_logger.debug("Vector store to run not found, skipping vector store access check")
return True
#########################################################
@ -8101,6 +8178,12 @@ async def vector_store_access_check(
return True
def _provider_vector_store_id(vector_store_id: str) -> str:
if not is_base64_encoded_unified_id(vector_store_id):
return vector_store_id
return extract_provider_resource_id_from_unified_id(vector_store_id) or vector_store_id
def _can_object_call_vector_stores(
object_type: Literal["key", "team", "org", "user"],
vector_store_ids_to_run: Sequence[str],
@ -8119,8 +8202,9 @@ def _can_object_call_vector_stores(
if len(object_permissions.vector_stores) == 0:
return True
allowed_stores: Final = frozenset(map(_provider_vector_store_id, object_permissions.vector_stores))
for vector_store_id in vector_store_ids_to_run:
if vector_store_id not in object_permissions.vector_stores:
if _provider_vector_store_id(vector_store_id) not in allowed_stores:
raise ProxyException(
message=f"User not allowed to access vector store. Tried to access {vector_store_id}. Only allowed to access {object_permissions.vector_stores}",
type=ProxyErrorTypes.get_vector_store_access_error_type_for_object(object_type),

116
litellm/proxy/auth/user_api_key_auth.py
View file

@ -40,6 +40,7 @@ from litellm.integrations.otel.model.config import is_otel_v2_enabled
from litellm.integrations.otel.runtime import phase_event, phase_span, seed_request_identity
from litellm.litellm_core_utils.dd_tracing import tracer
from litellm.litellm_core_utils.dot_notation_indexing import get_nested_value
from litellm.models.end_user import LiteLLM_EndUserTable
from litellm.proxy._types import *
from litellm.proxy.agent_endpoints.auth.agent_caller import agent_caller_from_headers
from litellm.proxy.auth.auth_checks import ( # noqa: F401 # legacy module exports
@ -85,6 +86,7 @@ from litellm.proxy.auth.auth_checks import ( # noqa: F401 # legacy module expo
jwt_key_mapping_cache_key,
key_access_group_grants_model,
key_model_aliases_for_auth_check,
model_max_budget_of,
resolve_and_validate_end_user_id,
resolve_default_end_user_budget,
route_skips_budget_checks,
@ -384,6 +386,16 @@ class _TeamModelBudgetLimiter(Protocol):
) -> bool: ...
class _EndUserModelBudgetLimiter(Protocol):
async def is_end_user_within_model_budget(
self,
end_user_id: str,
end_user_model_max_budget: Mapping[str, object],
model: str,
team_id: str | None = None,
) -> bool: ...
class _EveryModelBudgetLimiter(Protocol):
async def is_key_within_model_budget(self, user_api_key_dict: UserAPIKeyAuth, model: str) -> bool: ...
@ -530,6 +542,49 @@ async def _check_team_model_budget(
)
async def _end_user_model_max_budget(
end_user_id: str | None,
end_user_object: LiteLLM_EndUserTable | None,
key_end_user_budget_id: str | None,
prisma_client: PrismaClient | None,
user_api_key_cache: UserApiKeyCache,
parent_otel_span: Span | None,
) -> Mapping[str, object] | None:
budget_table: Final = (
end_user_object.litellm_budget_table
if end_user_object is not None
else (
await resolve_default_end_user_budget(
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
key_end_user_budget_id=key_end_user_budget_id,
parent_otel_span=parent_otel_span,
)
if end_user_id and prisma_client is not None
else None
)
)
return model_max_budget_of(budget_table) if budget_table is not None else None
async def _check_end_user_model_budget(
end_user_id: str | None,
end_user_model_max_budget: Mapping[str, object] | None,
model_max_budget_limiter: _EndUserModelBudgetLimiter,
models: Sequence[str],
team_id: str | None,
) -> None:
if not end_user_id or not end_user_model_max_budget:
return
for model_name in models:
await model_max_budget_limiter.is_end_user_within_model_budget(
end_user_id=end_user_id,
end_user_model_max_budget=end_user_model_max_budget,
model=model_name,
team_id=team_id,
)
async def _is_within_every_model_budget(
valid_token: UserAPIKeyAuth,
model_max_budget_limiter: _EveryModelBudgetLimiter,
@ -2094,6 +2149,10 @@ async def _resolve_object_permission_for_unresolvable_team(
)
def _custom_auth_token_for_this_request(response: object) -> UserAPIKeyAuth:
return UserAPIKeyAuth.model_validate(response).model_copy()
async def user_api_key_auth_builder(
request: Request,
api_key: str,
@ -2172,7 +2231,7 @@ async def user_api_key_auth_builder(
request=request, api_key=api_key, user_custom_auth=user_custom_auth
)
if response is not None and isinstance(response, UserAPIKeyAuth):
validated = UserAPIKeyAuth.model_validate(response)
validated = _custom_auth_token_for_this_request(response)
if getattr(litellm, "enable_post_custom_auth_checks", False):
validated = await _run_post_custom_auth_checks(
valid_token=validated,
@ -2188,7 +2247,7 @@ async def user_api_key_auth_builder(
custom_auth_api_key = True
elif user_custom_auth is not None:
response = await user_custom_auth(request=request, api_key=api_key)
validated = UserAPIKeyAuth.model_validate(response)
validated = _custom_auth_token_for_this_request(response)
if getattr(litellm, "enable_post_custom_auth_checks", False):
validated = await _run_post_custom_auth_checks(
valid_token=validated,
@ -3129,7 +3188,7 @@ async def validate_resolved_virtual_key( # noqa: C901 # Preserve ordering of e
)
# Recompute after a potential budget-fallback rewrite so
# the end-user check below validates the final model
# the user check below validates the final model
current_model = _get_model_from_request_context(
request_data=request_data,
route=route,
@ -3147,23 +3206,6 @@ async def validate_resolved_virtual_key( # noqa: C901 # Preserve ordering of e
models=current_models,
)
# Check 5b. End-user model max budget
end_user_mmb: Final = valid_token.end_user_model_max_budget
if (
end_user_mmb is not None
and isinstance(end_user_mmb, dict)
and len(end_user_mmb) > 0
and current_models
and valid_token.end_user_id is not None
):
for model_name in current_models:
await model_max_budget_limiter.is_end_user_within_model_budget(
end_user_id=valid_token.end_user_id,
end_user_model_max_budget=end_user_mmb,
model=model_name,
team_id=valid_token.team_id,
)
# Check 6: Additional Common Checks across jwt + key auth
if valid_token.team_id is not None:
try:
@ -3733,9 +3775,7 @@ async def run_centralized_common_checks(
project_object: Final[LiteLLM_ProjectTableCachedObj | None] = (
None if isinstance(project_result, BaseException) else project_result
)
end_user_object: Final[LiteLLM_EndUserTable | None] = (
None if isinstance(end_user_result, BaseException) else end_user_result
)
end_user_object: Final = end_user_result if isinstance(end_user_result, LiteLLM_EndUserTable) else None
global_proxy_spend: float | None = None if isinstance(global_spend_result, BaseException) else global_spend_result
carry_team_and_user_budget_state(
valid_token=user_api_key_auth_obj,
@ -3795,6 +3835,21 @@ async def run_centralized_common_checks(
parent_otel_span=parent_otel_span,
keep_token_limits=user_custom_auth is not None,
)
end_user_model_max_budget: Final = await _end_user_model_max_budget(
end_user_id=end_user_id,
end_user_object=end_user_object,
key_end_user_budget_id=key_end_user_budget_id,
prisma_client=prisma_client,
user_api_key_cache=user_api_key_cache,
parent_otel_span=parent_otel_span,
)
if end_user_id and (
not user_api_key_auth_obj.authenticated_by_custom_auth
or user_api_key_auth_obj.end_user_model_max_budget is None
):
user_api_key_auth_obj.end_user_model_max_budget = ( # rebind-ok: the spend hook charges from this token
dict(end_user_model_max_budget) if end_user_model_max_budget else None
)
skip_budget_checks: Final = _should_skip_budget_checks(
request_data=request_data,
@ -3887,6 +3942,21 @@ async def run_centralized_common_checks(
user_api_key_cache=user_api_key_cache,
proxy_logging_obj=proxy_logging_obj,
)
await _check_end_user_model_budget(
end_user_id=end_user_id,
end_user_model_max_budget=user_api_key_auth_obj.end_user_model_max_budget,
model_max_budget_limiter=model_max_budget_limiter,
team_id=user_api_key_auth_obj.team_id,
models=_get_model_names_for_budget_checks(
model=_get_model_from_request_context(
request_data=request_data,
route=route,
request=request,
llm_router=llm_router,
team_id=user_api_key_auth_obj.team_id,
)
),
)
if sub_call and RouteChecks.is_llm_api_route(route=route):
await virtual_key_max_budget_check(
valid_token=user_api_key_auth_obj,

156
litellm/proxy/client/cli/commands/agents.py
View file

@ -5,16 +5,18 @@ import shutil
import subprocess
import sys
import tempfile
from collections.abc import Callable, Mapping, Sequence
from collections import Counter
from collections.abc import Callable, Hashable, Iterable, Mapping, Sequence
from dataclasses import dataclass
from itertools import chain
from pathlib import Path
from types import MappingProxyType
from typing import Final, TypeAlias, TypeVar
from urllib.parse import urlsplit
import click
import requests
from pydantic import JsonValue, TypeAdapter, ValidationError
from pydantic import ConfigDict, JsonValue, TypeAdapter, ValidationError
from litellm.proxy.common_utils.codex_model_catalog import (
CODEX_BASE_INSTRUCTIONS_PATH,
@ -497,6 +499,113 @@ def _fetch_model_listing(
return listing.data
class _DeploymentParams(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
model: str
custom_llm_provider: str | None = None
api_base: str | None = None
litellm_credential_name: str | None = None
reasoning_effort: JsonValue = None
class _DeploymentInfo(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
team_public_model_name: str | None = None
blocked: bool | None = None
class _Deployment(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
model_name: str
litellm_params: _DeploymentParams
model_info: _DeploymentInfo | None = None
class _DeploymentListing(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True)
data: tuple[_Deployment, ...]
@dataclass(frozen=True, slots=True)
class ModelUpstream:
"""The OpenAI model every deployment behind a public name routes to, and the reasoning effort they all pin."""
model: str
reasoning_effort: str | None = None
_NO_UPSTREAMS: Final[Mapping[str, ModelUpstream]] = MappingProxyType({})
_Pinned: Final = TypeVar("_Pinned", bound=Hashable)
def _public_name(deployment: _Deployment) -> str:
team_name: Final = deployment.model_info.team_public_model_name if deployment.model_info is not None else None
return team_name or deployment.model_name
def _is_paused(deployment: _Deployment) -> bool:
return deployment.model_info is not None and deployment.model_info.blocked is True
def _is_openai_host(params: _DeploymentParams) -> bool:
if params.api_base is None:
return not params.litellm_credential_name
host: Final = urlsplit(params.api_base).hostname or ""
return host == "api.openai.com" or host.endswith(".api.openai.com")
def _openai_model(params: _DeploymentParams) -> str | None:
if params.custom_llm_provider not in (None, "openai") or not _is_openai_host(params):
return None
provider, slash, name = params.model.partition("/")
if not slash:
return params.model
return name if provider == "openai" else None
def _pinned_effort(params: _DeploymentParams) -> str | None:
return params.reasoning_effort if isinstance(params.reasoning_effort, str) else None
def _shared_by_every_deployment(values: Iterable[tuple[str, _Pinned]]) -> Mapping[str, _Pinned]:
distinct: Final = frozenset(values)
values_per_name: Final = Counter(name for name, _ in distinct)
return MappingProxyType({name: value for name, value in distinct if values_per_name[name] == 1})
def _upstreams_by_public_name(deployments: Sequence[_Deployment]) -> Mapping[str, ModelUpstream]:
live: Final = tuple(d for d in deployments if not _is_paused(d))
models: Final = _shared_by_every_deployment((_public_name(d), _openai_model(d.litellm_params)) for d in live)
efforts: Final = _shared_by_every_deployment((_public_name(d), _pinned_effort(d.litellm_params)) for d in live)
return MappingProxyType(
{name: ModelUpstream(model, efforts.get(name)) for name, model in models.items() if model is not None}
)
def _fetch_model_upstreams(
base_url: str,
api_key: str,
*,
get: Callable[..., requests.Response],
) -> Mapping[str, ModelUpstream]:
url: Final = base_url.rstrip("/") + "/v1/model/info"
try:
resp: Final = get(url, headers=MappingProxyType({"Authorization": f"Bearer {api_key}"}), timeout=10)
except requests.RequestException:
return _NO_UPSTREAMS
if resp.status_code != 200:
return _NO_UPSTREAMS
try:
listing: Final = _DeploymentListing.model_validate_json(resp.content)
except ValidationError:
return _NO_UPSTREAMS
return _upstreams_by_public_name(listing.data)
class _OpenCodeLimit(LiteLLMBaseModel):
context: int
output: int
@ -576,8 +685,21 @@ def opencode_model_sync_env(
return MappingProxyType({OPENCODE_CONFIG_CONTENT_ENV: opencode_provider_config(base_url, listing)})
def _codex_row(model: ListedModel, upstream: ModelUpstream | None) -> CodexCatalogRow:
return CodexCatalogRow(
id=model.id,
mode=model.mode,
max_input_tokens=model.max_input_tokens,
upstream_model=upstream.model if upstream is not None else None,
reasoning_effort=upstream.reasoning_effort if upstream is not None else None,
)
def codex_model_catalog(
models: Sequence[ListedModel], stock: Sequence[CodexStockModel], instructions: str
models: Sequence[ListedModel],
stock: Sequence[CodexStockModel],
instructions: str,
upstreams: Mapping[str, ModelUpstream],
) -> str | None:
"""The `model_catalog_json` body listing the proxy's chat models, or None if there are none.
@ -586,14 +708,23 @@ def codex_model_catalog(
installed Codex knows keeps that Codex's own entry and the proxy only
decides its place in the picker: the listing orders it, lists it even when
Codex hides it or keeps it off the API, and keeps Codex's upgrade nudge only
when the model it points at is served too. A model Codex does not know gets the fallback
when the model it points at is served too. An alias of such a model, per
`upstreams`, takes that entry too, minus the defaults Codex picks for the
model: no service tier or upgrade nudge, the reasoning effort its
deployments pin, and a window no larger than its listed max_input_tokens.
A model Codex does not know gets the fallback
entry, with the same base instructions Codex itself uses so the agent never
runs without a system prompt. The catalog is the one `/v1/models?client_version=...`
serves, built from the installed Codex's own entries instead of the proxy's vendored ones.
runs without a system prompt. Apart from aliases, the catalog is the one
`/v1/models?client_version=...` serves, built from the installed Codex's own
entries instead of the proxy's vendored ones.
"""
rows: Final = tuple(CodexCatalogRow(id=m.id, mode=m.mode, max_input_tokens=m.max_input_tokens) for m in models)
rows: Final = tuple(_codex_row(m, upstreams.get(m.id)) for m in models)
body: Final = codex_models_response_json(
rows, stock=MappingProxyType({m.slug: m for m in stock}), instructions=instructions, byte_limit=None
rows,
stock=MappingProxyType({m.slug: m for m in stock}),
instructions=instructions,
byte_limit=None,
alias_defaults="deployment",
)
return body.json if body.listed else None
@ -678,12 +809,14 @@ def codex_model_sync_args(
Codex has no env or inline equivalent of OPENCODE_CONFIG_CONTENT: the catalog
must be a file, so it is written under $CODEX_HOME (default ~/.codex) and
atomically replaced on every launch. The Codex at `binary` first lists its
own models, so the ones the proxy serves keep that Codex's entries, and then
own models, so the ones the proxy serves keep that Codex's entries, an alias
included when every unpaused deployment /v1/model/info lists for it routes
to the same OpenAI model and Codex knows that model, and then
reads the file back once before it is handed over. The key never lands in
the file. A failed fetch, read, listing, write or read-back is reported
rather than raised: Codex still launches with its built-in catalog and takes
a proxy model by name via -m, and a rejected file stays on disk to be looked
at.
at. An unavailable /v1/model/info only leaves every alias on the fallback entry.
"""
listing: Final = _fetch_model_listing(base_url, api_key, get=get)
if isinstance(listing, ModelSyncSkipped):
@ -700,7 +833,8 @@ def codex_model_sync_args(
stock: Final = _stock_codex_models(binary, base_env, run=run)
if isinstance(stock, ModelSyncSkipped):
return stock
catalog: Final = codex_model_catalog(listing, stock, instructions)
upstreams: Final = _fetch_model_upstreams(base_url, api_key, get=get)
catalog: Final = codex_model_catalog(listing, stock, instructions, upstreams)
if catalog is None:
return ModelSyncSkipped(f"{base_url.rstrip('/')}/v1/models lists no chat models")
try:

9
litellm/proxy/common_request_processing.py
View file

@ -724,7 +724,7 @@ def _router_defaults(llm_router: Router | None) -> Mapping[str, object]:
return {name: value for name, value in _REQUEST_FIELDS.validate_python(configured).items() if value is not None}
def _counted_drop_policies(
def counted_drop_policies(
data: Mapping[str, object], llm_router: Router | None, team_id: str | None
) -> tuple[DropPolicy, ...]:
requested: Final = {**_router_defaults(llm_router), **data}
@ -740,7 +740,7 @@ def _counted_drop_policies(
def resolve_counted_tool_fields(
data: Mapping[str, object], llm_router: Router | None, team_id: str | None
) -> Mapping[str, object] | ProxyException:
resolved: Final = resolve_tool_fields(data, lambda: _counted_drop_policies(data, llm_router, team_id))
resolved: Final = resolve_tool_fields(data, lambda: counted_drop_policies(data, llm_router, team_id))
if isinstance(resolved, MalformedField):
return ProxyException(
message=resolved.message,
@ -2581,7 +2581,6 @@ class ProxyBaseLLMRequestProcessing:
# have mutated `self.data` in place, and the audit-trail snapshot taken in
# add_litellm_data_to_request predates that mutation.
refresh_proxy_server_request_body_snapshot(self.data, guardrails_applied=True)
verbose_proxy_logger.debug("receiving data: %s", self.data)
if "messages" in self.data and self.data["messages"]:
logging_obj.update_messages(self.data["messages"])
@ -4234,7 +4233,7 @@ class ProxyBaseLLMRequestProcessing:
e = transformed_exception
litellm_debug_info: Final = getattr(e, "litellm_debug_info", "")
verbose_proxy_logger.debug(
"\033[1;31mAn error occurred: %s %s\n\n Debug this by setting `--debug`, e.g. `litellm --model gpt-3.5-turbo --debug`",
"An error occurred: %s %s",
e,
litellm_debug_info,
)
@ -4597,7 +4596,7 @@ class ProxyBaseLLMRequestProcessing:
if transformed_exception is not None:
e = transformed_exception
verbose_proxy_logger.debug(
"\x1b[1;31mAn error occurred: %s\n\n Debug this by setting `--debug`, e.g. `litellm --model gpt-3.5-turbo --debug`",
"An error occurred: %s",
e,
)

62
litellm/proxy/common_utils/codex_model_catalog.py
View file

@ -24,7 +24,7 @@ from functools import cache
from itertools import accumulate, pairwise
from pathlib import Path
from types import MappingProxyType, NoneType
from typing import TYPE_CHECKING, Annotated, Final, Literal
from typing import TYPE_CHECKING, Annotated, Final, Literal, TypeAlias
from pydantic import ConfigDict, StringConstraints, TypeAdapter, ValidationError
@ -110,6 +110,12 @@ class CodexStockUpgrade(LiteLLMBaseModel):
model: str
class CodexReasoningLevel(LiteLLMBaseModel):
model_config = ConfigDict(frozen=True, extra="allow")
effort: str
class CodexStockModel(LiteLLMBaseModel):
"""One `ModelInfo` entry as Codex ships it or prints it from `codex debug models`.
@ -128,6 +134,8 @@ class CodexStockModel(LiteLLMBaseModel):
upgrade: CodexStockUpgrade | None = None
service_tiers: tuple[CodexServiceTier, ...] = ()
default_service_tier: str | None = None
supported_reasoning_levels: tuple[CodexReasoningLevel, ...] = ()
context_window: int | None = None
class CodexStockCatalog(LiteLLMBaseModel):
@ -135,19 +143,21 @@ class CodexStockCatalog(LiteLLMBaseModel):
CodexCatalogEntry = CodexFallbackModel | CodexStockModel
CodexAliasDefaults: TypeAlias = Literal["codex", "deployment"]
@dataclass(frozen=True, slots=True)
class CodexCatalogRow:
"""What the catalog needs to know about one listed model: its public id, the listing's mode and
input limit, the upstream model behind it, and the `model_info` values the operator set;
`service_tiers` holds one raw `model_info.service_tiers` value per deployment behind the id, None
where a deployment sets none."""
input limit, the upstream model behind it, the reasoning effort its deployments pin, and the
`model_info` values the operator set; `service_tiers` holds one raw `model_info.service_tiers`
value per deployment behind the id, None where a deployment sets none."""
id: str
mode: str | None = None
max_input_tokens: int | None = None
upstream_model: str | None = None
reasoning_effort: str | None = None
display_name: str | None = None
service_tiers: tuple[object, ...] = ()
@ -254,12 +264,42 @@ def _stock_for(row: CodexCatalogRow, stock: Mapping[str, CodexStockModel]) -> Co
return upstream if upstream is not None else stock.get(row.upstream_model.partition("/")[2])
def _alias_context_window(row: CodexCatalogRow, stock_model: CodexStockModel) -> Mapping[str, int]:
if row.max_input_tokens is None or stock_model.context_window is None:
return MappingProxyType({})
return MappingProxyType({"context_window": min(stock_model.context_window, row.max_input_tokens)})
def _alias_reasoning_level(row: CodexCatalogRow, stock_model: CodexStockModel) -> str | None:
efforts: Final = frozenset(level.effort for level in stock_model.supported_reasoning_levels)
return row.reasoning_effort if row.reasoning_effort in efforts else None
def _deployment_defaults(row: CodexCatalogRow, stock_model: CodexStockModel) -> Mapping[str, object]:
"""What an alias's deployments decide in place of the defaults Codex picks for the model they route to:
no service tier, verbosity, upgrade nudge or startup promo of Codex's own, so what the deployment
pins (or OpenAI's API default) stays in force, the pinned reasoning effort when Codex offers it and
none otherwise, and a window no larger than the alias's own limit."""
return MappingProxyType(
{
"service_tiers": (),
"default_service_tier": None,
"default_verbosity": None,
"upgrade": None,
"availability_nux": None,
"default_reasoning_level": _alias_reasoning_level(row, stock_model),
**_alias_context_window(row, stock_model),
}
)
def _entry(
index: int,
row: CodexCatalogRow,
stock: Mapping[str, CodexStockModel],
served: frozenset[str],
instructions: str,
alias_defaults: CodexAliasDefaults,
) -> CodexCatalogEntry:
stock_model: Final = _stock_for(row, stock)
stock_tiers: Final = MappingProxyType(
@ -278,7 +318,8 @@ def _entry(
upgrade: Final = (
stock_model.upgrade if stock_model.upgrade is not None and stock_model.upgrade.model in served else None
)
stock_name: Final = stock_model.display_name if row.id == stock_model.slug else row.id
is_alias: Final = row.id != stock_model.slug
stock_name: Final = row.id if is_alias else stock_model.display_name
listing: Final = {
"slug": row.id,
"display_name": row.display_name if row.display_name is not None else stock_name,
@ -286,6 +327,7 @@ def _entry(
"visibility": "list",
"supported_in_api": True,
"upgrade": upgrade,
**(_deployment_defaults(row, stock_model) if is_alias and alias_defaults == "deployment" else {}),
}
if configured is None:
return stock_model.model_copy(update=listing)
@ -351,6 +393,7 @@ def codex_models_response_json(
stock: Mapping[str, CodexStockModel] | None = None,
instructions: str | None = None,
byte_limit: int | None = CODEX_CATALOG_BYTE_LIMIT,
alias_defaults: CodexAliasDefaults = "codex",
) -> CodexCatalogBody:
"""Codex's `ModelsResponse` body for `rows`, in listing order, cut to Codex's byte limit.
@ -359,13 +402,18 @@ def codex_models_response_json(
byte limit silently and keeps its bundled one, so the body holds only the entries that fit,
the ones offering a service tier taken first and the rest in listing order, an entry too large
for the bytes left passed over, every kept entry at its listing position; `byte_limit=None`
keeps every entry.
keeps every entry. A row listed under another name than the Codex model its `upstream_model`
names (an alias) takes that model's entry; `alias_defaults="deployment"` drops the defaults
Codex picks for that model in favor of what the alias's deployments pin, while `"codex"` keeps
them, as Codex would use them for the model's own name.
"""
known: Final = bundled_codex_models() if stock is None else stock
prompt: Final = codex_base_instructions() if instructions is None else instructions
drivable: Final = tuple(row for row in rows if _codex_can_drive(row))
served: Final = frozenset(row.id for row in drivable)
entries: Final = tuple(_entry(index, row, known, served, prompt) for index, row in enumerate(drivable))
entries: Final = tuple(
_entry(index, row, known, served, prompt, alias_defaults) for index, row in enumerate(drivable)
)
serialized: Final = tuple(_serialized(entry) for entry in entries)
kept: Final = _kept_positions(entries, serialized, byte_limit)
return CodexCatalogBody(

2
litellm/proxy/common_utils/http_parsing_utils.py
View file

@ -845,5 +845,3 @@ def _add_vector_store_id_from_path(request_data: dict, request: Request) -> None
"populate_request_with_path_params: Updated request_data with vector_store_ids=%s",
request_data.get("vector_store_ids"),
)
else:
verbose_proxy_logger.debug("populate_request_with_path_params: No vector_store_id present in path=%s", path)

96
litellm/proxy/common_utils/openai_error_payload.py
View file

@ -2,14 +2,17 @@
``type`` is a required string and ``param`` is nullable, neither of which the literal
string ``"None"`` satisfies."""
import sys
from collections.abc import Mapping
from types import MappingProxyType
from typing import Final
from fastapi import status
from fastapi import HTTPException, status
from litellm.constants import STRINGIFIED_NONE
from litellm.proxy._types import ProxyException
from litellm.proxy._types import ProxyErrorTypes, ProxyException, proxy_exception_status_code
from litellm.proxy.db.db_lookup_gate import DBLookupDeadlineExceeded
from litellm.proxy.db.exception_handler import PrismaDBExceptionHandler, exception_chain
LITELLM_CALL_ID_HEADER: Final = "x-litellm-call-id"
@ -21,6 +24,19 @@ _OPENAI_ERROR_TYPE_BY_STATUS: Final[Mapping[int, str]] = MappingProxyType(
}
)
_AUTH_FAILURE_STATUS_CODES: Final = frozenset({status.HTTP_401_UNAUTHORIZED, status.HTTP_403_FORBIDDEN})
_FAILOVER_SQLSTATES: Final = frozenset({"25006", "57P01", "57P02", "57P03"})
_FAILOVER_REFUSALS: Final = (
"is not currently accepting connections",
"the database system is starting up",
"the database system is shutting down",
"the database system is in recovery mode",
"the database system is not yet accepting connections",
"terminating connection due to administrator command",
)
def attribute_of(value: object, name: str, default: object = None) -> object:
return getattr(value, name, default)
@ -61,6 +77,82 @@ def openai_error_param(exc: object) -> str | None:
return carried if isinstance(carried, str) and carried != STRINGIFIED_NONE else None
def proxy_exception_for(exc: Exception, default_status_code: int) -> ProxyException:
match exc:
case ProxyException():
return exc
case _ if _is_database_outage(exc):
return ProxyException(
message=PrismaDBExceptionHandler.database_unavailable_message(exc),
type=ProxyErrorTypes.no_db_connection.value,
param=None,
code=status.HTTP_503_SERVICE_UNAVAILABLE,
)
case HTTPException(status_code=status_code, detail=detail):
return _openai_shaped_proxy_exception(exc, str(detail), status_code)
case _:
return _openai_shaped_proxy_exception(
exc, str(exc) or type(exc).__name__, error_status_code(exc, default_status_code)
)
def _is_database_outage(exc: Exception) -> bool:
return any(map(_is_database_outage_link, exception_chain(exc)))
def _is_database_outage_link(exc: BaseException) -> bool:
match exc:
case DBLookupDeadlineExceeded():
return True
case Exception() if _prisma_is_loaded():
return _is_prisma_outage(exc)
case _:
return False
def _prisma_is_loaded() -> bool:
return sys.modules.get("prisma") is not None
def _is_prisma_outage(exc: Exception) -> bool:
if PrismaDBExceptionHandler.is_prisma_engine_internal_error(exc):
return True
if type(exc).__module__.partition(".")[0] != "prisma":
return False
return PrismaDBExceptionHandler.is_database_service_unavailable_error(exc) or _is_failover_refusal(exc)
def _is_failover_refusal(exc: Exception) -> bool:
if PrismaDBExceptionHandler.is_read_only_transaction_error(exc):
return True
if PrismaDBExceptionHandler.postgres_sqlstate(exc) in _FAILOVER_SQLSTATES:
return True
message: Final = str(exc).lower()
return any(refusal in message for refusal in _FAILOVER_REFUSALS)
def _openai_shaped_proxy_exception(exc: Exception, message: str, status_code: int) -> ProxyException:
answered_status: Final = int(proxy_exception_status_code(message, str(status_code)))
return ProxyException(
message=message,
type=_catch_all_error_type(exc, answered_status),
param=openai_error_param(exc),
code=answered_status,
)
def _catch_all_error_type(exc: Exception, status_code: int) -> str:
if _rejects_the_caller(exc, status_code):
return ProxyErrorTypes.auth_error.value
return openai_error_type(exc, status_code)
def _rejects_the_caller(exc: Exception, status_code: int) -> bool:
raised_as_http_answer: Final = isinstance(exc, HTTPException)
status_from_upstream: Final = error_status_code(exc, 0) == status_code
return status_code in _AUTH_FAILURE_STATUS_CODES and (raised_as_http_answer or not status_from_upstream)
def litellm_call_id_headers(litellm_call_id: str | None) -> dict[str, str] | None: # mutable-ok: ProxyException.headers
if litellm_call_id is None:
return None

84
litellm/proxy/credential_endpoints/endpoints.py
View file

@ -5,6 +5,7 @@ CRUD endpoints for storing reusable credentials.
import time
import unicodedata
from collections.abc import Mapping
from types import MappingProxyType
from typing import (
Annotated,
Final,
@ -23,7 +24,7 @@ from litellm.constants import (
GITHUB_COPILOT_PER_USER_AUTH_TYPE,
)
from litellm.litellm_core_utils.credential_accessor import CredentialAccessor
from litellm.litellm_core_utils.litellm_logging import get_masked_values
from litellm.litellm_core_utils.litellm_logging import get_masked_values, has_sensitive_keyword
from litellm.llms.anthropic.wif import (
ExportedJwks,
NotAnInternalIssuerCredential,
@ -83,6 +84,11 @@ _CREDENTIAL_DICT_ADAPTER: Final = TypeAdapter(dict[str, object])
_GITHUB_COPILOT_PROVIDER: Final = "github_copilot"
_DISPLAY_NAME_MAX_LENGTH: Final = 255
_DISPLAY_NAME_FORBIDDEN_CATEGORIES: Final = frozenset({"Cc", "Cf", "Cs", "Zl", "Zp"})
_MASK_UNMASKED_LENGTH: Final = 4
_MASK_ASTERISKS: Final = 4
_SHORT_SECRET_MASK: Final = "*****"
_MASKED_READ_BACK_DEPTH: Final = 20
_NO_FIELDS: Final[Mapping[str, object]] = MappingProxyType({})
def _normalized_github_copilot_values(
@ -395,7 +401,14 @@ async def get_credentials(
"""
try:
masked_credentials: Final = [
_credential_view(credential, get_masked_values(credential.credential_values))
_credential_view(
credential,
get_masked_values(
credential.credential_values,
unmasked_length=_MASK_UNMASKED_LENGTH,
number_of_asterisks=_MASK_ASTERISKS,
),
)
for credential in litellm.credential_list
]
return {"success": True, "credentials": masked_credentials}
@ -423,7 +436,11 @@ async def get_credential_by_name(
if credential.credential_name == credential_name:
return _credential_view(
credential,
get_masked_values(credential.credential_values, unmasked_length=4, number_of_asterisks=4),
get_masked_values(
credential.credential_values,
unmasked_length=_MASK_UNMASKED_LENGTH,
number_of_asterisks=_MASK_ASTERISKS,
),
)
raise HTTPException(
status_code=404,
@ -521,8 +538,8 @@ async def get_credential_by_model(
raise HTTPException(status_code=404, detail="Model not found")
masked_credential_values: Final = get_masked_values(
credential_values,
unmasked_length=4,
number_of_asterisks=4,
unmasked_length=_MASK_UNMASKED_LENGTH,
number_of_asterisks=_MASK_ASTERISKS,
)
credential: Final = CredentialItem(
credential_name=f"{model.model_name}-credential-{model_id}",
@ -908,6 +925,48 @@ def update_db_credential(
return merged_credential
def _is_masked_read_back(value: object) -> bool:
if not isinstance(value, str):
return False
if value == _SHORT_SECRET_MASK:
return True
shown: Final = _MASK_UNMASKED_LENGTH // 2
return len(value) == _MASK_UNMASKED_LENGTH + _MASK_ASTERISKS and value[shown:-shown] == "*" * _MASK_ASTERISKS
def _as_fields(value: object) -> Mapping[str, object] | None:
try:
return _CREDENTIAL_DICT_ADAPTER.validate_python(value, strict=True)
except ValidationError:
return None
def _kept_value(key: str, value: object, stored: Mapping[str, object], depth: int) -> tuple[str, object] | None:
if not has_sensitive_keyword(key):
return key, value
fields: Final = _as_fields(value)
if fields is not None and depth < _MASKED_READ_BACK_DEPTH:
return key, _without_masked_read_backs(fields, _as_fields(stored.get(key)) or _NO_FIELDS, depth + 1)
if not _is_masked_read_back(value):
return key, value
if key in stored:
return key, stored[key]
return None
def _without_masked_read_backs(
requested: Mapping[str, object], stored: Mapping[str, object], depth: int = 0
) -> Mapping[str, object]:
kept: Final = (_kept_value(key, value, stored, depth) for key, value in requested.items())
return _CREDENTIAL_DICT_ADAPTER.validate_python(
MappingProxyType(dict(entry for entry in kept if entry is not None))
)
def _stored_objects(stored_values: Mapping[str, object]) -> Mapping[str, object]:
return MappingProxyType({key: value for key, value in stored_values.items() if _as_fields(value) is not None})
@router.patch(
"/credentials/{credential_name:path}",
dependencies=[Depends(user_api_key_auth)],
@ -936,12 +995,12 @@ async def update_credential(
)
requested_display_name: Final = _normalized_display_name(credential.display_name)
_reject_overlapping_credential_values(credential)
incoming_values: Final = _CREDENTIAL_DICT_ADAPTER.validate_python(
requested_values: Final = _CREDENTIAL_DICT_ADAPTER.validate_python(
_resolve_deployment_credentials(llm_router, credential.model_id)
if credential.model_id
else credential.credential_values or {}
)
_reject_non_admin_wif_fields(_incoming_wif_fields(incoming_values, credential), user_api_key_dict)
_reject_non_admin_wif_fields(_incoming_wif_fields(requested_values, credential), user_api_key_dict)
if prisma_client is None:
raise HTTPException(
status_code=500,
@ -952,19 +1011,20 @@ async def update_credential(
if db_credential is None:
raise _not_found_unless_config_defined(credential_name, "Credential not found in DB.")
_reject_non_admin_wif_fields(_stored_wif_fields(db_credential), user_api_key_dict)
decrypted_stored_credential: Final = decrypted_credential(db_credential)
decrypted_values: Final = _CREDENTIAL_DICT_ADAPTER.validate_python(
decrypted_stored_credential.credential_values
)
incoming_values: Final = _without_masked_read_backs(requested_values, _stored_objects(decrypted_values))
patch: Final = CredentialItem(
credential_name=credential_name,
display_name=(
requested_display_name if "display_name" in credential.model_fields_set else db_credential.display_name
),
credential_info=_CREDENTIAL_DICT_ADAPTER.validate_python(credential.credential_info),
credential_values=incoming_values,
credential_values=_CREDENTIAL_DICT_ADAPTER.validate_python(incoming_values),
credential_values_to_delete=credential.credential_values_to_delete,
)
decrypted_stored_credential: Final = decrypted_credential(db_credential)
decrypted_values: Final = _CREDENTIAL_DICT_ADAPTER.validate_python(
decrypted_stored_credential.credential_values
)
stored_info: Final = _CREDENTIAL_DICT_ADAPTER.validate_python(
db_credential.credential_info # pyright: ignore[reportUnknownMemberType] # untyped credential field
)

4
litellm/proxy/db/exception_handler.py
View file

@ -36,7 +36,7 @@ _LOCK_TIMEOUT_SQLSTATE: Final = "55P03"
_LOCK_TIMEOUT_PHRASE: Final = "canceling statement due to lock timeout"
def _exception_chain(e: BaseException) -> Iterator[BaseException]:
def exception_chain(e: BaseException) -> Iterator[BaseException]:
current = e # rebind-ok: advances one link per iteration of the bounded walk
for _ in range(_MAX_EXCEPTION_CHAIN_DEPTH):
yield current
@ -49,7 +49,7 @@ def _exception_chain(e: BaseException) -> Iterator[BaseException]:
def _database_service_unavailable_errors(e: BaseException) -> tuple[Exception, ...]:
return tuple(
link
for link in _exception_chain(e)
for link in exception_chain(e)
if isinstance(link, Exception) and PrismaDBExceptionHandler.is_database_service_unavailable_error(link)
)

3
litellm/proxy/example_config_yaml/custom_callbacks.py
View file

@ -28,7 +28,7 @@ class MyCustomHandler(CustomLogger):
def __init__(self):
blue_color_code: Final = "\033[94m"
reset_color_code: Final = "\033[0m"
print_verbose(f"{blue_color_code}Initialized LiteLLM custom logger")
print_verbose(f"{blue_color_code}Initialized LiteLLM custom logger{reset_color_code}")
try:
print_verbose("Logger Initialized with following methods:")
methods: Final = [
@ -40,7 +40,6 @@ class MyCustomHandler(CustomLogger):
# Pretty print_verbose the methods
for method in methods:
print_verbose(f" - {method}")
print_verbose(f"{reset_color_code}")
except Exception:
pass

73
litellm/proxy/google_endpoints/endpoints.py
View file

@ -1,11 +1,11 @@
from collections.abc import Mapping
from itertools import chain
from types import MappingProxyType
from typing import Final
from typing import Annotated, Final
from fastapi import APIRouter, Depends, HTTPException, Request, Response
from fastapi.responses import JSONResponse, ORJSONResponse
from pydantic import BaseModel, ConfigDict, Field, JsonValue, TypeAdapter, ValidationError
from pydantic import BaseModel, BeforeValidator, ConfigDict, Field, JsonValue, TypeAdapter, ValidationError
from litellm.litellm_core_utils.json_validation_rule import normalize_json_schema_types
from litellm.proxy._types import *
@ -27,14 +27,67 @@ from litellm.proxy.common_utils.token_count_settings import count_source_fields
from litellm.types.llms.vertex_ai import TokenCountDetailsResponse
def _google_repeated_field(value: JsonValue) -> JsonValue:
match value:
case None:
return []
case dict():
return [value]
case list():
return [item for item in value if item is not None]
case _:
return value
def _google_content(value: JsonValue) -> JsonValue:
match value:
case {"parts": parts}:
return {**value, "parts": _google_repeated_field(parts)}
case _:
return value
def _google_contents(value: JsonValue) -> JsonValue:
match _google_repeated_field(value):
case list() as contents:
return [_google_content(content) for content in contents]
case other:
return other
_GOOGLE_PART_TEXT_AND_METADATA_FIELDS: Final = frozenset(
{"text", "thought", "thoughtSignature", "thought_signature", "partMetadata", "part_metadata"}
)
def _google_part_has_non_text_content(part: Mapping[str, JsonValue]) -> bool:
return any(field is not None for key, field in part.items() if key not in _GOOGLE_PART_TEXT_AND_METADATA_FIELDS)
def _google_system_part(value: JsonValue) -> JsonValue:
match value:
case dict() if value.get("text") is None and _google_part_has_non_text_content(value):
return {key: field for key, field in value.items() if key != "text"}
case dict():
return {"text": None, **value}
case _:
return value
_GOOGLE_REPEATED_FIELD: Final = BeforeValidator(_google_repeated_field)
class _GoogleCountTokensTextPart(BaseModel):
model_config = ConfigDict(frozen=True)
text: str
text: str | None
class _GoogleCountTokensSystemInstruction(BaseModel):
model_config = ConfigDict(frozen=True)
parts: tuple[_GoogleCountTokensTextPart, ...]
parts: Annotated[
tuple[Annotated[_GoogleCountTokensTextPart, BeforeValidator(_google_system_part)], ...],
_GOOGLE_REPEATED_FIELD,
] = ()
class _GoogleCountTokensFunctionDeclaration(BaseModel):
@ -47,16 +100,16 @@ class _GoogleCountTokensFunctionDeclaration(BaseModel):
class _GoogleCountTokensTool(BaseModel):
model_config = ConfigDict(frozen=True, populate_by_name=True)
function_declarations: tuple[_GoogleCountTokensFunctionDeclaration, ...] = Field(
function_declarations: Annotated[tuple[_GoogleCountTokensFunctionDeclaration, ...], _GOOGLE_REPEATED_FIELD] = Field(
default=(), alias="functionDeclarations"
)
class _GoogleCountTokensRequest(BaseModel):
model_config = ConfigDict(frozen=True, extra="ignore", populate_by_name=True)
contents: tuple[Mapping[str, JsonValue], ...] = ()
contents: Annotated[tuple[Mapping[str, JsonValue], ...], BeforeValidator(_google_contents)] = ()
system_instruction: _GoogleCountTokensSystemInstruction | None = Field(default=None, alias="systemInstruction")
tools: tuple[_GoogleCountTokensTool, ...] = ()
tools: Annotated[tuple[_GoogleCountTokensTool, ...], _GOOGLE_REPEATED_FIELD] = ()
router: Final = APIRouter(
@ -112,7 +165,11 @@ def _google_count_tokens_request(model_name: str, data: object) -> TokenCountReq
GoogleGenAIAdapter().translate_generate_content_to_completion(model_name, contents).get("messages", [])
)
system: Final = (
[{"type": "text", "text": part.text} for part in validated_data.system_instruction.parts]
[
{"type": "text", "text": part.text}
for part in validated_data.system_instruction.parts
if part.text is not None
]
if validated_data.system_instruction is not None
else None
)

14
litellm/proxy/guardrails/_content_utils.py
View file

@ -62,6 +62,20 @@ def is_non_conversational_call_type(call_type: str) -> bool:
return call_type in NON_CONVERSATIONAL_CALL_TYPES
_PASSTHROUGH_EMBEDDINGS_ENDPOINT: Final = re.compile(r"/?(?:[\w-]+/)*embeddings/?")
def is_non_conversational_request(call_type: str, data: Mapping[str, object]) -> bool:
if is_non_conversational_call_type(call_type):
return True
endpoint: Final = data.get("endpoint")
return (
call_type == "allm_passthrough_route"
and isinstance(endpoint, str)
and _PASSTHROUGH_EMBEDDINGS_ENDPOINT.fullmatch(endpoint) is not None
)
TEXT_PART_TYPES: Final[frozenset[str]] = frozenset(
{"text", "input_text", "output_text", "summary_text", "reasoning_text"}
)

6
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py
View file

@ -49,6 +49,7 @@ from litellm.llms.base_llm.guardrail_translation.utils import (
from litellm.llms.bedrock.base_aws_llm import BaseAWSLLM, bedrock_bearer_token, run_aws_signing
from litellm.llms.bedrock.guardrail_attachments import find_request_attachments
from litellm.llms.custom_httpx.http_handler import (
AsyncHTTPHandler,
get_async_httpx_client,
httpxSpecialProvider,
)
@ -286,9 +287,12 @@ class BedrockGuardrail(CustomGuardrail, BaseAWSLLM):
streaming_sampling_rate: int | None = None,
streaming_end_of_stream_only: bool | None = None,
streaming_buffer_release_on_scan: bool | None = None,
async_handler: AsyncHTTPHandler | None = None,
**kwargs,
):
self.async_handler = get_async_httpx_client(llm_provider=httpxSpecialProvider.GuardrailCallback)
self.async_handler = async_handler or get_async_httpx_client(
llm_provider=httpxSpecialProvider.GuardrailCallback
)
self._set_streaming_params(
BedrockGuardrailStreamingParams.from_extras(
MappingProxyType(

27
litellm/proxy/guardrails/guardrail_hooks/generic_guardrail_api/example_config.yaml
View file

@ -6,20 +6,19 @@ model_list:
model: gpt-4
api_key: os.environ/OPENAI_API_KEY
litellm_settings:
guardrails:
- guardrail_name: "my-generic-guardrail"
litellm_params:
guardrail: generic_guardrail_api
mode: pre_call # Options: pre_call, post_call, during_call, [pre_call, post_call]
api_key: os.environ/GENERIC_GUARDRAIL_API_KEY # Optional if using Bearer auth
api_base: http://localhost:8080 # Required. Endpoint /beta/litellm_basic_guardrail_api is automatically appended
unreachable_fallback: fail_closed # Options: fail_closed (default, raise), fail_open (proceed if endpoint unreachable or upstream returns 502/503/504)
default_on: false # Set to true to apply to all requests by default
additional_provider_specific_params:
# Any additional parameters your guardrail API needs
api_version: "v1"
custom_param: "value"
guardrails:
- guardrail_name: "my-generic-guardrail"
litellm_params:
guardrail: generic_guardrail_api
mode: pre_call # Options: pre_call, post_call, during_call, [pre_call, post_call]
api_key: os.environ/GENERIC_GUARDRAIL_API_KEY # Optional if using Bearer auth
api_base: http://localhost:8080 # Required. Endpoint /beta/litellm_basic_guardrail_api is automatically appended
unreachable_fallback: fail_closed # Options: fail_closed (default, raise), fail_open (proceed if endpoint unreachable or upstream returns 502/503/504)
default_on: false # Set to true to apply to all requests by default
additional_provider_specific_params:
# Any additional parameters your guardrail API needs
api_version: "v1"
custom_param: "value"
# Usage examples:

29
litellm/proxy/guardrails/guardrail_hooks/panw_prisma_airs/panw_prisma_airs.py
View file

@ -30,6 +30,7 @@ from litellm.integrations.custom_guardrail import (
from litellm.llms.base_llm.guardrail_translation.utils import (
effective_scan_only_tool_results_for_guardrail,
effective_skip_system_message_for_guardrail,
effective_skip_tool_message_for_guardrail,
role_out_of_guardrail_scope,
)
from litellm.llms.custom_httpx.http_handler import (
@ -37,12 +38,16 @@ from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
from litellm.llms.openai.responses.guardrail_translation.handler import scannable_instructions
from litellm.llms.openai.responses.guardrail_translation.handler import (
scannable_instructions,
tool_output_texts,
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.common_utils.callback_utils import (
add_guardrail_scan_id,
add_guardrail_to_applied_guardrails_header,
)
from litellm.responses.litellm_completion_transformation.transformation import TOOL_CALL_OUTPUT_ITEM_TYPES
from litellm.types.guardrails import GuardrailEventHooks
from litellm.types.llms.base import LiteLLMBaseModel
from litellm.types.llms.openai import AllMessageValues
@ -112,12 +117,16 @@ class _ResponsesInputItem(LiteLLMBaseModel):
type: str | None = None
role: str | None = None
content: str | tuple[_ResponsesContentPart, ...] | None = None
output: object = None
call_id: object = None
def text_count(self, *, skip_system: bool) -> int:
if role_out_of_guardrail_scope(
(self.role or "").lower(), skip_system_message=skip_system, skip_tool_message=False
):
def text_count(self, *, skip_system: bool, skip_tool: bool) -> int:
is_tool_output: Final = self.type in TOOL_CALL_OUTPUT_ITEM_TYPES
role: Final = "tool" if is_tool_output else (self.role or "").lower()
if role_out_of_guardrail_scope(role, skip_system_message=skip_system, skip_tool_message=skip_tool):
return 0
if is_tool_output:
return len(tool_output_texts(self))
if isinstance(self.content, str):
return 1
if self.content is None:
@ -1692,13 +1701,14 @@ class PanwPrismaAirsHandler(CustomGuardrail):
request_data: Mapping[str, object],
*,
skip_system: bool,
skip_tool: bool,
) -> frozenset[int] | None:
"""Return the ``texts`` indices flattened from Responses ``reasoning`` input items.
The Responses translation handler gives those model-authored items the default
``user`` role, so the latest-turn selection must not mistake one for a human turn.
Empty for requests without a Responses ``input`` item list; None when the raw items
(after the leading ``instructions`` text, both minus whatever ``skip_system`` drops)
(after the leading ``instructions`` text, both minus whatever ``skip_system`` and ``skip_tool`` drop)
do not account for every entry of ``texts``.
"""
try:
@ -1708,7 +1718,7 @@ class PanwPrismaAirsHandler(CustomGuardrail):
if not isinstance(raw_input, tuple):
return frozenset()
offset: Final = 0 if scannable_instructions(request_data, skip_system=skip_system) is None else 1
counts: Final = tuple(item.text_count(skip_system=skip_system) for item in raw_input)
counts: Final = tuple(item.text_count(skip_system=skip_system, skip_tool=skip_tool) for item in raw_input)
if offset + sum(counts) != len(texts):
return None
starts: Final = itertools.accumulate(counts, initial=offset)
@ -1738,7 +1748,10 @@ class PanwPrismaAirsHandler(CustomGuardrail):
if sources is None:
return None
reasoning: Final = self._reasoning_item_text_indices(
texts, request_data, skip_system=effective_skip_system_message_for_guardrail(self)
texts,
request_data,
skip_system=effective_skip_system_message_for_guardrail(self),
skip_tool=effective_skip_tool_message_for_guardrail(self),
)
if reasoning is None:
return None

68
litellm/proxy/health_check.py
View file

@ -131,6 +131,52 @@ def _should_inject_health_check_max_tokens(model_info: Mapping[str, object], mod
return (mode or "chat") in _MAX_TOKEN_SUPPORT_MODES
_AUDIO_OUTPUT_PROBE_PROVIDERS: Final[frozenset[str]] = frozenset({"openai", "azure"})
def _health_check_requires_audio_output(
model_info: Mapping[str, object], probe_params: Mapping[str, object], mode: str | None, deployment_model: str | None
) -> bool:
if (mode or "chat") != "chat":
return False
operator_modalities: Final = probe_params.get("modalities")
if isinstance(operator_modalities, list) and "audio" not in operator_modalities:
return False
model: Final = probe_params.get("model")
if not isinstance(model, str):
return False
provider_param: Final = probe_params.get("custom_llm_provider")
try:
_, provider, _, _ = litellm.get_llm_provider(
model=model, custom_llm_provider=provider_param if isinstance(provider_param, str) else None
)
except litellm.exceptions.BadRequestError:
return False
if provider not in _AUDIO_OUTPUT_PROBE_PROVIDERS:
return False
base_model: Final = model_info.get("base_model") or probe_params.get("base_model")
checked_model: Final = base_model if isinstance(base_model, str) and model == deployment_model else model
checked_model_prefix: Final = checked_model.partition("/")[0]
return litellm.supports_audio_output(
model=checked_model,
custom_llm_provider=checked_model_prefix if checked_model_prefix in _AUDIO_OUTPUT_PROBE_PROVIDERS else provider,
)
def _health_check_audio_output_params(
model_info: Mapping[str, object], probe_params: Mapping[str, object], mode: str | None, deployment_model: str | None
) -> Mapping[str, object]:
if not _health_check_requires_audio_output(model_info, probe_params, mode, deployment_model):
return MappingProxyType({})
voice: Final = model_info.get("health_check_voice")
return MappingProxyType(
{
"modalities": ["text", "audio"],
"audio": {"voice": voice if isinstance(voice, str) else "alloy", "format": "wav"},
}
)
# Health-check modes that forward `reasoning_effort` to the provider (chat-style calls).
_HEALTH_CHECK_MODES_SUPPORTING_REASONING_EFFORT: Final = frozenset((None, "chat", "completion"))
@ -782,9 +828,12 @@ def _resolve_health_check_max_tokens(model_info: dict, litellm_params: dict) ->
_PROBE_PARAMS_ADAPTER: Final = TypeAdapter(dict[str, object])
_MODEL_INFO_ADAPTER: Final = TypeAdapter(Mapping[str, object])
def update_litellm_params_for_health_check(model_info: dict, litellm_params: dict) -> dict:
def update_litellm_params_for_health_check(
model_info: dict, litellm_params: dict, deployment_model: str | None = None
) -> dict:
"""
Update the litellm params for health check.
@ -800,7 +849,15 @@ def update_litellm_params_for_health_check(model_info: dict, litellm_params: dic
- updates the `model` param with the `health_check_model` if it exists Doc: https://docs.litellm.ai/docs/proxy/health#wildcard-routes
- updates the `voice` param with the `health_check_voice` for `audio_speech` mode if it exists Doc: https://docs.litellm.ai/docs/proxy/health#text-to-speech-models
- for Bedrock models with region routing (bedrock/region/model), strips the litellm routing prefix but preserves the model ID, and pins `custom_llm_provider` to `bedrock` (only when the deployment hasn't already set one, so an explicit `bedrock_converse` survives) so the bare model id still resolves to the provider (e.g. cross-region ids like `us.cohere.embed-v4:0`)
- asks OpenAI and Azure chat models that support audio output for `modalities` text + audio and a wav
`audio` in the `health_check_voice` (default "alloy"), since those models reject text-only requests.
`base_model` (from `model_info`, then `litellm_params`) decides it only while the probe still targets
`deployment_model` (default: the incoming `litellm_params` model), so a `health_check_model` or a `model`
in `health_check_params` is judged on its own. Keys already set through `health_check_params` are kept,
and `modalities` set there without "audio" turns the audio fields off
"""
incoming_model: Final = _PROBE_PARAMS_ADAPTER.validate_python(litellm_params).get("model")
base_model_owner: Final = deployment_model or (incoming_model if isinstance(incoming_model, str) else None)
mode: Final = resolve_health_check_mode(
model_info,
litellm_params, # any-ok: untyped router config dict
@ -875,7 +932,14 @@ def update_litellm_params_for_health_check(model_info: dict, litellm_params: dic
)
probe_params: Final = _PROBE_PARAMS_ADAPTER.validate_python(litellm_params)
return {key: value for key, value in probe_params.items() if not is_model_capability_field(key)}
audio_output_params: Final = _health_check_audio_output_params(
_MODEL_INFO_ADAPTER.validate_python(model_info), probe_params, mode, base_model_owner
)
return {
key: value
for key, value in {**audio_output_params, **probe_params}.items()
if not is_model_capability_field(key)
}
_update_litellm_params_for_health_check: Final = update_litellm_params_for_health_check

22
litellm/proxy/health_endpoints/_health_endpoints.py
View file

@ -32,7 +32,6 @@ from litellm.proxy._types import (
EnterpriseLicenseData,
Litellm_EntityType,
LitellmUserRoles,
ProxyErrorTypes,
ProxyException,
SpecialModelNames,
UserAPIKeyAuth,
@ -47,6 +46,7 @@ from litellm.proxy.auth.auth_utils import (
from litellm.proxy.auth.model_checks import get_key_models
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_utils import http_parsing_utils
from litellm.proxy.common_utils.openai_error_payload import proxy_exception_for
from litellm.proxy.db.db_lookup_gate import db_lookup_stall_tracker
from litellm.proxy.db.exception_handler import PrismaDBExceptionHandler
from litellm.proxy.db.health_check_latest import (
@ -683,21 +683,9 @@ async def health_services_endpoint(
except Exception as e:
verbose_proxy_logger.error("litellm.proxy.proxy_server.health_services_endpoint(): Exception occured - %s", e)
verbose_proxy_logger.debug(traceback.format_exc())
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_500_INTERNAL_SERVER_ERROR),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_500_INTERNAL_SERVER_ERROR) from e
def convert_health_check_to_dict(check) -> dict:
@ -2330,9 +2318,11 @@ async def test_model_connection(
stored_params=_OBJECT_MAPPING.validate_python(config_litellm_params),
request_params=_OBJECT_MAPPING.validate_python(request_litellm_params),
)
stored_model: Final = _OBJECT_MAPPING.validate_python(config_litellm_params).get("model")
litellm_params = update_litellm_params_for_health_check(
model_info=dict(probe_model_info),
litellm_params=litellm_params,
deployment_model=stored_model if isinstance(stored_model, str) else None,
)
## Auth check, on the final probe params so health_check_params cannot retarget it afterwards

6
litellm/proxy/litellm_pre_call_utils.py
View file

@ -3024,13 +3024,7 @@ async def add_litellm_data_to_request(
getattr(request.state, "litellm_roi_estimator", False) is True
)
verbose_proxy_logger.debug(
"[PROXY] returned data from litellm_pre_call_utils: %s",
data,
)
# Team/Project credential overrides from model_config
# Placed after the debug log to avoid leaking credential secrets in logs
_apply_credential_overrides_from_model_config(
data=data,
user_api_key_dict=user_api_key_dict,

19
litellm/proxy/management_endpoints/internal_user_endpoints.py
View file

@ -46,6 +46,7 @@ from litellm.proxy.auth.password_policy import (
)
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import evict_and_broadcast
from litellm.proxy.common_utils.openai_error_payload import proxy_exception_for
from litellm.proxy.common_utils.user_api_key_cache import (
AUTH_OBJECTS_TARGET,
object_permission_cache_key,
@ -1729,21 +1730,9 @@ async def user_update(
except Exception as e:
verbose_proxy_logger.exception("litellm.proxy.proxy_server.user_update(): Exception occured - %s", e)
verbose_proxy_logger.debug(traceback.format_exc())
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
async def bulk_update_processed_users(

55
litellm/proxy/management_endpoints/key_management_endpoints.py
View file

@ -75,6 +75,7 @@ from litellm.proxy.common_utils.config_sync_pubsub import (
coordination_redis_cache,
publish_config_change,
)
from litellm.proxy.common_utils.openai_error_payload import proxy_exception_for
from litellm.proxy.common_utils.rbac_utils import check_org_admin_can_generate_keys
from litellm.proxy.common_utils.timezone_utils import get_budget_reset_time
from litellm.proxy.common_utils.user_api_key_cache import AUTH_OBJECTS_TARGET, UserApiKeyCache
@ -4087,21 +4088,9 @@ async def update_key_fn(
# update based on remaining passed in values
except Exception as e:
verbose_proxy_logger.exception("litellm.proxy.proxy_server.update_key_fn(): Exception occured - %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
@router.post(
@ -7244,21 +7233,9 @@ async def list_keys(
except Exception as e:
verbose_proxy_logger.exception("Error in list_keys: %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"error({e})"),
type=ProxyErrorTypes.internal_server_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", fastapi.status.HTTP_500_INTERNAL_SERVER_ERROR),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.internal_server_error,
param=getattr(e, "param", "None"),
code=fastapi.status.HTTP_500_INTERNAL_SERVER_ERROR,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=fastapi.status.HTTP_500_INTERNAL_SERVER_ERROR) from e
async def _apply_non_admin_alias_scope(
@ -7402,21 +7379,9 @@ async def key_aliases(
except Exception as e:
verbose_proxy_logger.exception("Error in key_aliases: %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"error({e})"),
type=ProxyErrorTypes.internal_server_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_500_INTERNAL_SERVER_ERROR),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.internal_server_error,
param=getattr(e, "param", "None"),
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_500_INTERNAL_SERVER_ERROR) from e
def _validate_sort_params(

69
litellm/proxy/management_endpoints/model_management_endpoints.py
View file

@ -77,6 +77,7 @@ from litellm.proxy.common_utils.encrypt_decrypt_utils import (
decrypt_value_helper,
encrypt_value_helper,
)
from litellm.proxy.common_utils.openai_error_payload import proxy_exception_for
from litellm.proxy.common_utils.user_api_key_cache import UserApiKeyCache
from litellm.proxy.db.routing_prisma_wrapper import WriterPinnedClient
from litellm.proxy.management.teams.authz import TEAM_ADMIN_ONLY, is_team_admin
@ -2845,21 +2846,9 @@ async def delete_model(
except Exception as e:
verbose_proxy_logger.exception("Failed to delete model. Due to error - %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
async def delete_team_model_alias(
@ -2899,6 +2888,13 @@ async def delete_team_model_alias(
return removed_model_aliases
def _model_not_added_to_db() -> HTTPException:
return HTTPException(
status_code=500,
detail={"error": "Failed to add model to db. Check your server logs for more details."},
)
@router.post(
"/model/new",
description="Allows adding new models to the model list in the config.yaml",
@ -3080,6 +3076,8 @@ async def add_new_model(
if isinstance(e, HTTPException):
raise
verbose_proxy_logger.exception("Exception in add_new_model: %s", e)
if model_response is None:
raise _model_not_added_to_db() from e
else:
raise HTTPException(
@ -3088,10 +3086,7 @@ async def add_new_model(
)
if model_response is None:
raise HTTPException(
status_code=500,
detail={"error": "Failed to add model to db. Check your server logs for more details."},
)
raise _model_not_added_to_db()
## CREATE AUDIT LOG ##
asyncio.create_task(
@ -3121,21 +3116,9 @@ async def add_new_model(
except Exception as e:
verbose_proxy_logger.exception("litellm.proxy.proxy_server.add_new_model(): Exception occured - %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
#### MODEL MANAGEMENT ####
@ -3364,21 +3347,9 @@ async def update_model(
return None if model_response is None else _model_write_response(model_response, member_write)
except Exception as e:
verbose_proxy_logger.exception("litellm.proxy.proxy_server.update_model(): Exception occured - %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
@router.post(

19
litellm/proxy/management_endpoints/organization_endpoints.py
View file

@ -41,6 +41,7 @@ from litellm.proxy.auth.auth_checks import (
)
from litellm.proxy.auth.user_api_key_auth import user_api_key_auth
from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import evict_and_broadcast
from litellm.proxy.common_utils.openai_error_payload import proxy_exception_for
from litellm.proxy.common_utils.timezone_utils import get_budget_reset_time
from litellm.proxy.common_utils.user_api_key_cache import UserApiKeyCache
from litellm.proxy.management_endpoints.budget_management_endpoints import (
@ -1396,21 +1397,9 @@ async def organization_member_add(
)
except Exception as e:
verbose_proxy_logger.exception("Error adding member to organization: %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_500_INTERNAL_SERVER_ERROR),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_500_INTERNAL_SERVER_ERROR,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_500_INTERNAL_SERVER_ERROR) from e
async def find_member_if_email(user_email: str, prisma_client: PrismaClient) -> LiteLLM_UserTable:

19
litellm/proxy/management_endpoints/team_endpoints.py
View file

@ -132,6 +132,7 @@ from litellm.proxy.common_utils.auth_cache_invalidation_pubsub import (
)
from litellm.proxy.common_utils.callback_utils import encrypt_callback_vars
from litellm.proxy.common_utils.json_merge_patch import apply_json_merge_patch
from litellm.proxy.common_utils.openai_error_payload import proxy_exception_for
from litellm.proxy.common_utils.user_api_key_cache import (
UserApiKeyCache,
team_membership_auth_cache_key,
@ -5526,21 +5527,9 @@ async def team_info(
e,
traceback.format_exc(),
)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
@router.get(

15
litellm/proxy/pass_through_endpoints/llm_passthrough_endpoints.py
View file

@ -50,6 +50,7 @@ from litellm.constants import (
from litellm.exceptions import BlockedPiiEntityError, GuardrailRaisedException
from litellm.integrations.custom_guardrail import guardrail_request_data_with_streaming, is_guardrail_intervention
from litellm.litellm_core_utils.aws_partition import get_aws_dns_suffix
from litellm.litellm_core_utils.fallback_generalizations import match_routing_generalization
from litellm.llms.anthropic.common_utils import AnthropicModelInfo, merge_anthropic_beta_headers
from litellm.llms.azure.passthrough.transformation import (
foreign_azure_deployment,
@ -698,6 +699,20 @@ async def typesafe_proxy_route(
return await endpoint_func(request, fastapi_response, user_api_key_dict)
_COST_MAP_ENTRIES: Final = TypeAdapter(Mapping[str, object])
def typesafe_allowlist_model_names() -> tuple[str, ...]:
if not passthrough_endpoint_router.has_credentials(custom_llm_provider="typesafe"):
return ()
cost_map: Final = _COST_MAP_ENTRIES.validate_python(litellm.model_cost) # pyright: ignore[reportUnknownMemberType] # legacy registry is validated at this boundary
priced_names: Final = (name.removeprefix("typesafe/") for name in cost_map if name.startswith("typesafe/"))
jev_models_typesafe_serves: Final = sorted(
name for name in priced_names if match_routing_generalization(name) == "typesafe"
)
return ("typesafe/*", *jev_models_typesafe_serves)
@router.post(
"/laya/v1/systemone",
tags=["Laya Pass-through", "pass-through"],

9
litellm/proxy/pass_through_endpoints/passthrough_endpoint_router.py
View file

@ -1,4 +1,5 @@
import json
import os
from collections.abc import Callable
from typing import TYPE_CHECKING, Final
@ -88,6 +89,14 @@ class PassthroughEndpointRouter:
)
return _resolve_oidc_reference(get_secret_str(_env_variable_name))
def has_credentials(self, custom_llm_provider: str) -> bool:
if self._get_deployment_api_key(custom_llm_provider=custom_llm_provider, region_name=None) is not None:
return True
env_variable_name: Final = self._get_default_env_variable_name_passthrough_endpoint(
custom_llm_provider=custom_llm_provider,
)
return bool(os.environ.get(env_variable_name))
def _get_deployment_api_key(
self,
custom_llm_provider: str,

84
litellm/proxy/proxy_server.py
View file

@ -524,6 +524,7 @@ from litellm.proxy.common_utils.openai_error_payload import (
LITELLM_CALL_ID_HEADER,
headers_with_litellm_call_id,
litellm_call_id_headers,
proxy_exception_for,
with_litellm_call_id,
)
from litellm.proxy.common_utils.periodic_reload_schedule import (
@ -6606,7 +6607,7 @@ class ProxyConfig:
reset_color_code: Final = "\033[0m"
for key, value in litellm_settings.items():
if key == "cache" and value is True:
print_startup_message(f"{blue_color_code}\nSetting Cache on Proxy")
print_startup_message(f"{blue_color_code}\nSetting Cache on Proxy{reset_color_code}")
from litellm.caching.caching import Cache
cache_params = {}
@ -9825,7 +9826,7 @@ async def async_assistants_data_generator(response, user_api_key_dict: UserAPIKe
request_data=request_data,
)
verbose_proxy_logger.debug(
"\x1b[1;31mAn error occurred: %s\n\n Debug this by setting `--debug`, e.g. `litellm --model gpt-3.5-turbo --debug`",
"An error occurred: %s",
e,
)
if isinstance(e, HTTPException):
@ -10655,7 +10656,7 @@ async def async_data_generator(
)
client_error: Final[Exception] = e if replacement is None else replacement
verbose_proxy_logger.debug(
"\x1b[1;31mAn error occurred: %s\n\n Debug this by setting `--debug`, e.g. `litellm --model gpt-3.5-turbo --debug`",
"An error occurred: %s",
e,
)
@ -12390,6 +12391,12 @@ async def model_list(
hidden_names: Final = blocked_names | unhealthy_names
from litellm.proxy.pass_through_endpoints.llm_passthrough_endpoints import typesafe_allowlist_model_names
asks_for_allowlist_names: Final = (
include_model_access_groups is True and return_wildcard_routes is True and only_model_access_groups is not True
)
# If scope=expand and user has admin privileges, return all proxy models
if should_expand_scope:
# Get all proxy models as if user is a proxy admin
@ -12410,7 +12417,10 @@ async def model_list(
all_models = get_complete_model_list(
key_models=[],
team_models=[],
proxy_model_list=proxy_model_list,
proxy_model_list=[
*proxy_model_list,
*(typesafe_allowlist_model_names() if asks_for_allowlist_names else ()),
],
user_model=None,
infer_model_from_keys=False,
return_wildcard_routes=include_wildcard_routes,
@ -12471,7 +12481,7 @@ async def model_list(
# Otherwise, use the normal behavior (current implementation)
# Get available models for the user
all_models = await get_available_models_for_user(
available_models: Final = await get_available_models_for_user(
user_api_key_dict=user_api_key_dict,
llm_router=llm_router,
general_settings=general_settings,
@ -12484,6 +12494,11 @@ async def model_list(
return_wildcard_routes=include_wildcard_routes,
user_api_key_cache=user_api_key_cache,
)
offers_typesafe_allowlist_names: Final = (
asks_for_allowlist_names and team_id is None and user_api_key_has_admin_view(user_api_key_dict)
)
typesafe_names_to_offer: Final = typesafe_allowlist_model_names() if offers_typesafe_allowlist_names else ()
all_models = [*available_models, *(name for name in typesafe_names_to_offer if name not in available_models)]
undiscoverable_names: Final = undiscoverable_model_names(
all_models, llm_router, user_api_key_dict, team_id or user_api_key_dict.team_id
@ -18437,7 +18452,6 @@ async def async_queue_request(
"body": copy.copy(data), # use copy instead of deepcopy
}
verbose_proxy_logger.debug("receiving data: %s", data)
data["model"] = (
general_settings.get("completion_model", None) # server default
or user_model # model name passed via cli args
@ -18523,25 +18537,13 @@ async def async_queue_request(
await proxy_logging_obj.post_call_failure_hook(
user_api_key_dict=user_api_key_dict, original_exception=e, request_data=data
)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
elif isinstance(e, litellm.BadRequestError):
if isinstance(e, litellm.BadRequestError):
raise ProxyException(
message=e.message, type="invalid_request_error", param=getattr(e, "param", None), code=e.status_code
)
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
) from e
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
@app.get("/fallback/login", tags=["experimental"], include_in_schema=False)
@ -19820,21 +19822,9 @@ async def update_config(
except Exception as e:
verbose_proxy_logger.error("litellm.proxy.proxy_server.update_config(): Exception occured - %s", e)
verbose_proxy_logger.debug(traceback.format_exc())
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_400_BAD_REQUEST) from e
### CONFIG GENERAL SETTINGS
@ -21017,21 +21007,9 @@ async def get_config(
}
except Exception as e:
verbose_proxy_logger.exception("litellm.proxy.proxy_server.get_config(): Exception occured - %s", e)
if isinstance(e, HTTPException):
raise ProxyException(
message=getattr(e, "detail", f"Authentication Error({e})"),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=getattr(e, "status_code", status.HTTP_400_BAD_REQUEST),
)
elif isinstance(e, ProxyException):
raise e
raise ProxyException(
message="Authentication Error, " + str(e),
type=ProxyErrorTypes.auth_error,
param=getattr(e, "param", "None"),
code=status.HTTP_400_BAD_REQUEST,
)
if isinstance(e, ProxyException):
raise
raise proxy_exception_for(e, default_status_code=status.HTTP_500_INTERNAL_SERVER_ERROR) from e
@router.get(

4
litellm/proxy/public_endpoints/agent_create_fields.json
View file

@ -127,8 +127,8 @@
{
"key": "agent_id",
"label": "Agent ID",
"placeholder": "asst_abc123",
"tooltip": "The agent/assistant ID from your Azure AI Foundry project (e.g., asst_abc123)",
"placeholder": "my-agent",
"tooltip": "The agent name from your Azure AI Foundry project, optionally with a version (e.g., my-agent or my-agent:2), or a classic assistant ID (e.g., asst_abc123)",
"required": true,
"field_type": "text",
"default_value": null,

28
litellm/proxy/public_endpoints/provider_create_fields.json
View file

@ -3519,6 +3519,34 @@
],
"default_model_placeholder": "inferenceapis/deepseek-ai/DeepSeek-V4-Flash"
},
{
"provider": "StreamLake",
"provider_display_name": "StreamLake",
"litellm_provider": "streamlake",
"credential_fields": [
{
"key": "api_base",
"label": "API Base",
"placeholder": "https://vanchin.streamlake.ai/api/gateway/v1/endpoints",
"tooltip": null,
"required": false,
"field_type": "text",
"options": null,
"default_value": null
},
{
"key": "api_key",
"label": "API Key",
"placeholder": null,
"tooltip": null,
"required": true,
"field_type": "password",
"options": null,
"default_value": null
}
],
"default_model_placeholder": "streamlake/<model-or-endpoint>"
},
{
"provider": "Snowflake",
"provider_display_name": "Snowflake",

4
litellm/responses/bridge_check.py
View file

@ -9,6 +9,7 @@ from litellm.llms.azure_ai.chat.transformation import AzureAIGPT5Config
from litellm.llms.azure_ai.common_utils import (
azure_ai_supports_native_responses,
foundry_chat_rejects_function_tools_while_reasoning,
is_foundry_project_agent,
)
from litellm.llms.openai.chat.gpt_reasoning_transformation import OpenAIGPT5Config
from litellm.secret_managers.main import get_secret_str
@ -181,6 +182,9 @@ def responses_api_bridge_check(
elif pinned_mode == "chat" and backend_serves_chat_completions(model_info):
model_info["mode"] = "chat"
if custom_llm_provider == LlmProviders.AZURE_AI.value and is_foundry_project_agent(model):
model_info["mode"] = "responses"
if web_search_options is not None and custom_llm_provider == "xai":
model_info["mode"] = "responses"
model = model.replace("responses/", "")

110
litellm/responses/litellm_completion_transformation/transformation.py
View file

@ -321,7 +321,7 @@ def _responses_file_part(item: Mapping[str, object]) -> _ResponsesFilePart:
)
def _decode_responses_media(item: Mapping[str, object]) -> _ResponsesMediaPart | None:
def decode_responses_media(item: Mapping[str, object]) -> _ResponsesMediaPart | None:
kind: Final = item.get("type")
if kind in ("input_image", "image_url", "computer_screenshot") and any(key in item for key in ("image_url", "url")):
image: Final = _responses_image_part(item)
@ -383,6 +383,52 @@ def _attribute_fields(value: object) -> dict[str, object]:
return dict(cast("Iterable[tuple[str, object]]", value)) # cast-ok: dict() raises on non-pair values, as before
def tool_output_to_chat_content(
output: object,
) -> str | list[ToolMessageContentPart]:
if output is None:
return ""
if isinstance(output, str):
return output
single_media: Final = decode_responses_media(output) if isinstance(output, Mapping) else None
if single_media is not None:
return [single_media]
if isinstance(output, list):
normalized_blocks: Final[list[ToolMessageContentPart]] = []
text_acc: Final[list[str]] = []
for part in output:
if not isinstance(part, dict):
continue
part_type = part.get("type")
if part_type in ("input_text", "output_text", "text"):
txt = part.get("text")
if isinstance(txt, str) and txt:
text_acc.append(txt)
normalized_blocks.append({"type": "text", "text": txt})
elif (media := decode_responses_media(part)) is not None:
normalized_blocks.append(media)
if any(b.get("type") != "text" for b in normalized_blocks):
return normalized_blocks
if text_acc:
return "".join(text_acc)
try:
return json.dumps(output)
except Exception:
return str(output)
try:
return json.dumps(output)
except Exception:
return str(output)
TOOL_CALL_OUTPUT_ITEM_TYPES: Final = frozenset(
{"function_call_output", "custom_tool_call_output", "computer_call_output", "tool_result"}
)
def _input_item_role(input_item: Mapping[str, object]) -> str:
return cast(str, input_item.get("role") or "user") # cast-ok: client-supplied role forwarded verbatim, unvalidated
@ -2371,12 +2417,7 @@ class LiteLLMCompletionResponsesConfig:
"""
Check if the input item is a tool call output
"""
return input_item.get("type") in [
"function_call_output",
"custom_tool_call_output",
"computer_call_output",
"tool_result", # Anthropic/MCP format
]
return input_item.get("type") in TOOL_CALL_OUTPUT_ITEM_TYPES
@staticmethod
def _is_input_item_function_call(input_item: Mapping[str, object]) -> bool:
@ -2400,58 +2441,7 @@ class LiteLLMCompletionResponsesConfig:
if not call_id:
return []
def _normalize_function_call_output_to_tool_content(
output: object,
) -> str | list[ToolMessageContentPart]:
"""
Preserve structured media for downstream chat adapters and token estimation.
Text-only outputs retain their existing string representation.
"""
if output is None:
return ""
if isinstance(output, str):
return output
single_media: Final = _decode_responses_media(output) if isinstance(output, Mapping) else None
if single_media is not None:
return [single_media]
# Some adapters represent tool output as a list of "input_*" parts
if isinstance(output, list):
normalized_blocks: Final[list[ToolMessageContentPart]] = []
text_acc: Final[list[str]] = []
for part in output:
if not isinstance(part, dict):
continue
part_type = part.get("type")
if part_type in ("input_text", "output_text", "text"):
txt = part.get("text")
if isinstance(txt, str) and txt:
text_acc.append(txt)
normalized_blocks.append({"type": "text", "text": txt})
elif (media := _decode_responses_media(part)) is not None:
normalized_blocks.append(media)
if any(b.get("type") != "text" for b in normalized_blocks):
return normalized_blocks
if text_acc:
return "".join(text_acc)
try:
# last resort: keep something meaningful for providers that require a string
import json as _json
return _json.dumps(output)
except Exception:
return str(output)
# Fallback for dict/number/etc.
try:
import json as _json
return _json.dumps(output)
except Exception:
return str(output)
tool_content: Final = _normalize_function_call_output_to_tool_content(tool_call_output.get("output"))
tool_content: Final = tool_output_to_chat_content(tool_call_output.get("output"))
output_cache_control: Final = _input_item_cache_control(tool_call_output)
tool_output_message: Final = (
ChatCompletionToolMessage(role="tool", content=tool_content, tool_call_id=str(call_id))
@ -2606,7 +2596,7 @@ class LiteLLMCompletionResponsesConfig:
content_list.append(item)
elif isinstance(item, dict):
if (
media := _decode_responses_media(
media := decode_responses_media(
fields := _validate_responses_media(_STR_KEY_DICT_ADAPTER, item)
)
) is not None:

41
litellm/responses/streaming_iterator.py
View file

@ -39,7 +39,11 @@ from litellm.constants import (
)
from litellm.exceptions import MidStreamFallbackError, RateLimitError
from litellm.litellm_core_utils.asyncify import run_async_function
from litellm.litellm_core_utils.core_helpers import process_response_headers
from litellm.litellm_core_utils.core_helpers import (
process_response_headers,
provider_reported_usage_cost,
set_response_cost_in_hidden_params,
)
from litellm.litellm_core_utils.exception_mapping_utils import with_deployment_api_base
from litellm.litellm_core_utils.internal_call_metadata import (
is_background_response,
@ -67,6 +71,7 @@ from litellm.types.llms.openai import (
ResponsesAPIResponse,
ResponsesAPIStreamEvents,
ResponsesAPIStreamingResponse,
replayable_responses_input,
)
from litellm.types.responses.main import GenericResponseOutputItem, OutputText
from litellm.types.utils import CallTypes, ModelResponse
@ -1825,20 +1830,32 @@ def _estimate_usage_safely(
def stamp_responses_usage_cost(
response_obj: ResponsesAPIResponse | None, logging_obj: LiteLLMLoggingObj | None
) -> None:
if response_obj is None or logging_obj is None:
"""The terminal response's ``usage.cost`` is the bill litellm writes to the spend row. It is priced on a copy
shaped the way the success handler shapes the spend row's result (the reported ``usage.cost`` in the
response-cost header, the usage in chat shape), so the client and the row read the same number."""
if response_obj is None:
return
usage_obj: Final[ResponseAPIUsage | None] = _usage_as_model(getattr(response_obj, "usage", None))
if usage_obj is None:
return
response_obj.usage = usage_obj # rebind-ok: the stamped cost has to ride on the response the client receives
if isinstance(getattr(usage_obj, "cost", None), (int, float)):
response_obj.usage = usage_obj # rebind-ok: the typed usage has to ride on the response the client receives
if logging_obj is None:
return
priced: Final = response_obj.model_copy(
update={"usage": ResponseAPILoggingUtils.transform_response_api_usage_to_chat_usage(usage_obj)}
)
priced.hidden_params = {}
set_response_cost_in_hidden_params(
priced, provider_reported_usage_cost(usage_obj, logging_obj.model_call_details.get("custom_llm_provider"))
)
try:
cost: Final[float | None] = logging_obj.response_cost_calculator(result=response_obj)
except Exception:
bill: Final = logging_obj.response_cost_calculator(result=priced)
except Exception: # noqa: BLE001 # a pricing failure leaves the client usage as the upstream sent it
return
if isinstance(cost, (int, float)) and cost > 0:
setattr(usage_obj, "cost", cost)
if isinstance(bill, bool) or not isinstance(bill, (int, float)):
return
if bill > 0 or isinstance(getattr(usage_obj, "cost", None), (int, float)):
setattr(usage_obj, "cost", float(bill))
def opening_item_payload(payload: Mapping[str, object]) -> Mapping[str, object]:
@ -2349,15 +2366,17 @@ def _ws_event_error(event: Mapping[str, object]) -> object:
return response.get("error") if _is_json_object(response) else None
def _restore_input_item_ids(items: Sequence[object]) -> Sequence[object]:
return ResponsesAPIRequestUtils.restore_encrypted_content_item_ids_in_input(copy.deepcopy(list(items)))
def _replayable_input_items(items: Sequence[object]) -> object:
return replayable_responses_input(
ResponsesAPIRequestUtils.restore_encrypted_content_item_ids_in_input(copy.deepcopy(list(items)))
)
def _restored_container_fields(container: Mapping[str, object]) -> Mapping[str, object]:
input_items: Final = container.get("input")
previous_response_id: Final = container.get("previous_response_id")
restored: Final = {
"input": _restore_input_item_ids(input_items) if _is_json_array(input_items) else input_items,
"input": _replayable_input_items(input_items) if _is_json_array(input_items) else input_items,
"previous_response_id": (
ResponsesAPIRequestUtils.decode_previous_response_id_to_original_previous_response_id(previous_response_id)
if isinstance(previous_response_id, str)

6
litellm/responses/utils.py
View file

@ -766,6 +766,12 @@ class ResponsesAPIRequestUtils:
model_id=model_id,
response_id=decoded_response_id,
)
except ValueError:
return DecodedResponseId(
custom_llm_provider=None,
model_id=None,
response_id=response_id,
)
except Exception as e:
verbose_logger.debug("Error decoding response_id '%s': %s", response_id, e)
return DecodedResponseId(

35
litellm/router.py
View file

@ -4921,6 +4921,7 @@ class Router:
deployment=deployment, forwarded_keys=kwargs.pop(_ALIAS_MARKER_FORWARDED_PARAMS_KWARG, ())
):
kwargs.pop(key, None)
kwargs.pop("daily_reset_time", None) # pyright: ignore[reportUnknownMemberType] # kwargs is an untyped dict
defaults: Final = _LITELLM_PARAMS_ADAPTER.validate_python(self.default_litellm_params)
self._merge_tools_from_deployment(deployment=deployment, kwargs=kwargs, defaults=defaults)
@ -5668,7 +5669,7 @@ class Router:
def _image_generation(self, prompt: str, model: str, **kwargs):
model_name: Final = ""
try:
verbose_router_logger.debug("Inside _image_generation()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _image_generation()- model: %s", model)
deployment: Final = self.get_available_deployment(
model=model,
messages=[{"role": "user", "content": "prompt"}],
@ -5730,7 +5731,7 @@ class Router:
async def _aimage_generation(self, prompt: str, model: str, **kwargs):
model_name = model
try:
verbose_router_logger.debug("Inside _image_generation()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _image_generation()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
deployment: Final = await self.async_get_available_deployment(
model=model,
@ -5815,7 +5816,7 @@ class Router:
async def _atranscription(self, file: FileTypes, model: str, **kwargs):
model_name: Final = model
try:
verbose_router_logger.debug("Inside _atranscription()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _atranscription()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
deployment: Final = await self.async_get_available_deployment(
model=model,
@ -5908,7 +5909,7 @@ class Router:
async def _aspeech(self, model: str, input: str, voice: str | None = None, **kwargs):
model_name: Final = model
try:
verbose_router_logger.debug("Inside _aspeech()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _aspeech()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
deployment: Final = await self.async_get_available_deployment(
model=model,
@ -5970,7 +5971,7 @@ class Router:
async def _arerank(self, model: str, **kwargs):
model_name = None
try:
verbose_router_logger.debug("Inside _rerank()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _rerank()- model: %s", model)
deployment: Final = await self.async_get_available_deployment(
model=model,
specific_deployment=kwargs.pop("specific_deployment", None),
@ -6079,7 +6080,7 @@ class Router:
async def _atext_completion(self, model: str, prompt: str, **kwargs):
try:
verbose_router_logger.debug("Inside _atext_completion()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _atext_completion()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
deployment: Final = await self.async_get_available_deployment(
model=model,
@ -6149,7 +6150,7 @@ class Router:
async def _aadapter_completion(self, adapter_id: str, model: str, **kwargs):
try:
verbose_router_logger.debug("Inside _aadapter_completion()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _aadapter_completion()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
deployment: Final = await self.async_get_available_deployment(
model=model,
@ -6243,7 +6244,7 @@ class Router:
kwargs=kwargs,
metadata_variable_name="litellm_metadata",
)
verbose_router_logger.debug("Inside aguardrail() - guardrail_name: %s; kwargs: %s", guardrail_name, kwargs)
verbose_router_logger.debug("Inside aguardrail() - guardrail_name: %s", guardrail_name)
response: Final = await self.async_function_with_fallbacks(**kwargs)
return response
@ -6324,9 +6325,7 @@ class Router:
controls: Final = per_request_fallback_controls(kwargs)
kwargs[MID_STREAM_FALLBACK_CONTROLS_KEY] = controls # rebind-ok: forwarded to every hop
self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs, metadata_variable_name="litellm_metadata")
verbose_router_logger.debug(
"Inside ageneric_api_call_with_fallbacks() - model: %s; kwargs: %s", model, kwargs
)
verbose_router_logger.debug("Inside ageneric_api_call_with_fallbacks() - model: %s", model)
response: Final = await self.async_function_with_fallbacks(**kwargs)
return response
except Exception as e:
@ -7146,9 +7145,7 @@ class Router:
handler_name: Final = original_function.__name__
metadata_variable_name: Final = get_router_metadata_variable_name(function_name="generic_api_call")
try:
verbose_router_logger.debug(
"Inside _generic_api_call() - handler: %s, model: %s; kwargs: %s", handler_name, model, kwargs
)
verbose_router_logger.debug("Inside _generic_api_call() - handler: %s, model: %s", handler_name, model)
self._update_kwargs_before_fallbacks(
model=model,
kwargs=kwargs,
@ -7223,7 +7220,7 @@ class Router:
def _embedding(self, input: str | list[object], model: str, **kwargs):
model_name = None
try:
verbose_router_logger.debug("Inside embedding()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside embedding()- model: %s", model)
with embedding_deployment_selection():
deployment: Final = self.get_available_deployment(
model=model,
@ -7312,7 +7309,7 @@ class Router:
async def _aembedding(self, input: str | list[object], model: str, **kwargs):
model_name = None
try:
verbose_router_logger.debug("Inside _aembedding()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _aembedding()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
with embedding_deployment_selection():
deployment: Final = await self.async_get_available_deployment(
@ -7396,7 +7393,7 @@ class Router:
try:
from litellm.router_utils.common_utils import add_model_file_id_mappings
verbose_router_logger.debug("Inside _atext_completion()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _atext_completion()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
healthy_deployments: Final = await self.async_get_healthy_deployments(
model=model,
@ -7634,7 +7631,7 @@ class Router:
**kwargs,
) -> LiteLLMBatch:
try:
verbose_router_logger.debug("Inside _acreate_batch()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _acreate_batch()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
with batch_create_routing():
deployment: Final = await self.async_get_available_deployment(
@ -7833,7 +7830,7 @@ class Router:
**kwargs,
) -> LiteLLMBatch:
try:
verbose_router_logger.debug("Inside _acancel_batch()- model: %s; kwargs: %s", model, kwargs)
verbose_router_logger.debug("Inside _acancel_batch()- model: %s", model)
parent_otel_span: Final = get_parent_otel_span_from_kwargs(kwargs)
deployment: Final = await self.async_get_available_deployment(
model=model,

64
litellm/router_utils/pre_call_checks/model_rate_limit_check.py
View file

@ -12,7 +12,7 @@ is logged the first time such a deployment is seen.
"""
import contextlib
from collections.abc import Callable
from collections.abc import Callable, Mapping
from datetime import datetime, timedelta
from typing import TYPE_CHECKING, Any, Final, Literal
@ -38,7 +38,7 @@ from litellm.router_utils.pre_call_checks.io_token_rate_limit_check import (
)
from litellm.router_utils.routing_read_batch import ROUTER_USAGE_TARGET
from litellm.types.caching import RedisPipelineIncrementOperation
from litellm.types.router import RouterErrors
from litellm.types.router import RouterErrors, parse_daily_reset_time
from litellm.types.utils import StandardLoggingPayload
from litellm.utils import get_utc_datetime
@ -58,42 +58,49 @@ class RoutingArgs:
DAY_FORMAT: Final = "%Y-%m-%d"
def seconds_until_utc_midnight(now: datetime) -> int:
next_midnight: Final = (now + timedelta(days=1)).replace(hour=0, minute=0, second=0, microsecond=0)
return max(1, int((next_midnight - now).total_seconds()))
def daily_window_day(now: datetime, reset_offset: timedelta) -> str:
return (now - reset_offset).strftime(DAY_FORMAT)
def _first_set(*values: int | None) -> int | None:
return next((value for value in values if value is not None), None)
def seconds_until_daily_reset(now: datetime, reset_offset: timedelta = timedelta(0)) -> int:
window_start: Final = (now - reset_offset).replace(hour=0, minute=0, second=0, microsecond=0)
next_reset: Final = window_start + timedelta(days=1) + reset_offset
return max(1, int((next_reset - now).total_seconds()))
class _DailyLimitFields(BaseModel):
class _DailyLimitParams(BaseModel):
"""rpd, tpd and daily_reset_time live in litellm_params only."""
model_config = ConfigDict(frozen=True, extra="ignore", from_attributes=True)
model: str | None = None
tpd: int | None = None
rpd: int | None = None
daily_reset_time: str | None = None
def reset_offset(self) -> timedelta:
return parse_daily_reset_time(self.daily_reset_time)
class _DailyLimitParams(_DailyLimitFields):
model: str | None = None
class _DailyLimitModelInfo(_DailyLimitFields):
class _DailyLimitModelInfo(BaseModel):
model_config = ConfigDict(frozen=True, extra="ignore", from_attributes=True)
id: str | None = None
class _DailyLimitDeployment(_DailyLimitFields):
class _DailyLimitDeployment(BaseModel):
model_config = ConfigDict(frozen=True, extra="ignore", from_attributes=True)
model_name: str = ""
litellm_params: _DailyLimitParams = _DailyLimitParams()
model_info: _DailyLimitModelInfo = _DailyLimitModelInfo()
def limits(self) -> tuple[int | None, int | None]:
return (
_first_set(self.tpd, self.litellm_params.tpd, self.model_info.tpd),
_first_set(self.rpd, self.litellm_params.rpd, self.model_info.rpd),
)
return self.litellm_params.tpd, self.litellm_params.rpd
def cache_keys(self, current_day: str) -> tuple[str, str]:
def reset_offset(self) -> timedelta:
return self.litellm_params.reset_offset()
def cache_keys(self, now: datetime) -> tuple[str, str]:
prefix: Final = f"{self.model_info.id}:{self.litellm_params.model}"
current_day: Final = daily_window_day(now, self.reset_offset())
return f"{prefix}:tpd:{current_day}", f"{prefix}:rpd:{current_day}"
def limit_error(self, kind: Literal["tpd", "rpd"], limit: int, usage: int, now: datetime) -> litellm.RateLimitError:
@ -104,12 +111,19 @@ class _DailyLimitDeployment(_DailyLimitFields):
response=httpx.Response(
status_code=429,
content=f"{RouterErrors.user_defined_ratelimit_error.value} {kind} limit={limit}. current usage={usage}. id={self.model_info.id}, model_group={self.model_name}",
headers={"retry-after": str(seconds_until_utc_midnight(now))},
headers={"retry-after": str(seconds_until_daily_reset(now, self.reset_offset()))},
request=httpx.Request(method="model_rate_limit_check", url="https://github.com/BerriAI/litellm"),
),
)
def _reset_offset_from_kwargs(kwargs: Mapping[str, object]) -> timedelta:
litellm_params: Final = kwargs.get("litellm_params")
if litellm_params is None:
return timedelta(0)
return _DailyLimitParams.model_validate(litellm_params).reset_offset()
class ModelRateLimitingCheck(CustomLogger):
"""
Pre-call check that enforces TPM/RPM or ITPM/OTPM limits on model deployments.
@ -242,7 +256,7 @@ class ModelRateLimitingCheck(CustomLogger):
tpd_limit, rpd_limit = deployment.limits()
if tpd_limit is None and rpd_limit is None:
return
tpd_key, rpd_key = deployment.cache_keys(now.strftime(DAY_FORMAT))
tpd_key, rpd_key = deployment.cache_keys(now)
if tpd_limit is not None:
current_tpd: Final = self._get_current_tpm(tpd_key, tpd_limit)
if current_tpd is not None and current_tpd >= tpd_limit:
@ -394,7 +408,7 @@ class ModelRateLimitingCheck(CustomLogger):
current_minute: Final = dt.strftime("%H-%M")
tpm_key, rpm_key = self._get_cache_keys(deployment, current_minute)
tpd_key, rpd_key = daily.cache_keys(dt.strftime(DAY_FORMAT))
tpd_key, rpd_key = daily.cache_keys(dt)
model_id: Final = deployment.get("model_info", {}).get("id")
model_name: Final = deployment.get("litellm_params", {}).get("model")
@ -510,7 +524,8 @@ class ModelRateLimitingCheck(CustomLogger):
dt: Final = self.clock()
current_minute: Final = dt.strftime("%H-%M")
tpm_key: Final = f"{model_id}:{model}:tpm:{current_minute}"
tpd_key: Final = f"{model_id}:{model}:tpd:{dt.strftime(DAY_FORMAT)}"
reset_offset: Final = _reset_offset_from_kwargs(kwargs) # pyright: ignore[reportUnknownArgumentType] # CustomLogger passes an untyped kwargs dict
tpd_key: Final = f"{model_id}:{model}:tpd:{daily_window_day(dt, reset_offset)}"
verbose_router_logger.debug("[TPM TRACKING] Incrementing %s by %s", tpm_key, total_tokens)
@ -570,7 +585,8 @@ class ModelRateLimitingCheck(CustomLogger):
dt: Final = self.clock()
current_minute: Final = dt.strftime("%H-%M")
tpm_key: Final = f"{model_id}:{model}:tpm:{current_minute}"
tpd_key: Final = f"{model_id}:{model}:tpd:{dt.strftime(DAY_FORMAT)}"
reset_offset: Final = _reset_offset_from_kwargs(kwargs) # pyright: ignore[reportUnknownArgumentType] # CustomLogger passes an untyped kwargs dict
tpd_key: Final = f"{model_id}:{model}:tpd:{daily_window_day(dt, reset_offset)}"
tokens: Final = int(total_tokens)
self._increment(tpm_key, tokens, RoutingArgs.ttl)

3
litellm/router_utils/search_api_router.py
View file

@ -155,10 +155,9 @@ class SearchAPIRouter:
available_search_tool_names: Final = [tool.get("search_tool_name") for tool in router_instance.search_tools]
verbose_router_logger.debug(
"Inside SearchAPIRouter.async_search_with_fallbacks() - search_tool_name: %s, Available Search Tools: %s, kwargs: %s",
"Inside SearchAPIRouter.async_search_with_fallbacks() - search_tool_name: %s, Available Search Tools: %s",
search_tool_name,
available_search_tool_names,
kwargs,
)
# Use the existing retry/fallback infrastructure

3
litellm/rust_bridge/host/routes/ocr.py
View file

@ -7,12 +7,13 @@ from typing import Final
from pydantic import TypeAdapter
import litellm
from litellm.litellm_core_utils.core_helpers import drops_unsupported_params
from litellm.llms.base_llm.ocr.transformation import PROVIDER_NATIVE_RESPONSE_KEY, OCRResponse
from litellm.rust_bridge import failures
from litellm.rust_bridge.failures import UpstreamFailure
from litellm.rust_bridge.public_call import optional_str
__all__ = ("UpstreamFailure", "map_failure", "response")
__all__ = ("UpstreamFailure", "drops_unsupported_params", "map_failure", "response")
_RESPONSE_ADAPTER: Final = TypeAdapter(dict[str, object])

1
litellm/types/litellm_params.py
View file

@ -191,6 +191,7 @@ class DeploymentOptions:
otpm: int | None = None
rpd: int | None = None
tpd: int | None = None
daily_reset_time: str | None = None
kubernetes_pod_discovery: bool | None = None
default_api_key_rpm_limit: int | None = None
default_api_key_tpm_limit: int | None = None

8
litellm/types/llms/azure_ai.py
View file

@ -1,6 +1,6 @@
from typing import Literal
from typing_extensions import Required, TypedDict
from typing_extensions import NotRequired, ReadOnly, Required, TypedDict
class ImageEmbeddingInput(TypedDict, total=False):
@ -15,3 +15,9 @@ class ImageEmbeddingRequest(TypedDict, total=False):
input: Required[list[ImageEmbeddingInput]]
dimensions: int
encoding_format: EncodingFormat
class FoundryAgentReference(TypedDict):
type: ReadOnly[Literal["agent_reference"]]
name: ReadOnly[str]
version: ReadOnly[NotRequired[str]]

106
litellm/types/llms/openai.py
View file

@ -1,5 +1,5 @@
import builtins
from collections.abc import Iterable, Mapping
from collections.abc import Iterable, Mapping, Sequence
from enum import Enum
from os import PathLike
from typing import IO, TYPE_CHECKING, Any, Final, Generic, Literal, Optional, TypeAlias, Union
@ -66,6 +66,7 @@ from pydantic import (
ConfigDict,
Discriminator,
Field,
FieldSerializationInfo,
NonNegativeInt,
PrivateAttr,
SerializerFunctionWrapHandler,
@ -1487,15 +1488,66 @@ One of: completed, failed, in_progress, cancelled, queued, or incomplete.
"""
_NULL_DROPPED_OUTPUT_ITEM_KEYS: Final = frozenset(
(
("reasoning", "status"),
("reasoning", "content"),
("reasoning", "encrypted_content"),
("tool_search_call", "created_by"),
("tool_search_output", "created_by"),
)
)
_NULL_DROPPED_REASONING_ITEM_KEYS: Final = frozenset(("status", "content", "encrypted_content"))
def _as_json_object(value: object) -> Mapping[str, object] | None:
return value if isinstance(value, dict) else None
def _as_json_array(value: object) -> Sequence[object] | None:
return value if isinstance(value, list) else None
def _drops_created_by(value: object, *, drops_set_created_by: bool) -> bool:
return value is None or drops_set_created_by
def _drops(item_type: object, key: object, value: object, *, drops_set_created_by: bool) -> bool:
if key == "created_by":
return _drops_created_by(value, drops_set_created_by=drops_set_created_by)
return value is None and item_type == "reasoning" and key in _NULL_DROPPED_REASONING_ITEM_KEYS
def _without_created_by(entry: object, *, drops_set_created_by: bool) -> object:
entry_object: Final = _as_json_object(entry)
if entry_object is None:
return entry
return {
k: v
for k, v in entry_object.items()
if k != "created_by" or not _drops_created_by(v, drops_set_created_by=drops_set_created_by)
}
def _replayable_output_value(item_type: object, key: object, value: object, *, drops_set_created_by: bool) -> object:
entries: Final = _as_json_array(value)
if item_type != "shell_call_output" or key != "output" or entries is None:
return value
return [_without_created_by(entry, drops_set_created_by=drops_set_created_by) for entry in entries]
def replayable_responses_item(
item: Mapping[str, object], *, drops_set_created_by: bool = False
) -> Mapping[str, object]:
item_type: Final = item.get("type")
return {
k: _replayable_output_value(item_type, k, v, drops_set_created_by=drops_set_created_by)
for k, v in item.items()
if not _drops(item_type, k, v, drops_set_created_by=drops_set_created_by)
}
def replayable_responses_input(value: object, *, drops_set_created_by: bool = False) -> object:
items: Final = _as_json_array(value)
if items is None:
return value
return [
replayable_responses_item(obj, drops_set_created_by=drops_set_created_by)
if (obj := _as_json_object(item)) is not None
else item
for item in items
]
class ResponsesAPIResponse(BaseLiteLLMOpenAIResponseObject):
@ -1570,34 +1622,16 @@ class ResponsesAPIResponse(BaseLiteLLMOpenAIResponseObject):
@field_serializer("output", mode="wrap")
@classmethod
def _serialize_output_filter_reasoning_nulls(cls, value, handler, _info):
def _serialize_output_as_replayable(
cls, value: builtins.object, handler: SerializerFunctionWrapHandler, _info: FieldSerializationInfo
):
"""
Filter null status/content/encrypted_content from reasoning output items.
Mirrors the request-side filtering in
OpenAIResponsesAPIConfig._handle_reasoning_item() which filters these
same fields before sending requests to providers.
Without this, reasoning items include null fields that cause SDK errors
(e.g., the OpenAI C# SDK crashes on status=null).
Issue: https://github.com/BerriAI/litellm/issues/16824
Drop the nulls the SDK types default in but providers never send: created_by on every output item
(OpenAI rejects it when `output` is replayed as `input`), and status/content/encrypted_content on
reasoning items (the OpenAI C# SDK crashes on status=null, issue #16824).
"""
serialized: Final = handler(value)
if not isinstance(serialized, list):
return serialized
return [
(
{
k: v
for k, v in item.items()
if v is not None or (item.get("type"), k) not in _NULL_DROPPED_OUTPUT_ITEM_KEYS
}
if isinstance(item, dict)
else item
)
for item in serialized
]
serialized: Final[object] = handler(value)
return replayable_responses_input(serialized)
@property
def output_text(self) -> str:

23
litellm/types/router.py
View file

@ -4,6 +4,7 @@ litellm.Router Types - includes RouterConfig, UpdateRouterConfig, ModelInfo etc
import datetime
import enum
import re
from collections.abc import Container, Mapping, Sequence
from dataclasses import dataclass, field
from typing import (
@ -518,6 +519,19 @@ def holds_secret_pointer(param_name: str) -> bool:
_RESERVED_INIT_KEYS: Final = frozenset({"self", "params", "__class__"})
_DAILY_RESET_TIME_PATTERN: Final = re.compile(r"\d{2}:\d{2}")
def parse_daily_reset_time(value: str | None) -> datetime.timedelta:
"""`daily_reset_time` is "HH:MM" in UTC, when the rpd/tpd window starts over; unset means midnight UTC."""
if value is None:
return datetime.timedelta(0)
if _DAILY_RESET_TIME_PATTERN.fullmatch(value) is None:
raise ValueError(f"daily_reset_time must be HH:MM in UTC, got {value!r}")
parsed: Final = datetime.time.fromisoformat(value)
return datetime.timedelta(hours=parsed.hour, minutes=parsed.minute)
class GenericLiteLLMParams(CredentialLiteLLMParams, CustomPricingLiteLLMParams):
"""
LiteLLM Params without 'model' arg (used across completion / assistants api)
@ -535,6 +549,7 @@ class GenericLiteLLMParams(CredentialLiteLLMParams, CustomPricingLiteLLMParams):
otpm: int | None = None
tpd: int | None = None
rpd: int | None = None
daily_reset_time: str | None = None
timeout: float | str | httpx.Timeout | SDKTimeout | None = None # if str, pass in as os.environ/
# read timeout of a live stream, a faked stream takes the longer of this and timeout. If str, pass in as os.environ/
stream_timeout: float | str | None = None
@ -626,6 +641,13 @@ class GenericLiteLLMParams(CredentialLiteLLMParams, CustomPricingLiteLLMParams):
valkey_text_field: str | None = None
valkey_embedding_field: str | None = None
@field_validator("daily_reset_time")
@classmethod
def _daily_reset_time_is_utc_hh_mm(cls, value: str | None) -> str | None:
if value is not None:
parse_daily_reset_time(value)
return value
@field_validator("timeout")
@classmethod
def normalize_timeout(
@ -742,6 +764,7 @@ class LiteLLMParamsTypedDict(TypedDict, total=False):
otpm: int | None
tpd: ReadOnly[int | None]
rpd: ReadOnly[int | None]
daily_reset_time: ReadOnly[str | None]
order: int | None
weight: int | None
max_parallel_requests: int | None

3
litellm/types/utils.py
View file

@ -2878,7 +2878,7 @@ class TranscriptionUsageTokensObject(LiteLLMBaseModel):
input_tokens: int
output_tokens: int
total_tokens: int
input_token_details: TranscriptionUsageInputTokenDetailsObject
input_token_details: TranscriptionUsageInputTokenDetailsObject | None = None
class TranscriptionResponse(OpenAIObject):
@ -4593,6 +4593,7 @@ class LlmProviders(str, Enum):
META = "meta"
SAIL = "sail"
REKA = "reka"
STREAMLAKE = "streamlake"
LITELLM_AGENT = "litellm_agent"
CURSOR = "cursor"
BEDROCK_MANTLE = "bedrock_mantle"

75
litellm/utils.py
View file

@ -304,7 +304,7 @@ except (ImportError, AttributeError, TypeError):
# Convert to str (if necessary)
claude_json_str = json.dumps(json_data)
import importlib.metadata
from collections.abc import AsyncIterator, Callable, Collection, Iterable, Iterator, Mapping, Sequence
from collections.abc import AsyncIterator, Callable, Collection, Iterable, Iterator, Mapping, Sequence, Sized
from typing import TYPE_CHECKING, Any, Final, Literal, Protocol, TypeAlias, TypeVar, cast, runtime_checkable
from typing_extensions import assert_never
@ -606,9 +606,23 @@ def custom_llm_setup():
litellm.provider_list.append(custom_llm["provider"])
if custom_llm["provider"] not in litellm._custom_providers:
_warn_if_built_in_provider_shadows_custom_handler(custom_llm["provider"])
litellm._custom_providers.append(custom_llm["provider"])
def _warn_if_built_in_provider_shadows_custom_handler(provider: str) -> None:
from litellm.llms.openai_like.json_loader import JSONProviderRegistry
if provider not in LlmProvidersSet and not JSONProviderRegistry.exists(provider):
return
verbose_logger.warning(
"custom_provider_map provider '%s' has the same name as a built-in LiteLLM provider, which handles "
"'%s/' models instead of your custom_handler. Rename the provider and its models to use your handler",
provider,
provider,
)
def add_custom_logger_callback_to_specific_event(callback: str, logging_event: Literal["success", "failure"]) -> None:
"""
Add a custom logger callback to the specific event
@ -4067,7 +4081,9 @@ def get_optional_params_transcription(
"timestamp_granularities": None,
}
non_default_params = {k: v for k, v in passed_params.items() if (k in default_params and v != default_params[k])}
non_default_params: Final = {
k: v for k, v in passed_params.items() if (k in default_params and v != default_params[k])
}
optional_params = {}
## raise exception if non-default value passed for non-openai/azure embedding calls
@ -4104,6 +4120,19 @@ def get_optional_params_transcription(
model=model,
drop_params=drop_params if drop_params is not None else False,
)
elif custom_llm_provider == "bedrock":
from litellm.llms.bedrock.audio_transcription import (
UnsupportedTranscriptionParams,
map_openai_transcription_params,
)
match map_openai_transcription_params(
non_default_params, drop_params=drop_params is True or litellm.drop_params is True
):
case UnsupportedTranscriptionParams(message=message):
raise UnsupportedParamsError(message=message, llm_provider="bedrock")
case mapped_params:
optional_params = dict(mapped_params)
elif provider_config is not None: # custom audio transcription config
supported_params = provider_config.get_supported_openai_params(model=model)
_check_valid_arg(supported_params=supported_params)
@ -4118,7 +4147,7 @@ def get_optional_params_transcription(
optional_params=optional_params,
passed_params=passed_params,
custom_llm_provider=custom_llm_provider,
openai_params=OPENAI_TRANSCRIPTION_PARAMS,
openai_params=list(default_params) if custom_llm_provider == "bedrock" else OPENAI_TRANSCRIPTION_PARAMS,
additional_drop_params=kwargs.get("additional_drop_params", None),
)
@ -5059,8 +5088,10 @@ def get_optional_params(
supported_params: List[str] - supported params from the litellm config
"""
verbose_logger.info("\nLiteLLM completion() model= %s; provider = %s", model, custom_llm_provider)
verbose_logger.debug("\nLiteLLM: Params passed to completion() %s", passed_params)
verbose_logger.debug("\nLiteLLM: Non-Default params passed to completion() %s", non_default_params)
if verbose_logger.isEnabledFor(logging.DEBUG):
verbose_logger.debug(
"\nLiteLLM: Non-Default params passed to completion() %s", _debug_params_repr(non_default_params)
)
unsupported_params: Final = {}
for k in non_default_params:
if k not in supported_params:
@ -5601,7 +5632,9 @@ def get_optional_params(
additional_drop_params=additional_drop_params,
)
if _print_verbose_is_active():
print_verbose(f"Final returned optional params: {redact_credentials_in_payload(optional_params)}")
print_verbose(
f"Final returned optional params: {_debug_params_repr(redact_credentials_in_payload(optional_params))}"
)
optional_params = apply_openai_param_overrides(
optional_params=optional_params,
non_default_params=non_default_params,
@ -6610,8 +6643,6 @@ def get_model_info_helper(
model=model, custom_llm_provider=custom_llm_provider or declared_authenticating_provider(model)
)
verbose_logger.debug("checking potential_model_names in litellm.model_cost: %s", potential_model_names)
combined_model_name: Final = potential_model_names["combined_model_name"]
stripped_model_name: Final = potential_model_names["stripped_model_name"]
combined_stripped_model_name: Final = potential_model_names["combined_stripped_model_name"]
@ -6714,6 +6745,7 @@ def get_model_info_helper(
key, _model_info = generalization
if _model_info is None or key is None:
verbose_logger.debug("model not found in litellm.model_cost, tried: %s", potential_model_names)
raise ModelNotMappedError(_model_not_mapped_message(model, custom_llm_provider))
_input_cost_per_token: float | None = _model_info.get("input_cost_per_token")
if _input_cost_per_token is None:
@ -8533,6 +8565,23 @@ def get_valid_models(
return [] # NON-Blocking
_MAX_INLINE_DEBUG_ARG_CHARS: Final = 500
def _debug_arg_repr(value: object) -> str:
"""Short form of a call argument, since the outgoing provider request log already prints the request body."""
rendered: Final = repr(value)
if len(rendered) <= _MAX_INLINE_DEBUG_ARG_CHARS:
return rendered
if isinstance(value, Sized):
return f"<{type(value).__name__} len={len(value)}>"
return f"<{type(value).__name__}>"
def _debug_params_repr(params: Mapping[str, object]) -> str:
return "{" + ", ".join(f"{key!r}: {_debug_arg_repr(value)}" for key, value in params.items()) + "}"
def print_args_passed_to_litellm(original_function, args, kwargs):
if not is_debugging_on():
return
@ -8553,9 +8602,9 @@ def print_args_passed_to_litellm(original_function, args, kwargs):
):
return
args_str: Final = ", ".join(map(repr, args))
args_str: Final = ", ".join(map(_debug_arg_repr, args))
redacted_kwargs: Final = redact_credentials_in_payload(kwargs)
kwargs_str: Final = ", ".join(f"{key}={value!r}" for key, value in redacted_kwargs.items())
kwargs_str: Final = ", ".join(f"{key}={_debug_arg_repr(value)}" for key, value in redacted_kwargs.items())
print_verbose(
"\n",
) # new line before
@ -9899,8 +9948,14 @@ class ProviderConfigManager:
elif litellm.LlmProviders.AZURE_AI == provider:
from litellm.llms.azure_ai.common_utils import (
azure_ai_supports_native_responses,
is_foundry_project_agent,
)
from litellm.llms.azure_ai.responses.transformation import (
AzureAIAgentResponsesAPIConfig,
)
if is_foundry_project_agent(model):
return AzureAIAgentResponsesAPIConfig()
if azure_ai_supports_native_responses(model, api_base):
return litellm.AzureAIResponsesAPIConfig()
return None

16
litellm/vector_stores/vector_store_registry.py
View file

@ -36,7 +36,7 @@ else:
_DB_ROW: Final = TypeAdapter(Mapping[str, object], config=ConfigDict(strict=True, hide_input_in_errors=True))
_ITEMS: Final = TypeAdapter(Sequence[object], config=ConfigDict(strict=True))
_TOOL: Final = TypeAdapter(Mapping[str, object], config=ConfigDict(strict=True))
_STORE_ID_ITEMS: Final = TypeAdapter(list[object], config=ConfigDict(strict=True))
_STORE_ID_ITEMS: Final = TypeAdapter(list[object] | tuple[object, ...], config=ConfigDict(strict=True))
def _named_store_ids(store_ids: object) -> tuple[str, ...]:
@ -49,7 +49,11 @@ def _named_store_ids(store_ids: object) -> tuple[str, ...]:
return tuple(store_id for store_id in items if isinstance(store_id, str))
def _tool_vector_store_ids(tool: object) -> tuple[str, ...]:
def listed_vector_store_ids(store_ids: object) -> tuple[str, ...]:
return _named_store_ids(store_ids) if isinstance(store_ids, (list, tuple)) else ()
def vector_store_ids_in_tool(tool: object) -> tuple[str, ...]:
try:
return _named_store_ids(_TOOL.validate_python(tool).get("vector_store_ids"))
except ValidationError:
@ -195,7 +199,7 @@ class VectorStoreRegistry:
tools_to_remove: Final[list[int]] = []
for i, tool in enumerate(tools):
tool_vector_store_ids = _tool_vector_store_ids(tool)
tool_vector_store_ids = vector_store_ids_in_tool(tool)
if not tool_vector_store_ids:
continue
@ -305,7 +309,7 @@ class VectorStoreRegistry:
List of vector stores with tool parameters merged into litellm_params
"""
# Pop vector_store_ids from params
vector_store_ids: Final[list[str]] = non_default_params.pop("vector_store_ids", None) or []
vector_store_ids: Final = [*listed_vector_store_ids(non_default_params.pop("vector_store_ids", None))]
# Extract params from tools and collect IDs
params_by_id = self.get_and_pop_recognised_vector_store_tools(tools=tools, vector_store_ids=vector_store_ids)
@ -353,7 +357,7 @@ class VectorStoreRegistry:
List of vector stores with tool parameters merged into litellm_params
"""
# Pop vector_store_ids from params
vector_store_ids: Final[list[str]] = non_default_params.pop("vector_store_ids", None) or []
vector_store_ids: Final = [*listed_vector_store_ids(non_default_params.pop("vector_store_ids", None))]
# Extract params from tools and collect IDs
params_by_id = self.get_and_pop_recognised_vector_store_tools(tools=tools, vector_store_ids=vector_store_ids)
@ -423,7 +427,7 @@ class VectorStoreRegistry:
validated_tools: Final = _ITEMS.validate_python(tools)
except ValidationError:
return vector_store_ids
return [*vector_store_ids, *chain.from_iterable(_tool_vector_store_ids(tool) for tool in validated_tools)]
return [*vector_store_ids, *chain.from_iterable(vector_store_ids_in_tool(tool) for tool in validated_tools)]
def load_vector_stores_from_config(self, vector_stores_config: list[dict]):
"""

481
model_prices_and_context_window.json
View file

@ -6101,6 +6101,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6133,6 +6134,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6164,6 +6166,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6230,6 +6233,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6293,6 +6297,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -6369,6 +6374,7 @@
"text",
"audio"
],
"supports_audio_output": true,
"supports_function_calling": true,
"supports_native_streaming": true,
"supports_parallel_function_calling": true,
@ -41933,6 +41939,120 @@
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-southeast-2/mistral.voxtral-mini-3b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 4.12e-08,
"input_cost_per_token_batches": 2.06e-08,
"input_cost_per_token_flex": 2.06e-08,
"input_cost_per_token_priority": 7.21e-08,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4.12e-08,
"output_cost_per_token_batches": 2.06e-08,
"output_cost_per_token_flex": 2.06e-08,
"output_cost_per_token_priority": 7.21e-08,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-southeast-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-southeast-2/mistral.voxtral-small-24b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 1.03e-07,
"input_cost_per_token_batches": 5.15e-08,
"input_cost_per_token_flex": 5.15e-08,
"input_cost_per_token_priority": 1.8025e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 3.09e-07,
"output_cost_per_token_batches": 1.545e-07,
"output_cost_per_token_flex": 1.545e-07,
"output_cost_per_token_priority": 5.4075e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-southeast-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-northeast-1/mistral.voxtral-mini-3b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 5e-08,
"input_cost_per_token_batches": 3e-08,
"input_cost_per_token_flex": 3e-08,
"input_cost_per_token_priority": 9e-08,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 5e-08,
"output_cost_per_token_batches": 3e-08,
"output_cost_per_token_flex": 3e-08,
"output_cost_per_token_priority": 9e-08,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-northeast-1/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/ap-northeast-1/mistral.voxtral-small-24b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 1.2e-07,
"input_cost_per_token_batches": 6e-08,
"input_cost_per_token_flex": 6e-08,
"input_cost_per_token_priority": 2.1e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 3.6e-07,
"output_cost_per_token_batches": 1.8e-07,
"output_cost_per_token_flex": 1.8e-07,
"output_cost_per_token_priority": 6.3e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/ap-northeast-1/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/eu-west-2/mistral.voxtral-mini-3b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 6e-08,
"input_cost_per_token_batches": 3e-08,
"input_cost_per_token_flex": 3e-08,
"input_cost_per_token_priority": 1.1e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 6e-08,
"output_cost_per_token_batches": 3e-08,
"output_cost_per_token_flex": 3e-08,
"output_cost_per_token_priority": 1.1e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/eu-west-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"bedrock/eu-west-2/mistral.voxtral-small-24b-2507": {
"deprecation_date": "2027-03-30",
"input_cost_per_token": 1.6e-07,
"input_cost_per_token_batches": 8e-08,
"input_cost_per_token_flex": 8e-08,
"input_cost_per_token_priority": 2.8e-07,
"litellm_provider": "bedrock",
"max_input_tokens": 32768,
"max_output_tokens": 8192,
"mode": "chat",
"output_cost_per_token": 4.7e-07,
"output_cost_per_token_batches": 2.3e-07,
"output_cost_per_token_flex": 2.3e-07,
"output_cost_per_token_priority": 8.2e-07,
"source": "https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonBedrock/current/eu-west-2/index.json",
"supports_audio_input": true,
"supports_system_messages": true,
"supports_native_structured_output": true
},
"mistral/codestral-2508": {
"cache_read_input_token_cost": 3e-08,
"input_cost_per_token": 3e-07,
@ -61269,6 +61389,359 @@
"supports_tool_choice": true,
"supports_vision": true
},
"streamlake/deepseek-v4-flash-0731": {
"cache_read_input_token_cost": 1.4e-08,
"input_cost_per_token": 4.4e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 1.32e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/DeepSeek-V4-Pro-0813": {
"cache_read_input_token_cost": 4.4e-08,
"input_cost_per_token": 1.32e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048566,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 3.96e-06,
"supports_function_calling": true,
"supports_prompt_caching": true,
"supports_reasoning": true,
"supports_response_schema": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/DeepSeek-V3": {
"cache_read_input_token_cost": 1.14e-07,
"input_cost_per_token": 2.86e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 56000,
"max_output_tokens": 8000,
"mode": "chat",
"output_cost_per_token": 1.143e-06,
"supports_function_calling": true,
"supports_reasoning": false,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/GLM-5.3": {
"cache_read_input_token_cost": 2.6e-07,
"input_cost_per_token": 1.4e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 4.4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_assistant_prefill": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/GLM-5.3-Flash": {
"cache_read_input_token_cost": 3e-08,
"input_cost_per_token": 1.5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 5e-07,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_response_schema": true,
"supports_vision": true,
"supports_assistant_prefill": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/Kimi-K2-Instruct-0905": {
"cache_read_input_token_cost": 1.43e-07,
"input_cost_per_token": 5.71e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 224000,
"max_output_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 2.286e-06,
"supports_function_calling": true,
"supports_reasoning": false,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-r1": {
"cache_read_input_token_cost": 2.29e-07,
"input_cost_per_token": 5.71e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 98304,
"max_output_tokens": 16000,
"mode": "chat",
"output_cost_per_token": 2.286e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v3.1": {
"cache_read_input_token_cost": 2.29e-07,
"input_cost_per_token": 5.71e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 98304,
"max_output_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 1.714e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4-flash": {
"cache_read_input_token_cost": 2.8e-08,
"input_cost_per_token": 1.4e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 2.8e-07,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4-pro": {
"cache_read_input_token_cost": 1.45e-07,
"input_cost_per_token": 1.74e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 3.48e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4.1-flash-1017": {
"cache_read_input_token_cost": 6e-09,
"input_cost_per_token": 3e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048576,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/deepseek-v4.1-flash-260910": {
"cache_read_input_token_cost": 6e-09,
"input_cost_per_token": 3e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 1048568,
"max_output_tokens": 384000,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/glm-5": {
"cache_read_input_token_cost": 2e-07,
"input_cost_per_token": 1e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 202752,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/glm-5.1": {
"cache_read_input_token_cost": 2.6e-07,
"input_cost_per_token": 1.4e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 202752,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 4.4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/glm-5.2": {
"cache_read_input_token_cost": 2.6e-07,
"input_cost_per_token": 1.4e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1048566,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 4.4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/kimi-k2.6": {
"cache_read_input_token_cost": 1.6e-07,
"input_cost_per_token": 9.5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 262144,
"max_output_tokens": 256000,
"mode": "chat",
"output_cost_per_token": 4e-06,
"supports_function_calling": true,
"supports_vision": true,
"supports_reasoning": false,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/kimi-k2.7-code": {
"cache_read_input_token_cost": 1.9e-07,
"input_cost_per_token": 9.5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 262144,
"max_output_tokens": 32000,
"mode": "chat",
"output_cost_per_token": 4e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/mimo-v2.6-flash": {
"cache_read_input_token_cost": 2.8e-09,
"input_cost_per_token": 1.4e-07,
"litellm_provider": "streamlake",
"mode": "chat",
"output_cost_per_token": 2.8e-07,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/mimo-v2.6-pro": {
"cache_read_input_token_cost": 3.6e-09,
"input_cost_per_token": 4.35e-07,
"litellm_provider": "streamlake",
"mode": "chat",
"output_cost_per_token": 8.7e-07,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/minimax-m2.5": {
"cache_creation_input_token_cost": 3.75e-07,
"cache_read_input_token_cost": 3e-08,
"input_cost_per_token": 3e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 196608,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 1.2e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/minimax-m3": {
"cache_read_input_token_cost": 1.2e-07,
"cache_read_input_token_cost_above_512k_tokens": 2.4e-07,
"input_cost_per_token": 6e-07,
"input_cost_per_token_above_512k_tokens": 1.2e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 1000000,
"max_output_tokens": 512000,
"mode": "chat",
"output_cost_per_token": 2.4e-06,
"output_cost_per_token_above_512k_tokens": 4.8e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.5-397b-a17b": {
"cache_read_input_token_cost": 1.2e-07,
"input_cost_per_token": 6e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 258048,
"max_output_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 3.6e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.6-27b": {
"input_cost_per_token": 6e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 229376,
"max_output_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 3.6e-06,
"supports_function_calling": true,
"supports_vision": true,
"supports_reasoning": false,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.6-35b-a3b": {
"input_cost_per_token": 3.75e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 229376,
"max_output_tokens": 64000,
"mode": "chat",
"output_cost_per_token": 2.25e-06,
"supports_function_calling": true,
"supports_vision": true,
"supports_reasoning": false,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.8-2.4t-a95b": {
"cache_read_input_token_cost": 2.5e-07,
"input_cost_per_token": 2e-06,
"litellm_provider": "streamlake",
"max_input_tokens": 983616,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 6e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"cache_creation_input_token_cost": 2.5e-06,
"explicit_cache_read_input_token_cost": 1.7e-07,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"streamlake/qwen3.8-27b": {
"cache_read_input_token_cost": 1e-07,
"input_cost_per_token": 5e-07,
"litellm_provider": "streamlake",
"max_input_tokens": 983616,
"max_output_tokens": 128000,
"mode": "chat",
"output_cost_per_token": 3e-06,
"supports_function_calling": true,
"supports_reasoning": true,
"supports_vision": true,
"cache_creation_input_token_cost": 6.25e-07,
"explicit_cache_read_input_token_cost": 5e-08,
"supports_prompt_caching": true,
"source": "https://www.streamlake.ai/document/DOC/mgrnm4xm362hvp5wyce"
},
"deepseek-v4-flash": {
"cache_creation_input_token_cost": 0.0,
"cache_read_input_token_cost": 6e-09,
@ -62584,6 +63057,14 @@
"litellm_provider": "anthropic"
}
},
{
"name": "typesafe-jev-ids",
"pattern": "^jev-(?:latest|preview|1\\.13\\.0|1\\.14\\.0)$",
"description": "A bare Jev id that has a typesafe/ price entry (jev-latest, jev-preview, jev-1.13.0, jev-1.14.0), the name the /typesafe pass-through forwards to TypeSafe. Listed by name, matched case-insensitively like every rule here, so foo/jev-latest or a model group named like a Jev id TypeSafe does not serve, such as jev-1.0 or jev-internal-router, never resolves to typesafe. A new Jev version goes here together with its price entry. Provider inference feeds the proxy's provider/* wildcard access checks, so this rule is what lets typesafe/* on a key, team or project allowlist cover a jev-latest request.",
"model_info": {
"litellm_provider": "typesafe"
}
},
{
"name": "claude-family-baseline",
"pattern": "claude-[a-z]+-\\d+(?:[-.]\\d+)?(?:-\\d{8})?",

103
osv-scanner.toml
View file

@ -1,27 +1,102 @@
[[IgnoredVulns]]
id = "GHSA-w8v5-vhqr-4h9v"
ignoreUntil = 2026-11-01
reason = "diskcache has no fixed release published; remove this entry once one exists"
[[IgnoredVulns]]
id = "GHSA-v4x8-3wr5-3wq8"
ignoreUntil = 2026-11-20
reason = "the only fixed filelock is 4.2.1, a new major that requires Python >= 3.11 while litellm still supports 3.10, and the advisory is in ReadWriteLock, which litellm never uses (only FileLock); bump filelock and remove this entry once 3.10 is dropped or a 3.x fix ships"
reason = "filelock ReadWriteLock advisory, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-vfj7-8cjw-p6xm"
ignoreUntil = 2026-11-03
reason = "braces has no fixed release published (3.0.3, 2024-05-21, is latest and last_affected); dev-only via knip > fast-glob > micromatch; remove this entry once one exists"
id = "GHSA-567q-5qvw-pf92"
ignoreUntil = 2026-11-20
reason = "Windows-only local DoS (CVSS 2.5) in filelock soft locks; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-6q48-6c7w-9gw2"
ignoreUntil = 2026-10-23
reason = "certifi 2026.10.19 (published 2026-10-20) is inside the 3-day uv exclude-newer window until 2026-10-23 02:24 UTC; bump certifi and remove this entry once it clears"
id = "GHSA-9vhv-jxww-xfm5"
ignoreUntil = 2026-11-20
reason = "Windows-only local DoS (CVSS 3.3) in filelock ReadWriteLock, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-6627-m3xw-93mp"
ignoreUntil = 2026-11-20
reason = "filelock StrictSoftFileLock claim directories ignore the lock mode, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-69hj-f68j-2879"
ignoreUntil = 2026-11-20
reason = "filelock UnixFileLock truncates and chmods a file another user renames or hard-links in at the lock path; litellm's CLI takes FileLock only inside ~/.litellm, which it creates owner-only, and huggingface_hub takes it in the Hugging Face cache, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting; the attack needs another local user who can rename or link files in that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-8mx5-7m5m-r77g"
ignoreUntil = 2026-11-20
reason = "filelock WindowsFileLock ignores mode; the proxy image is Linux, so only Windows installs run it; there litellm's CLI takes FileLock only inside the owner-only ~/.litellm, whose lock file holds no data, and huggingface_hub takes it in the Hugging Face cache, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-9c4x-gm8v-wgc4"
ignoreUntil = 2026-11-20
reason = "filelock StrictSoftFileLock and SoftReadWriteLock adopt another user's protocol directory, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-9j6q-wvj5-jgjg"
ignoreUntil = 2026-11-20
reason = "filelock SoftFileLock symlink swap leaks whether a path exists; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-cwqr-hh33-gh44"
ignoreUntil = 2026-11-20
reason = "filelock ReadWriteLock stalls on a crafted lock database, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-fq56-3cx3-fjxp"
ignoreUntil = 2026-11-20
reason = "filelock SoftFileLock stale-lock break removes a live lock on a reused inode; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-j8f7-rjxc-mr56"
ignoreUntil = 2026-11-20
reason = "filelock ReadWriteLock follows a symlink into SQLite, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed in 4.0.9, which litellm's own filelock<4.0 cap blocks on every Python until it is lifted (#48635, #48802); remove once that lands"
[[IgnoredVulns]]
id = "GHSA-m462-7rxw-wvpg"
ignoreUntil = 2026-11-20
reason = "filelock StrictSoftFileLock follows a symlink swapped in at <path>.filelock, which no package in the lock takes (litellm and huggingface_hub, filelock's only dependents, use just FileLock and SoftFileLock); fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-rfpc-qcw5-3pc4"
ignoreUntil = 2026-11-20
reason = "filelock SoftFileLock removes a live holder's marker when another user forges its PID; litellm never takes SoftFileLock, but huggingface_hub falls back to it in the Hugging Face cache when the filesystem has no flock, which litellm reaches only to download the Kimi K3 tokenizer for Bedrock token counting, and the attack needs another local user who can write that cache; fixed only in 4.2.1, and filelock 4.1.1+ needs Python >= 3.11; litellm's own filelock<4.0 cap keeps every Python on 3.32.7 until it is lifted for 3.11+ (#48635, #48802); remove once that lands and 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-pcg2-g6cj-q92c"
ignoreUntil = 2026-11-27
reason = "tornado 6.6 (published 2026-10-21) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 6.5.10 while 3.11+ locks 6.6; litellm starts no tornado server or client, since semantic-router and aurelio-sdk only declare it and tenacity and sentry-sdk import it only to wrap tornado code when it is present, so tornado's HTTP header parsing is never reached; remove this entry once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-pw9p-v7r6-gx4p"
ignoreUntil = 2026-11-27
reason = "tornado 6.6 (published 2026-10-21) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 6.5.10 while 3.11+ locks 6.6; litellm starts no tornado server or client, since semantic-router and aurelio-sdk only declare it and tenacity and sentry-sdk import it only to wrap tornado code when it is present, so tornado's RedirectHandler is never reached; remove this entry once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-r4j5-9xpp-3985"
ignoreUntil = 2026-11-27
reason = "tornado 6.6 (published 2026-10-21) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 6.5.10 while 3.11+ locks 6.6; litellm starts no tornado server or client, since semantic-router and aurelio-sdk only declare it and tenacity and sentry-sdk import it only to wrap tornado code when it is present, so tornado's chunked body reading is never reached; remove this entry once 3.10 is dropped"
[[IgnoredVulns]]
id = "GHSA-w8v5-vhqr-4h9v"
ignoreUntil = 2026-11-27
reason = "diskcache still has no fixed release published (checked 2026-10-28), and it is reached only through the optional caching extra; remove this entry once one exists"
[[IgnoredVulns]]
id = "GHSA-45c3-73f7-pv4m"
ignoreUntil = 2026-11-27
reason = "dnspython 2.9.0 (published 2026-10-09) is the only fix and requires Python >= 3.11, so the Python 3.10 resolution stays on 2.8.0 while 3.11+ locks 2.9.0; the advisory is CPU use in dns.name.from_wire_parser and litellm only reaches dnspython through pydantic EmailStr, which never does DNS lookups; remove this entry once 3.10 is dropped or a 3.10-compatible fix ships"
[[IgnoredVulns]]
id = "GHSA-xcx6-jx8p-x3h5"
ignoreUntil = 2026-11-24
reason = "@azure/msal-node is fixed only in 7.1.0 (published 2026-10-13), but it is dev-only in vscode-extension via @vscode/vsce > @azure/identity, and the latest @azure/identity (4.14.0) still requires msal-node ^6.0.0, so the lock cannot move without forcing a major override; the advisory is in the auth_code flow, which extension packaging does not run; bump and remove this entry once @azure/identity accepts msal-node >= 7.1.0"
[[PackageOverrides]]
name = "pillow"
version = "12.3.0"
ecosystem = "PyPI"
vulnerability.ignore = true
effectiveUntil = 2026-10-27
reason = "pillow 13.0.0 requires Python >= 3.11, so the Python 3.10 resolution of the ci and dev groups stays on 12.3.0; 3.11+ is on 13.0.0"
effectiveUntil = 2026-11-27
reason = "pillow 13.0.0 requires Python >= 3.11, so the Python 3.10 resolution of the ci and dev groups and of the mlflow extra stays on 12.3.0; 3.11+ is on 13.0.0; pillow has never shipped an older-major release after the next major (no 10.x after 11.0.0, no 11.x after 12.0.0), so 13.x is the only fix line; extended to 2026-11-27 with maintainer approval, same date as the dnspython 3.10 entry; remove once 3.10 is dropped"

26
provider_endpoints_support.json
View file

@ -3785,6 +3785,32 @@
"generateContent": true
}
},
"streamlake": {
"display_name": "StreamLake (`streamlake`)",
"url": "https://docs.litellm.ai/docs/providers/openai_compatible",
"endpoints": {
"chat_completions": true,
"responses": true,
"text_completion": true,
"embeddings": false,
"image_generations": false,
"image_edits": false,
"audio_transcriptions": false,
"audio_speech": false,
"rerank": false,
"ocr": false,
"search": false,
"moderations": false,
"realtime": false,
"video_generations": false,
"batches": false,
"messages": true,
"interactions": true,
"systemone": false,
"a2a": true,
"generateContent": true
}
},
"tencent": {
"display_name": "Tencent TokenHub (`tencent`)",
"url": "https://docs.litellm.ai/docs/providers/tencent",

2
pyproject.toml
View file

@ -360,11 +360,13 @@ cache-keys = [
]
constraint-dependencies = [
"tornado>=6.5.8",
"tornado>=6.6; python_version >= '3.11'",
"aiohttp>=3.14.2,<4.0",
"packaging>=24.0",
"soupsieve>=2.8.4",
"httplib2>=0.32.0",
"setuptools>=84.1.0",
"dnspython>=2.9.0; python_version >= '3.11'",
]
override-dependencies = [
# a2a-sdk 1.x requires packaging>=24.0; lunary 1.4.x still caps at <24.0.

26
tests/benchmarks/test_json_fragment_accumulator_benchmarks.py Normal file
View file

@ -0,0 +1,26 @@
from collections.abc import Iterator
from typing import Final
import pytest
from litellm.litellm_core_utils.json_fragment_accumulator import JSONFragmentAccumulator
CONCATENATED_VALUES: Final = 20_000
def _popped_values(accumulator: JSONFragmentAccumulator) -> Iterator[object]:
while True:
found, value = accumulator.pop_next_value()
if not found:
return
yield value
@pytest.mark.benchmark
def test_draining_many_concatenated_values() -> None:
accumulator: Final = JSONFragmentAccumulator()
accumulator.append('{"a": 1}' * CONCATENATED_VALUES)
drained: Final = sum(1 for _ in _popped_values(accumulator))
assert drained == CONCATENATED_VALUES

2
tests/integration/authorization/test_team_admin_gate.py
View file

@ -746,7 +746,7 @@ def test_internal_user_model_creation_prohibition_refuses_team_admin_but_not_pro
"error": {
"message": "Model creation is disabled for internal users by disable_model_add_for_internal_users.",
"type": "auth_error",
"param": "None",
"param": None,
"code": "403",
}
},

4
tests/integration/authorization/test_user_self_update_guard.py
View file

@ -38,7 +38,7 @@ def _escalation_error(field: str) -> dict[str, JsonValue]:
"error": {
"message": f"{{'error': \"Non-admin users cannot modify '{field}' on their own record. Contact your proxy admin.\"}}",
"type": "auth_error",
"param": "None",
"param": None,
"code": "403",
}
}
@ -49,7 +49,7 @@ def _role_change_error() -> dict[str, JsonValue]:
"error": {
"message": "Only proxy admins can modify user roles.",
"type": "auth_error",
"param": "None",
"param": None,
"code": "403",
}
}

4
tests/integration/management/test_team_delete_large_membership.py
View file

@ -255,8 +255,8 @@ def _team_not_found_body(team_id: str) -> dict[str, JsonValue]:
return {
"error": {
"message": f"{{'message': 'Team not found, passed team id: {team_id}.'}}",
"type": "auth_error",
"param": "None",
"type": "invalid_request_error",
"param": None,
"code": "404",
}
}

41
tests/integration/providers/test_responses_compact_wire.py
View file

@ -3,6 +3,7 @@ from __future__ import annotations
import json
import os
import uuid
from types import MappingProxyType
from typing import Final
import openai
@ -65,6 +66,11 @@ def test_responses_compact_routes_forward_complete_request_and_typed_items(gatew
"call_id": "call_previous",
"output": "The record is ready.",
}
saved_compaction_item: Final = {
"id": f"cmp_{uuid.uuid4().hex}",
"type": "compaction",
"encrypted_content": "synthetic-saved-state",
}
requests: Final = (
{
"input": "Compact this conversation.",
@ -75,6 +81,20 @@ def test_responses_compact_routes_forward_complete_request_and_typed_items(gatew
"tools": [_FUNCTION_TOOL],
},
)
saved_compaction_request: Final = {
"input": [{**saved_compaction_item, "created_by": None}, {**function_output_item, "created_by": None}],
"instructions": "Compact the saved conversation again.",
}
forwarded_bodies: Final = MappingProxyType(
{
None: requests[0],
requests[1]["instructions"]: requests[1],
saved_compaction_request["instructions"]: {
**saved_compaction_request,
"input": [saved_compaction_item, function_output_item],
},
}
)
output_items: Final = [
{
"id": f"rs_compact_{uuid.uuid4().hex}",
@ -98,18 +118,7 @@ def test_responses_compact_routes_forward_complete_request_and_typed_items(gatew
assert request.method == "POST"
assert request.headers["authorization"] == f"Bearer {_API_KEY}", dict(request.headers)
body: Final = _JSON_OBJECT.validate_json(request.body)
compact_params: Final = (
{
"model": _MODEL,
**requests[0],
}
if body.get("input") == requests[0]["input"]
else {
"model": _MODEL,
**requests[1],
}
)
assert body == compact_params, request.body
assert body == {"model": _MODEL, **forwarded_bodies[body.get("instructions")]}, request.body
return Reply(
body=json.dumps(
{
@ -128,16 +137,20 @@ def test_responses_compact_routes_forward_complete_request_and_typed_items(gatew
).encode()
)
sent_bodies: Final = (*requests, saved_compaction_request)
with wire_server(respond) as wire, gateway.scenario() as scenario:
model: Final = _deployment(scenario, wire)
for alias in _ALIASES:
for request_body in requests:
for request_body in sent_bodies:
response: Final = gateway.request(
"POST",
alias,
{"model": model, **request_body},
)
assert response.status_code == 200, response.text
wire_output: Final = _JSON_OBJECT.validate_json(response.content)["output"]
assert isinstance(wire_output, list), response.text
assert [item for item in wire_output if not isinstance(item, dict) or "created_by" in item] == []
compacted: Final = CompactedResponse.model_validate_json(response.content)
assert compacted.id != completed_id, compacted.model_dump_json()
assert same_response(compacted.id, completed_id), compacted.model_dump_json()
@ -201,7 +214,7 @@ def test_responses_compact_routes_forward_complete_request_and_typed_items(gatew
"user": None,
"store": None,
}, response.text
assert [request.target for request in drain_contract_requests(wire)] == ["/v1/responses/compact"] * 6
assert [request.target for request in drain_contract_requests(wire)] == ["/v1/responses/compact"] * (len(_ALIASES) * len(sent_bodies))
@pytest.mark.parametrize("prefix", ("/v1", "", "/openai/v1"))

10
tests/integration/sdk/test_router_cooldown_fallback_sdk.py
View file

@ -122,12 +122,12 @@ async def test_cooldown_skips_only_the_failed_duplicate_model_deployment() -> No
@pytest.mark.parametrize(
"router_kwargs",
[{"fallbacks": [{"primary": "backup"}]}, {"default_fallbacks": "backup"}],
ids=["string_fallback_target", "string_default_fallbacks"],
"fallbacks",
[[{"primary": "backup"}], [{"*": "backup"}]],
ids=["string_fallback_target", "string_catch_all_target"],
)
@pytest.mark.asyncio
async def test_string_valued_fallbacks_still_cool_down_the_failing_primary(router_kwargs: dict[str, object]) -> None:
async def test_string_valued_fallbacks_still_cool_down_the_failing_primary(fallbacks: list[dict[str, object]]) -> None:
with wire_server(_peer) as wire:
router: Final = Router(
model_list=[
@ -136,8 +136,8 @@ async def test_string_valued_fallbacks_still_cool_down_the_failing_primary(route
],
allowed_fails=0,
num_retries=0,
**router_kwargs,
)
router.fallbacks = fallbacks
with pytest.raises(litellm.AuthenticationError):
await router.acompletion(
model="primary",

5
tests/test_litellm_rust/ocr/test_failures.py
View file

@ -108,7 +108,10 @@ async def test_ocr_contract_malformed_document_is_actionable(
@pytest.mark.asyncio
@pytest.mark.parametrize("asynchronous", [False, True], ids=["sync", "async"])
@pytest.mark.parametrize("option,value,field", [("pages", [-1], "pages"), ("features", [1], "features")])
@pytest.mark.parametrize(
"option,value,field",
[("pages", [-1], "pages"), ("features", [1], "features"), ("outputContentFormat", "html", "drop_params")],
)
async def test_ocr_contract_azure_invalid_options_are_bad_requests(
server: RecordingServer,
asynchronous: bool,

30
tests/test_litellm_rust/ocr/test_requests.py
View file

@ -3,6 +3,7 @@ from typing import Final
import pytest
import litellm
from tests.test_litellm_rust.ocr.data import (
OCR_DOCUMENT,
OCR_RESPONSE,
@ -102,3 +103,32 @@ def test_native_ocr_encodes_python_file_input_and_drops_unknown_arguments(server
"model": "mistral-ocr-latest",
"document": {"type": "image_url", "image_url": "data:image/png;base64,YWJj"},
}
@pytest.mark.parametrize(
"drop_params,global_drop_params",
[(True, False), (" TRUE ", False), (None, True), (False, True)],
ids=["per_call", "per_call_spelled", "global", "global_wins"],
)
def test_azure_document_intelligence_drop_params_drops_an_unsupported_output_content_format(
server: RecordingServer,
monkeypatch: pytest.MonkeyPatch,
drop_params: object,
global_drop_params: bool,
) -> None:
monkeypatch.setattr(litellm, "drop_params", global_drop_params)
server.default_response = ResponseSpec(
body={"status": "succeeded", "analyzeResult": {"pages": [{"pageNumber": 1, "lines": [{"content": "hello"}]}]}}
)
response: Final = call_ocr(
server,
model="azure_ai/doc-intelligence/prebuilt-layout",
outputContentFormat="html",
drop_params=drop_params,
)
assert_served_natively(server, response)
assert response.pages[0].markdown == "hello"
assert "outputContentFormat" not in server.requests[0].path
assert "drop_params" not in server.requests[0].path

4
tests/unit/AGENTS.md
View file

@ -8,6 +8,10 @@ the same tests against Google's live spec
Unit tests must not depend on the current time or execution speed. Use fixed timestamps for fixtures
and controlled clocks for time-dependent behavior. Incidental clock reads inside production code are allowed
A unix socket goes under the `socket_dir` or `socket_path` fixture, never `tmp_path`, whose paths run past macOS's
104-byte AF_UNIX limit and fail only on a Mac. Code that opens a socket in its own `tempfile` directory runs under
`socket_tempdir`
## What good looks like
```python

34
tests/unit/conftest.py
View file

@ -2,6 +2,8 @@ import asyncio
import base64
import importlib
import os
import shutil
import tempfile
from collections.abc import Callable, Coroutine, Iterator
from dataclasses import dataclass, field
from pathlib import Path
@ -217,6 +219,38 @@ def _flush_client_caches() -> None:
_reset_aws_auth_caches()
_MACOS_LONGEST_SOCKET_PATH: Final = 103
_LONGEST_SOCKET_SUFFIX: Final = len("/sock12345678/d/litellm-pgbouncer-12345678/.s.PGSQL.65535")
def _socket_root() -> str:
default: Final = tempfile.gettempdir()
fits: Final = len(os.fsencode(default)) + _LONGEST_SOCKET_SUFFIX <= _MACOS_LONGEST_SOCKET_PATH
return default if fits else "/tmp"
@pytest.fixture
def socket_dir(tmp_path: Path) -> Iterator[Path]:
"""``tmp_path`` behind a short link, so unix socket paths fit the sun_path limit that it exceeds on macOS."""
link_dir: Final = Path(tempfile.mkdtemp(prefix="sock", dir=_socket_root()))
link: Final = link_dir / "d"
link.symlink_to(tmp_path, target_is_directory=True)
yield link
shutil.rmtree(link_dir, ignore_errors=True)
@pytest.fixture
def socket_path(socket_dir: Path) -> Path:
return socket_dir / "spend.sock"
@pytest.fixture
def socket_tempdir(socket_dir: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
"""Point ``tempfile``'s default directory at ``socket_dir``, for code that picks its own socket directory."""
monkeypatch.setattr(tempfile, "tempdir", str(socket_dir))
return socket_dir
@pytest.fixture(autouse=True, scope="session")
def bundled_tiktoken_cache() -> None:
importlib.import_module("litellm.litellm_core_utils.default_encoding")

Some files were not shown because too many files have changed in this diff Show more