* fix(guardrails): skip streaming guardrail rounds that re-scan cleared output
Streaming guardrails scanned the finished answer twice at end of stream
whenever the chunk count landed on a multiple of the sampling rate, ran
sampled rounds whose payload was identical to the previous one, and on
/v1/messages could scan an empty text before the first content chunk.
Every redundant round is a paid guardrail provider call.
Each endpoint handler now exposes a scan key describing what a round
would hand to apply_guardrail (the text so far, plus tool calls once the
stream has ended), and the unified streaming hook skips a sampled or
end-of-stream round whose key equals the last scanned one or carries
nothing to scan yet. Rounds that carry tool calls are never skipped.
* test(guardrails): expect one end-of-stream scan when the terminal chunk is sampled
Update sampled cadence expectations and use tuple-backed scan state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sort the chat-tool keys once and number duplicates with groupby instead of
rescanning every preceding key per position, so the guardrail merge stays
O(n log n) on client-supplied tool lists. Drop the comment that restated the
unsupported-tool warning in the Responses-to-chat transformation.
Any pre_call guardrail on /v1/responses flattened Codex namespace tools
into ns__member functions and wrote the flattened list back to the
request, so the model called mcp__server__tool with no namespace and
Codex rejected the call as unsupported.
The handler now keeps the client's original tools, hands the guardrail a
deep copy of the flattened ones, and rebuilds data["tools"] by matching
the guardrail's output to the originals by type and name. Unchanged
tools go back as the original objects, a dropped or edited namespace
member changes only that member, and tools the guardrail injects are
still appended.
Fixes#39183
The chat-completions bridge emitted Responses output items whose item ids
were raw Anthropic tool ids (toolu_/srvtoolu_), which OpenAI rejects on
replay with "Expected an ID that begins with 'fc'", breaking router
fallback conversations from gpt-5 to claude models.
Four fixes, composable and independently useful:
- emission: bridge output items get fc_/ctc_-prefixed item ids while
call_id stays raw so tool_result pairing keeps working (streaming and
non-streaming share the same helpers)
- openai replay: request transformation drops tool call item ids that do
not match OpenAI's own shapes instead of forwarding them, gated to
OpenAI and Azure, since the API accepts the items with no id at all
- anthropic replay: a replayed srvtoolu_ call whose paired server tool
result is unavailable degrades to a plain client tool_use instead of a
dangling server_tool_use that 400s the client's tool_result
- tool-only turns no longer emit a message output item with output_text
text null, matching native OpenAI output
A guardrail modify_response verdict on a streaming request only produced a
proper replacement on /v1/messages: the chat completions and Responses API
translations had no build_block_sse_chunks, so the ModifyResponseException
re-raised and surfaced as an in-stream 500 error frame (or a whole-request
500 in buffered mode) instead of the documented 200 replacement.
Implement build_block_sse_chunks for both OpenAI translations: chat emits a
content delta plus a finish_reason content_filter chunk with real usage;
Responses emits the typed event sequence (standalone via
build_synthetic_response_events pre-stream, or an output-item continuation
under the in-progress response id mid-stream) ending in response.completed.
The Responses-to-chat transform dropped the filename OpenAI requires next to
file_data, so a request carrying an inline PDF counted 13 tokens instead of 36
and a real completion through the chat bridge got a 400.
Assistant list content was forwarded to /v1/responses/input_tokens as chat
`text` blocks, which the Responses API rejects (it accepts only output_text
and refusal inside an assistant turn). The 400 sent the whole request to the
local tokenizer, so any conversation with an assistant turn silently lost
provider-exact counting, including the image counting added in 73ab647b1c.
Assistant content now collapses to the plain string the Responses API counts
identically, and image parts are kept to user turns where they are legal.
The chat-to-Responses reverse transform kept only text blocks, so an image
input was dropped before the count went to OpenAI. A 256x256 image request
counted 13 tokens instead of 268.
_input_item_provenance converted every input prefix, so an n-item request paid
for n+1 full conversions. It now converts each item once, glues consecutive
function_call items (plus their trailing-assistant context) into units so the
transform's tool_call merging is reproduced inside the unit conversion, and
verifies the unit concatenation against one full conversion, bailing to the
full-conversion fallback on any mismatch. Messages from multi-item units are
tainted, which keeps parallel tool calls patchable exactly like the old prefix
pass while unpredicted merges fall back safely.
A guardrail handing back a non-list structured_messages payload (the
HiddenLayer v2 evaluation dict) previously fell through the length-mismatch
fallback and 500ed converting the dict's keys as messages. The write-back is
now skipped for non-list payloads, restoring the previous no-write-back
behavior on the Responses surface.
Also refreshes the compresr texts-mirror docstring, which still claimed the
Responses translation cannot round-trip structured_messages.
- _write_back_structured_messages now patches only the rewritten rows back
into the original input items, so reasoning items (encrypted_content),
function_call ids, and web_search_call items survive a guardrail rewrite
verbatim; rewrites that cannot be row-mapped fall back to the previous
full conversion
- the responses bridge stream snapshot restores usage hidden in
_hidden_params when stream_options is unset, so converted fake streams
report real input_tokens instead of 0
GPT-5 and later accept a top-level anyOf natively and call tools better with it intact, so the flattening now runs only for the gpt-4, gpt-3.5, chatgpt-4o, o1, o3, and o4 families. Non-dict tool entries pass through untouched, a typeless root that carries properties counts as an object, and the bounded $ref walker is listed in the recursion detector allowlist.
Message-rewriting guardrails such as Headroom return their rewrite in
structured_messages and leave texts untouched. The responses guardrail
translation only mapped texts back, so compression never reached the
upstream request on /v1/responses while the retrieve tool still got
injected. Convert the returned messages back to Responses input (plus
instructions) the way the chat and Anthropic handlers already do, and
keep developer messages as input_text in the chat-to-responses bridge.
Resolves LIT-6494
OpenAI's function-calling validator rejects tool parameters carrying
oneOf/anyOf/allOf/enum/const/not at the top level, while the ChatGPT
backend Codex talks to natively accepts them, so an MCP tool declaring a
top-level union 400s through the proxy. Merge the branches into the
object schema for OpenAI itself only, walking the namespace-nested tools
current Codex builds send, on both /v1/responses and /v1/responses/compact
A gpt-5 model accepts a non-default temperature only while its effective reasoning
effort resolves to "none". litellm had no representation of the effort a model applies
when the request omits reasoning_effort, so it substituted supports_none_reasoning_effort,
which is a different fact. Every model that supports "none" without defaulting to it
therefore had temperature forwarded and rejected upstream, and because the carve-out
returned before the drop_params branch, drop_params: true could not save it.
Declare the fact instead. A new cost-map key, default_reasoning_effort, states the effort
the provider applies when the request omits one, and one shared predicate resolves the
effective effort from it: an explicit reasoning_effort wins, otherwise the declared
default, otherwise the catalogue decides.
That last step matters because the cost map is fetched from the published branch at import
time, so it can be OLDER than the code reading it. On such a map every model looks
undeclared, and reading that as "reasoning is active" would strip temperature from the 39
gpt-5.1/5.2/5.4 entries that accept it, a regression caused by data lag rather than by
anything about the model. So an absent declaration is only meaningful once the catalogue
carries the key at all; a map that predates the feature keeps the answer litellm gave
before it existed, and the conservative answer applies from the moment the data lands.
The top_p/logprobs/top_logprobs gate carried the same assumption spelled differently and
now shares the predicate, as does the Responses API, which reimplemented the rule and is
what the default /v1/messages bridge routes openai models through. Azure normalises its
routing names in one resolver that every capability lookup goes through, which replaces
its bespoke per-lookup rewrite.
Declared on the 37 gpt-5.1/5.2/5.4 entries measured to accept temperature=0 today, so
their behaviour is unchanged. The 23 gpt-5.5/5.6 entries that reject it stay undeclared
and are fixed once the catalogue carries the key.
Resolves LIT-3797
Resolves LIT-5028
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
A test that asserts on the error inside its own except block passes when the
call stops raising, because nothing runs the handler. That is the exact case
the test exists to catch, so the regression lands green.
Rewrites all 111 such blocks into pytest.raises, which fails when the call
succeeds, and selects PT017 in ruff-tests.toml so no new one lands.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
When the resolved deployment is provider openai and the model is GPT-5.6 or
newer, the cache control hook now writes prompt_cache_breakpoint on the
targeted content block and sets prompt_cache_options to explicit mode unless
the caller already passed one. The /v1/messages bridges carry the marker
through (the Responses bridge moves a marked system prompt into a developer
message, since top-level instructions cannot hold one). Breakpoint counting
and the stand-down check recognise both marker kinds, and client breakpoints
already present in messages are no longer subtracted from the cap twice.
Fixes#37509
* feat(guardrails): add Compresr guardrail for query-aware context compression
Adds a first-class guardrail that compresses bulky message content (tool
outputs, RAG chunks, search results) through the Compresr API before the
request reaches the LLM, via the apply_guardrail / structured_messages hook
so it covers /chat/completions, /v1/messages, and /v1/responses (the latter
through the texts channel, mirrored only when the replacement is
unambiguous; anything ambiguous is left uncompressed).
Distinct from whole-conversation compressors:
- Query-aware: each message is compressed against the intent that produced
it (a tool output against its originating tool call's name + arguments,
resolved via tool_call_id; otherwise the last user message).
- Recoverable: each compressed message carries a hash marker and the request
gains a compresr_retrieve tool, so the model can pull the original content
back through the agentic loop when the compressed version is not enough.
Originals are cached in-process, scoped to the caller's virtual-key hash
plus the request's litellm_call_id, with a TTL and a per-call byte cap;
recovery is skipped when no caller scope is available so one caller can
never read another's originals. The store is per-process, so multi-worker
deployments need sticky routing (or enable_retrieval=false).
Fail-closed by default (fail_open configurable), SSRF-validated api_base
(alternate IP-literal encodings included), cross-tenant-isolated recovery
store, and upstream errors redacted from client-facing responses. The
outbound client follows redirects and re-resolves DNS per request, so the
api_base host/IP checks are defense-in-depth, not a full SSRF guarantee;
this is documented as a known limitation. Requests where nothing was
actually compressed are returned untouched (same object identity) so
handlers skip the write-back. Auto-discovered via the guardrail_hooks
registry.
* fix(guardrails): cap Compresr recovery store total memory
The recovery store bounded bytes per call and entry count, but had no
aggregate cap: 256 tracked call ids at the 10 MiB per-call default could
retain ~2.5 GiB per worker. A flood of requests with distinct
x-litellm-call-id values and large compressible tool outputs could
exhaust a shared proxy worker.
Add a global byte budget (_MAX_TOTAL_STORE_BYTES, 256 MiB) across all
entries. A running total is maintained on every insert/eviction so the
cap is enforced without re-encoding the whole store on the request path;
oldest entries are evicted once the budget is exceeded, always keeping
the most-recent entry so recovery still works for the request populating
the store. +2 regression tests.
* fix(guardrails): gate and bound Compresr recovery loop
Two hardening fixes to the compresr_retrieve agentic loop:
1. Only run the loop when a retrieve call resolves to recovery state this
guardrail actually created for the request. Previously the gate checked
only that the caller-supplied tool list contained a compresr_retrieve
function and that the model emitted a call, so a caller could define
their own same-named tool and force an extra provider round-trip with
nothing to recover. The plan now returns run_agentic_loop=False when no
requested hash resolves.
2. Bound the follow-up against retrieval amplification: each distinct hash
is expanded at most once (repeats get a short marker) and at most
_MAX_RETRIEVALS_PER_LOOP calls are honored, so prompting the model to
call compresr_retrieve many times with the same marker cannot balloon
the follow-up. _retrieve_original now returns None on miss.
+3 regression tests; two existing security tests updated to assert the
stronger veto behavior (forged/cross-tenant hashes now stop the loop
entirely instead of returning a not-found follow-up).
* fix(guardrails): warn when Compresr recovery is skipped without auth scope
When enable_retrieval is on (the default) but the proxy has no per-key
auth, the request has no caller scope, so recovery is silently disabled:
content is compressed but the compresr_retrieve tool is never injected and
the originals are dropped, with no runtime indication. Emit a one-shot
call-time warning so operators can see recovery is being suppressed and
configure virtual-key auth. +1 regression test.
* style(guardrails): tighten Compresr guardrail comments
Condense the verbose multi-line inline comments and the api_base docstring
to concise form. No behavior change.
* fix(guardrails): keep injected tool on Responses API + bound recovery markers by byte cap
Two fixes for reviewer-flagged defects in the Compresr guardrail:
- Responses API: _merge_tools_after_guardrail iterated only over the
request's original tools, dropping any tool a guardrail appended (the
compresr_retrieve recovery tool) whenever the request already had tools.
Keep the appended tools so recovery works on /v1/responses.
- Recovery markers: markers + originals were built for every compressed
target before the per-call byte cap trimmed the store, so an evicted
original left a marker the model could never retrieve. Attach recovery
only while the store (existing entries under the same key + this call's
originals) stays within the cap, so a shipped marker is always retrievable
-- including on a later turn that reuses the store key.
Adds regression tests for both paths.
* refactor(guardrails): extract _existing_originals to keep apply_guardrail under the complexity gate
The byte-cap fix added a branch to apply_guardrail, tipping it past the
C901 complexity ceiling. Move the store lookup into a small helper; no
behavior change.
* fix(guardrails): harden Compresr SSRF blocklist, re-arm no-scope warning, tolerate odd tool shapes
* fix(guardrails): rerun input guardrails on Compresr retrieval follow-up
* chore: remove unrelated deepkeep files committed by mistake
---------
Co-authored-by: charafkamel <charafkamel@live.com>
* fix(openai/responses): clamp max_output_tokens below API minimum
Claude Code sends a max_tokens=1 warmup probe when running /model, which
the Anthropic Messages -> Responses adapter forwards as max_output_tokens=1.
OpenAI's Responses API rejects values below 16, so the probe failed with a
400. Clamp anything below the minimum up to 16 in map_openai_params so all
Responses API entrypoints (direct, chat->responses, anthropic->responses)
are covered.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* refactor(openai/responses): extract _enforce_min_max_output_tokens helper
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(responses): Presidio PII masking for Azure WebSocket and streaming
Wire Presidio into native Responses WebSocket forwarding and fix streaming output unmasking so masked tokens are restored for HTTP and WS clients.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix unused imports in responses handlers
* fix(responses): address Greptile review - Azure WebSocket model URL and PII logging
- Add model_in_websocket_url() to BaseResponsesAPIConfig (default True) so
providers can opt out of ?model= being appended to WebSocket URLs.
- Override model_in_websocket_url() to return False for Azure, since Azure
sends the model in the response.create body, not the URL query string.
- Use this flag in llm_http_handler to conditionally append ?model=.
- Pass masked message to _store_input() instead of the original PII-containing
message so logging destinations do not receive unmasked PII.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): mask nested response.create input format for Presidio PII
Handle the nested {"type":"response.create","response":{"input":[...]}}
format in _mask_response_create. Previously only the flat top-level input
was masked; the nested shape bypassed Presidio and forwarded raw PII
upstream. Now both shapes are normalized and masked before forwarding.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: apply black formatting to llm_http_handler and streaming_iterator
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* style: suppress PLR0915 on async_responses_websocket
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): add apply_to_output masking on Responses API WebSocket path
Previously the WebSocket guardrail filter excluded callbacks with
apply_to_output=True, leaving model-generated PII unmasked before
returning to the client.
- Collect apply_to_output callbacks separately in llm_http_handler and
pass them to ResponsesWebSocketStreaming as output_guardrail_callbacks.
- Add _mask_response_completed method that calls check_pii(output_parse_pii=False)
on text blocks in response.completed events, masking model output PII.
- backend_to_client now chains unmask (pii_tokens) → mask (apply_to_output)
before forwarding each event to the client.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): unmask PII tokens in streaming delta events and warn on guardrail init failure
- Rename _unmask_response_completed -> _unmask_response_event and extend
it to also unmask response.output_text.delta (and other delta types)
so real-time streaming clients receive original values, not PII tokens.
- Split the broad except-and-swallow into ImportError (expected in SDK-only
environments) vs Exception (unexpected — now logs a warning so operators
know masking is disabled).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): enforce authorized model on WebSocket frames and remove proxy import
Security: add _enforce_authorized_model to ResponsesWebSocketStreaming that
overwrites both flat and nested model fields in every response.create frame
with the connection-authorized model, preventing deployment-substitution
attacks where an authenticated user sends a different model name in the frame
body after connecting with an allowed model.
Layering: remove the _OPTIONAL_PresidioPIIMasking isinstance check and proxy
import from the SDK handler. Use duck-typed checks (callable check_pii +
get_presidio_settings_from_request_data) so any guardrail implementing the
interface works, not just Presidio.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): add _unmask_pii_text to duck-typed contract and mask delta frames
- Add callable(_unmask_pii_text) check to the guardrail_callbacks filter so
a custom guardrail missing that method cannot cause an AttributeError and
silently kill the WebSocket session.
- Extend _mask_response_completed to also mask response.output_text.delta
(and other delta types) for apply_to_output callbacks, so real-time
streaming clients do not receive unredacted model-generated PII in deltas.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix Responses WebSocket guardrail edge cases
* fix(responses): log masked output and suppress deltas when apply_to_output active
- Move _store_event to after _mask_response_completed so logs receive the
redacted form, not raw model output containing PII.
- Suppress delta event forwarding when output_guardrail_callbacks are
present: per-fragment Presidio cannot catch PII that spans multiple
chunks (e.g. "alice@" + "example.com"). Clients receive only the
fully-masked response.completed, which Presidio scans on complete text.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(responses): mask and suppress response.output_item.done for apply_to_output
response.output_item.done carries completed item text in item.content[*].text
before response.completed arrives, allowing unmasked PII to reach the client.
- _unmask_response_event: unmask input-PII tokens in item.content[*].text
- _mask_response_completed: run check_pii on item.content[*].text for
apply_to_output callbacks (same as response.completed handling)
- backend_to_client suppression: also skip response.output_item.done when
output_guardrail_callbacks are active; client receives only the
fully-masked response.completed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(types): cast response_obj to ResponsesAPIResponse to satisfy mypy
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Revert "fix(types): cast response_obj to ResponsesAPIResponse to satisfy mypy"
This reverts commit d5969557628f9aff58948b9d37cc64d577f95a15.
* Revert "fix(responses): mask and suppress response.output_item.done for apply_to_output"
This reverts commit 219fd54ea3446f4399fde40c07ba0617e2834573.
* fix(types): accept dict responses in guardrail output write-back
Streaming response.completed events pass a dict response object, so widen
_apply_guardrail_responses_to_output to match its existing runtime handling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(responses): skip Presidio masking on suppressed WebSocket delta events
Delta events are dropped wholesale when apply_to_output masking is active,
so masking them first issued a wasted check_pii call per fragment. Move the
suppression check ahead of the unmask/mask passes; the event type is
invariant across both, so client-visible behavior is unchanged.
* test(responses): cover Responses WebSocket PII masking hooks
Add regression tests for the native Responses WebSocket guardrail path:
input masking and model enforcement in _mask_response_create, token
unmasking in _unmask_response_event, apply_to_output masking and delta
suppression in _mask_response_completed/backend_to_client, and the
get_websocket_url / model_in_websocket_url defaults for the base and
Azure configs. Raises diff coverage above the codecov patch target.
* fix(responses): suppress text-bearing done events under output PII masking
When apply_to_output masking is active on a native Responses WebSocket,
response.output_text.done, response.content_part.done, and
response.output_item.done carry the full model output before the masked
response.completed arrives, so an authenticated client could read
unmasked PII from those events. Suppress them alongside delta events; the
client receives only the fully-masked response.completed.
* refactor(responses): drop dead delta branch in WebSocket output masking
Delta events are suppressed in backend_to_client before _mask_response_completed
runs when output masking is active, so the method's delta-handling branch was
unreachable. Restrict it to response.completed and cover the Responses API
unmask path with a Pydantic ResponseCompletedEvent regression test.
* fix(presidio): flush buffered chat chunks on mixed unmask stream
_stream_pii_unmasking buffered ModelResponseStream chunks but returned
early once a /v1/responses event was seen, silently dropping the buffered
chat chunks. Flush them in order before switching to passthrough, mirroring
_stream_apply_output_masking, and cover it with a regression test.
* fix(responses): mask instructions and tool-call arguments in WebSocket PII path
Presidio masking on the native Responses WebSocket path left two gaps. On the
request side _mask_response_create only walked the input containers, so PII
placed in the instructions field of a response.create frame was forwarded
upstream and logged unmasked even with output_parse_pii enabled. Now both the
flat and nested instructions strings are masked alongside input.
On the response side _mask_response_completed only masked content text blocks,
so model-produced PII inside function-call arguments could reach the client when
apply_to_output was enabled, both via the standalone
response.function_call_arguments.done event and via the function_call output
items in response.completed. The done event is now suppressed under output
masking and completed function-call arguments are run through check_pii before
forwarding or logging.
* fix(responses): suppress reasoning_summary_text.done under output PII masking
* fix(responses): mask function_call_output.output in WebSocket PII path
response.create input items of type function_call_output carry
user-controlled text in output, not content, so the Presidio masking
pass forwarded that text upstream unmasked. Mask the output field
(string or list of text blocks) alongside content.
* fix(responses): mask reasoning summary PII in WebSocket output path
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tests): drop import-time completion call in test_register_model
test_update_model_cost_via_completion() was invoked at module scope, so it
ran during pytest collection and fired a live OpenAI completion. The local
test jobs glob the whole tests/local_testing folder and let pytest import
every file, narrowing what runs only afterward with -k, so this call
executed in every one of those jobs regardless of their filter. When the
request failed (for instance a 429 once the OpenAI account hit its quota),
collection of the file errored and aborted the entire session, which is why
langfuse, assistants, router and local_testing_part2 all reported
"ERROR collecting tests/local_testing/test_register_model.py" and never ran
their own tests.
Remove the stray call and add a regression that parses the module and fails
if any locally defined function is invoked at module scope again
* test: also guard async def from module-scope invocation
ast.AsyncFunctionDef is a distinct node from ast.FunctionDef, so an async
test invoked at module scope would have slipped past the guard. Collect
both kinds of definitions
* fix(responses): send Content-Type application/json on OpenAI responses requests
OpenAI's responses API now rejects body-less requests (GET/DELETE) that
arrive without a content type, returning 500 "Unsupported content type:
'application/octet-stream'. This API method only accepts 'application/json'
requests". litellm's create path got the header for free because httpx sets
it when a json body is present, but the delete/get handlers send no body and
so sent no content type. The official OpenAI SDK declares
Content-Type: application/json on every request; mirror that in
validate_environment so all OpenAI responses calls carry it.
This is what made tests/openai_endpoints_tests/test_e2e_openai_responses_api.py::test_basic_response
fail on the responses.delete() call.
* feat(openai): apply regional-processing cost uplift for EU/US data residency
OpenAI charges a 10% uplift on the latest GPT models when requests are
served from a regionalized hostname (eu./us.api.openai.com). Infer the
region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`,
and multiply the computed cost by a per-model
`regional_processing_uplift_multiplier_<region>` field.
https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW
* test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema
* fix(cost): tighten data_residency inference and restore model_cost in tests
- Only infer OpenAI data_residency when custom_llm_provider == "openai";
drop the implicit None fallback so non-OpenAI callers can't accidentally
pick up a regional tag from a stray OpenAI hostname.
- _local_model_cost_map fixture now snapshots and restores
litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak
state across the session.
* refactor(openai): move data_residency helper under llms/openai
* fix: thread data_residency through realtime stream cost calculation
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(cost): thread data_residency through batch_cost_calculator
Apply the OpenAI regional-processing uplift multiplier to retrieve_batch
cost paths so Batch API requests served via eu./us.api.openai.com are
priced at the same uplifted token rates as completions/transcriptions.
* refactor(openai): encapsulate provider check inside infer_openai_data_residency
Move the custom_llm_provider == "openai" guard from get_litellm_params
into the helper itself so the core utility no longer carries
provider-specific dispatch logic. Callers pass through the provider
unconditionally; the helper returns None for any non-OpenAI provider.
* fix(responses): thread data_residency through Responses logging params
The Responses API paths build their logging litellm_params dict after
provider resolution but did not include data_residency, so cost calc
saw None even when the effective api_base was a regional OpenAI host.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): handle empty streaming tool calls (#28549)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* [Feature][Bug Fix] Decouple Azure OpenAI Deployment ID from model name via base_model to fix gpt5 model routing (#28490)
* feat(azure): decouple deployment ID from model name via base_model
Azure OpenAI deployments have arbitrary names (deployment IDs) that may
not match the underlying model. Previously, model-type detection
(o-series, gpt-5, etc.) relied on substring matching against the
deployment name, causing misrouted configs and rejected params when
deployment names were non-standard (e.g. 'my-deployment-id' for gpt-5.2).
This change extends the existing base_model field to drive model-type
detection, config selection, supported param resolution, and param
mapping throughout the Azure call path:
- _get_azure_config() uses base_model for is_o_series/is_gpt_5 checks
- get_provider_chat_config() threads base_model for Azure
- get_supported_openai_params() accepts and uses base_model
- get_optional_params() accepts base_model and passes it to all Azure
config method calls (get_supported_openai_params, map_openai_params)
- azure.py completion handler uses base_model for GPT-5 detection
- Config internal methods (e.g. is_model_gpt_5_2_model) now receive
base_model so features like logprobs are correctly enabled
Fully backward compatible - when base_model is unset, behavior is
identical. Existing o_series/ and gpt5_series/ prefix workarounds
continue to work.
Usage in proxy config:
model_list:
- model_name: my-gpt5
litellm_params:
model: azure/my-deployment-id
model_info:
base_model: azure/gpt-5.2
Fixes: non-standard deployment names like 'prefix-gpt-5.2' rejecting
logprobs/top_logprobs despite the underlying model supporting them.
* Addressing Greptile comments.
* gemini-3.1-flash-lite pricing (#27933)
* feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers
* fix pricing
* add service tier
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
* fix(openai-responses): strip Anthropic cache_control from Responses API requests (#28431)
Squash-merged by litellm-agent from cwang-otto's PR.
* Treat None litellm_provider as wildcard in _check_provider_match (#28523)
Squash-merged by litellm-agent from adityasingh2400's PR.
* fix greptile
* fix: use _azure_detection_model in default Azure branch of get_supported_openai_params
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(openai-responses): strip cache_control on compact endpoint as well
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Felipe Garé <90070734+FelipeRodriguesGare@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: withomasmicrosoft <withomas@microsoft.com>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com>
Co-authored-by: Aditya Singh <60082699+adityasingh2400@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(router): add auto_router/quality_router for quality-tier routing (#25987)
* feat(router): add auto_router/quality_router for quality-tier routing
Adds a new auto-router type that routes a request to a model at a target
quality tier. The quality tier is inferred by re-using the existing
ComplexityRouter's classification, then mapped through an admin-configured
complexity_to_quality table. Each candidate model declares its own
quality_tier in model_info.litellm_routing_preferences.
Resolution strategy: exact tier match, else round up to the next higher
tier, else fall back to default_model.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): add capability-based filtering
Each deployment can declare a `capabilities: List[str]` field in
`model_info.litellm_routing_preferences` (e.g. ["vision",
"function_calling"]). Requests can pass `litellm_capabilities` in
`request_kwargs` to require specific capabilities — the router will only
route to deployments whose declared capabilities are a superset.
Resolution still walks tier (exact → round up), but at each tier filters
by capability before picking. Falls back to default_model only when it
also satisfies the required capabilities; otherwise raises rather than
silently routing to a model that lacks a required capability.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): expose routing decision in response headers
For transparency, expose the QualityRouter's routing decision in the
proxy response headers:
x-litellm-quality-router-model → picked model_name (e.g. "haiku-vision")
x-litellm-quality-router-tier → resolved quality tier (e.g. "1")
x-litellm-quality-router-complexity → ComplexityTier name (e.g. "SIMPLE")
Mechanism: the pre-routing hook stashes the decision in
request_kwargs["metadata"]["quality_router_decision"]. After the call
returns, Router.set_response_headers lifts the decision into
response._hidden_params["additional_headers"] alongside the existing
x-litellm-model-group / x-litellm-model-id headers. Existing metadata
keys (trace_id, user_id, etc.) are preserved.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): replace capabilities with keyword override
Drops the capability-based filtering in favor of a keyword-based override
for v0:
- RoutingPreferences.keywords: List[str] (replaces capabilities) — each
deployment can declare substring keywords.
- If any declared keyword (case-insensitive) appears in the user message,
the router short-circuits the complexity-classification flow and routes
to the matching deployment.
- Tiebreaker for overlapping keyword matches: quality_tier DESC, then
cheapest model_info.input_cost_per_token ASC. Unpriced models lose ties
to priced ones.
Decision metadata + headers now expose the override:
x-litellm-quality-router-via → "keyword" | "quality_tier"
x-litellm-quality-router-keyword → matched keyword (only on keyword route)
x-litellm-quality-router-complexity → complexity tier (only on tier route)
Removes:
- request_kwargs["litellm_capabilities"] reading
- _model_capabilities, _model_supports_capabilities,
_first_capable_model_at_tier, capability filter in
_resolve_model_for_quality_tier
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): add explicit `order` to RoutingPreferences
Adds an explicit priority field to RoutingPreferences for resolving
collisions deterministically:
RoutingPreferences.order: Optional[int] # lower wins; unset = +inf
Used as the PRIMARY tiebreaker in two places:
1. Keyword overlap: when multiple deployments declare the same matching
keyword, sort by (order ASC, quality_tier DESC, input_cost_per_token
ASC, model_name ASC). Explicit always beats implicit.
2. Tier resolution: when multiple deployments share a quality tier,
`_resolve_model_for_quality_tier` picks the one with the lowest
order. The tier list is now sorted at index-build time.
This lets admins make routing decisions explicit when the natural
quality-and-price ordering would pick the wrong model.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): reorder tiebreak to (quality, order, price)
Changes the tiebreak ordering so quality_tier always wins first, then
explicit `order` is used to break ties within the same tier, then price
breaks the rest:
1. quality_tier DESC ← best model wins first
2. order ASC ← explicit priority within a tier
3. input_cost_per_token ASC
4. model_name ASC
Previously `order` was the primary key — that meant a tier-2 model with
`order=1` would beat a tier-3 model with no `order`, which is the wrong
default. Now `order` only resolves collisions among same-tier candidates.
Tier resolution (within a single tier) keeps the same key minus quality:
(order ASC, cost ASC, name).
Test renames + flips:
- test_explicit_order_overrides_quality_tier → test_quality_wins_over_explicit_order
- new: test_order_breaks_tie_within_same_quality_tier
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* fix(quality_router): resolve Greptile review feedback
Addresses four P1 findings from PR review plus test coverage:
1. set_model_list missing quality_routers reset
- Hot-reloading the Router would leave stale QualityRouter instances
pointing at the old model_list. `set_model_list` now clears
`self.quality_routers` alongside the other indices.
2. Round-down fallback before default_model
- `_resolve_model_for_quality_tier` now rounds DOWN to the closest
lower tier after round-up fails, before falling back to
`default_model`. Degrades gracefully rather than jumping straight
off-tier.
3. RoutingPreferences validation bypass
- `_build_tier_index` now instantiates `RoutingPreferences(**prefs)`
so invalid shapes (e.g. non-int quality_tier) raise a clear
ValueError instead of silently succeeding.
4. Config-ordering dependency
- `_tier_to_models` is now built lazily on first access. Previously,
eager construction in `__init__` meant a QualityRouter deployment
had to appear AFTER all its referenced models in config.yaml,
because `Router._create_deployment` populates `model_list`
incrementally. Any `available_models` defined after the router
entry would silently be reported as missing.
Also adds 6 new tests covering each fix:
- test_invalid_quality_tier_type_raises_clear_error
- test_router_can_be_instantiated_before_its_targets_exist
- test_set_model_list_clears_quality_routers_registry
- test_rounds_down_when_no_higher_tier_exists
- test_rounds_down_prefers_closest_lower_tier
- test_prefers_round_up_over_round_down
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* style: apply black 24.10.0 formatting to pre-existing offenders
Unblocks the LiteLLM Linting check for this PR — these 12 files are already
failing `black --check` on main (the lint workflow only runs on PRs, so main
drifts). No behavior changes; formatting-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update litellm/router.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Support /v1/responses in complexity router (#26137)
* feat(proxy): add --reload flag for uvicorn hot reload (dev only)
Opt-in CLI flag, off by default, no env var. Only affects the uvicorn
run path; gunicorn/hypercorn paths and prod (which doesn't pass the
flag) are unaffected.
* Feature/add audio support for scaleway (#26110)
* feat(scaleway): add SCALEWAY to LlmProviders enum
* feat(scaleway): add audio transcription config and dispatch wiring
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(scaleway): add behavior tests for audio transcription config
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(scaleway): advertise audio_transcriptions in endpoint-support JSON
* docs(scaleway): document audio transcription support
* fix(scaleway): address PR review — plain-text response_format + missing-key fail-fast
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(scaleway): cover new response paths, drop gettysburg.wav coupling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Prompt Compression - add it to the proxy (#25729)
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Support /v1/responses in complexity router
Adds cross-format support to the complexity router via the guardrail
translation handler dispatch. Adds get_structured_messages to base
translation plus OpenAI chat, Responses, and Anthropic handlers.
Auto-router helper _extract_text_from_messages handles tool-call and
multimodal messages. Widens async_pre_routing_hook messages type to
Dict[str, Any].
Fixes https://github.com/BerriAI/litellm/issues/25134
* chore: apply black formatting
* fix: fallback to trying each handler when route inference fails
---------
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* test: cover _is_quality_router_deployment and init_quality_router_deployment
* fix: reset auto_routers on set_model_list to prevent hot-reload ValueError
* style: apply black formatting to websearch_interception and agentic_streaming_iterator
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
- Add OpenAITokenCounter using POST /v1/responses/input_tokens endpoint
- Add litellm.acount_tokens() public async API that auto-routes to provider APIs
- Add proxy endpoint POST /v1/responses/input_tokens for OpenAI-compatible counting
- Transform chat tools format to Responses API format for correct token counting
- Fall back to local tiktoken when provider API unavailable
Fixes#22302
When an OpenAI-compatible upstream provider emits minimal streaming event
payloads that omit required fields (e.g. created_at, output, output_index,
content_index), Pydantic raises a ValidationError crashing the SSE stream
and returning HTTP 500.
Fall back to model_construct() on ValidationError, consistent with the
existing pattern in transform_response_api_response for non-streaming.
Fixes https://github.com/BerriAI/litellm/issues/20570
Signed-off-by: Varun Chawla <varun_6april@hotmail.com>
* fix: check for model_response_choices before guardrail input
* test: add tests for responses api translation
* fix: protect other guardrail translations
* refactor: remove type ignores
* anthropic request body got mutated fix
* add warning when extra_body is provided but user is non premium
* fix: resolve mypy union-attr errors in anthropic guardrail handler
Cast choices[0] to Choices type before accessing .message attribute
to satisfy mypy's union type checking for Choices | StreamingChoices.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* add logger when model response has no choices for streaming /response and /messages
* update pyproject.toml as requested
* Revert "update pyproject.toml as requested"
This reverts commit 541a2b075a.
* update pyproject.toml as requested
* Revert "update pyproject.toml as requested"
This reverts commit 716ea0caa1.
---------
Co-authored-by: Xiaohan Fu <xiaohan@grayswan.ai>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
