mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
297 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
7d745521bf | fix(guardrails): merge synthesized tools under scan_only_tool_results and reject role-filtered no-op combos at init | ||
|
mateo-berri
|
c2998dea75 | fix(guardrails): guard tools write-back under scan_only_tool_results and warn on role-filtered no-op scans | ||
|
mateo-berri
|
d70e10982a |
fix(guardrails): keep tool-results-only scans off function definitions and merge scoped write-backs
Gate the OpenAI handler's tools forwarding behind scan_only_tool_results, matching the Anthropic handler, so a tool-results-only scan can no longer evaluate or rewrite trusted function definitions. When a guardrail returns a replacement structured_messages list, substitute the returned messages back into the positions their scoped originals came from instead of installing the scoped list as the whole conversation, so out-of-scope messages (system prompt, prior turns) survive redaction on both the OpenAI and Anthropic paths. |
||
|
mateo-berri
|
2ba4e91766 | feat(guardrails): add scan_only_tool_results to scope unified guardrails to tool results | ||
|
mateo-berri
|
bee787b4b5 |
fix(guardrails): scan /v1/messages tool traffic
Guardrails silently skipped three surfaces on the Anthropic Messages path, so an agent loop driven by /v1/messages ran unguarded: - The Anthropic input translation never walked tool_result blocks, so content returned by a local tool (a curl, a file read, an MCP call) reached the model unscanned in both the string and list content shapes, images inside a tool_result included. - tool_permission only understood ModelResponse, so an Anthropic non-streaming response or a raw SSE stream carrying tool_use blocks passed through with no rule ever evaluated. - ContentFilterGuardrail scanned inputs["texts"] but never inputs["tool_calls"], so the arguments a model proposes for a tool call went unchecked. Tool call arguments are parsed as JSON before filtering so a MASK action rewrites the value and leaves the payload valid JSON; non-JSON arguments fall back to scanning the raw string. Denied tool_use blocks are dropped from the Anthropic content array and replaced with a text block, and stop_reason resets to end_turn when nothing tool-shaped survives. |
||
|
mateo-berri
|
e8a80b9883 | docs(anthropic): state why usage-shape detection requires a cache key, pin Responses-shape rejection | ||
|
mateo-berri
|
bd04520d98 | Merge branch 'litellm_internal_staging' into litellm_gpt56_cache_token_pricing | ||
|
yucheng-berri
|
1018d18e6b
|
fix(anthropic): split mixed stream chunks by payload kind (#35289)
* fix(anthropic): split mixed reasoning stream chunks * style: use builtin generic annotation * fix(anthropic): split mixed stream chunks by payload kind The mixed-chunk split cleared only the fields it knew about on each deep-copied piece, so any other payload riding the chunk survived on both pieces: tool_calls were emitted as two tool_use blocks with the same id, thinking_blocks on the text piece emitted duplicated thinking into a text block while dropping the answer text, and chunks whose reasoning arrived only as thinking_blocks never split at all Rebuild each piece's delta from scratch with exactly one payload kind (reasoning, text, tool calls), ordered to match native Anthropic block order. Fresh Delta construction keeps unset attributes deleted, which matters because the translators branch on hasattr, and prevents future Delta fields from riding along on every piece * fix(anthropic): keep continuation and multi-choice chunks unsplit, emit signature-less thinking once Adversarial verification against the merge-base found three shapes where the payload-kind split changed behavior beyond its target: a mixed chunk carrying a tool argument continuation was torn into a truncated block plus a fabricated one, a multi-choice chunk lost its secondary choices' payload, and a signature-less thinking_blocks piece inherited the non-empty block start body so accumulators collected the thinking twice Continuation and multi-choice chunks now pass through the splitter untouched, matching the merge-base byte for byte, and signature-less thinking_blocks pieces are normalized to reasoning_content so the block start opens empty and the thinking text is emitted exactly once --------- Co-authored-by: Napuh <naamanynadiemas@gmail.com> |
||
|
Napuh
|
2f7574d7c1
|
fix(anthropic-adapter): open the first content block with the real upstream type so reasoning-first streams start with thinking (#34433)
* fix(anthropic-adapter): open first content block with the real upstream type * fix(anthropic): defer blank leading stream deltas |
||
|
tin-berri
|
32a4377acd
|
Merge pull request #34589 from BerriAI/litellm_lit4798_glm_stop_thinking
fix(anthropic-adapter): translate stop_sequences and disabled thinking for non-Claude targets |
||
|
Devin AI
|
8136c96284 | test(anthropic): cover usage-shape detection for cache token pricing | ||
|
Devin AI
|
ed366aafbe |
fix(anthropic): preserve prompt cache tokens in messages to responses api usage
Also map gpt-5.6 flex/priority cache_creation rates into ModelInfo so cache writes are not billed at the standard rate on those service tiers |
||
|
devin-ai-integration[bot]
|
96f58fac53
|
fix(router): don't cool down parent deployment on advisor sub-call failure (#33792)
* fix(router): don't cool down parent deployment on advisor sub-call failure Advisor orchestration issues a sub-call to a different provider/credentials than the selected deployment. When that sub-call fails (e.g. a 401 because no advisor API key is configured), the exception propagates up and the router's deployment_callback_on_failure attributes it to the healthy parent deployment's model_info.id, cooling it down and rejecting unrelated callers to the same model group. Tag advisor sub-call failures on the exception and skip cooldown for them in deployment_callback_on_failure. The exception is tagged rather than wrapped so its type is preserved and retry/fallback classification and the client-facing error are unchanged. Genuine executor/deployment failures are untagged and still cool down as before. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): tag advisor orchestration failures via provider-neutral util Address review on LIT-4565: move the cooldown-exemption marker into litellm/router_utils/cooldown_handlers.py so the router imports it at module top instead of an in-function anthropic import, and extend the exemption to AdvisorMaxIterationsError so a max-iterations orchestration failure no longer cools down the healthy executor deployment. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tin Chi Lo
|
5072590c27 |
fix(anthropic-adapter): dedupe reasoning_effort wrapping to close sibling gap
translate_thinking_for_model duplicated the same summary/auto_summary
wrapping logic as _translate_thinking_to_openai without the
disabled-thinking guard, so it could still wrap "none" into an
{effort, summary} dict when reasoning_auto_summary is enabled (caught
by Cursor Bugbot). Extract the wrapping rule into one shared
_apply_reasoning_summary_wrapping helper used by both call sites so
this invariant can't drift apart again.
|
||
|
Tin Chi Lo
|
fed03a41d1 |
test(anthropic-adapter): cover empty stop_sequences edge case
Codecov flagged the empty-list early-return in _translate_stop_sequences_to_openai as an uncovered line in the diff — add a regression test asserting stop_sequences=[] does not set new_kwargs["stop"]. |
||
|
Tin Chi Lo
|
9da21f38a9 |
fix(anthropic-adapter): keep disabled-thinking reasoning_effort a plain string
Guard against reasoning_auto_summary wrapping "none" into a dict when thinking is disabled — there's no reasoning trace to summarize, and non-Claude providers (e.g. Fireworks) expect reasoning_effort as a plain string. |
||
|
Tin Chi Lo
|
b3e27a0bc3 |
fix(anthropic-adapter): translate stop_sequences and disabled thinking for non-Claude targets
Claude Code's auto-mode classifier sends stop_sequences and thinking:
{type: disabled} on /v1/messages. The Anthropic adapter passed
stop_sequences through unchanged instead of mapping it to OpenAI's stop,
which Fireworks' OpenAI-compatible endpoint rejects with HTTP 400. It also
dropped disabled thinking instead of mapping it to reasoning_effort: none,
so the model spent its output budget on reasoning it was told to skip.
Resolves LIT-4798
|
||
|
yucheng-berri
|
76b0b10908
|
fix(guardrails): add /v1/messages support for Straiker plugin (#34548)
* fix(guardrails): add /v1/messages support for Straiker plugin - Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py) - Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs * fix(guardrails): gate cross-surface message resolution and cover streaming request data Resolve request messages only for surfaces that have a mapped translation handler. The unguarded fallback tried every registered handler in turn, which raised AttributeError out of the guardrail's error handling on list-shaped `input` bodies, and synthesized a chat message that was never sent for bodies it happened to parse. Prepare request data on the mid-stream Anthropic branch as well, matching the terminal branch and the OpenAI handler, so guardrails that scan before end-of-stream still receive identity metadata. Read usage from Anthropic dict responses so non-streaming /v1/messages reports token counts instead of null. Add regression coverage for the streaming request data on both the terminal and mid-stream branches; reverting either now fails. --------- Co-authored-by: cs-mehta <chandra@straiker.ai> |
||
|
shivam
|
a376f72400 |
fix(responses): stop treating stream_options as a Responses API param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
2ec2a92da2 | fix(anthropic): strip all remaining output_format schema keywords rejected by Anthropic | ||
|
Mateo Wang
|
46440e2df4
|
fix(anthropic): strip uniqueItems + other unsupported array/object constraints from output_format schema (#33981) (#34313)
* fix(anthropic): strip uniqueItems + other unsupported array/object constraints from output_format schema Anthropic's structured outputs (`output_format`) validate the JSON schema against a strict subset and reject cross-element / count constraints that a constrained-decoding grammar cannot enforce, returning a 400 `invalid_request_error`. `filter_anthropic_output_schema` already stripped the numeric / string / item-count constraints (minimum, maximum, exclusiveMinimum/Maximum, minLength, maxLength, minItems, maxItems) but still let these through: - uniqueItems - contains / minContains / maxContains - minProperties / maxProperties so a request using them fails with e.g. "output_format.schema: For 'array' type, property 'uniqueItems' is not supported". This is provider-visible: newer Claude models on the native `output_format` path (e.g. `azure_ai`) 400, while `vertex_ai` is unaffected because it is forced onto the permissive tool-use path (#18625 / #19201). Add the missing keywords to the unsupported-field set and the description map, and skip the advisory description note for a disabled boolean constraint (`uniqueItems: false`) so it isn't misdescribed as required. * fix(anthropic): serialize contains sub-schema in output_format advisory note Address Greptile review: the `contains` advisory note previously discarded the sub-schema, so the description only said an item must match "a schema" without saying which. It now serializes the sub-schema as JSON (e.g. "array must contain an item matching: {\"type\": \"integer\", \"const\": 1}"), matching the other stripped constraints which carry their value. Sub-schema (dict/list) values are json.dumps'd; scalar constraints are unchanged. * style(anthropic): apply ruff format to output_format filter change * style(test): ruff format anthropic schema filter tests * test(anthropic): cover output_format array/object constraint filtering in test_litellm tree Mirrors the schema-filter tests under tests/test_litellm/ so the coverage job exercises the new uniqueItems/contains/min-maxProperties handling and the uniqueItems: false branch. --------- Co-authored-by: Darien Kindlund <darien@kindlund.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
devin-ai-integration[bot]
|
fa6b209165
|
feat(guardrails): add only_scan_new_messages for per-session incremental scanning (#33278)
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(guardrails): use fixed TTL constant and revert unrelated test formatting Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy routes Bedrock through the unified apply_guardrail interface, so the flag had no effect live. Move incremental selection into apply_guardrail: filter the flat texts list against per-session scanned hashes, skip the Bedrock call when nothing is new, and mark hashes only after a successful (non-blocked) scan. Full-context fallback is preserved when there is no session id, the cache is unavailable, or a masking guardrail is configured. Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover session-id fallbacks and mark_texts_scanned guards Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover generic agent multi-turn incremental scan Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover incremental scan cache resolver fallbacks Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(guardrails): cover flag interactions and /v1/messages incremental scan semantics * feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable * test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Yucheng Zhu <yucheng@berri.ai> |
||
|
tin-berri
|
43e4af73f0
|
Merge pull request #33631 from BerriAI/litellm_lit4517_messages_mcp_gateway
feat(mcp): support MCP servers on the Anthropic /v1/messages API |
||
|
yuneng-jiang
|
966ff65fec
|
fix(anthropic): emit message_start once in Responses stream adapter (#32667) (#33793)
* fix(anthropic): emit message_start once in Responses stream adapter * test(anthropic): cover response.created message_start guard branch Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Napuh <55241721+Napuh@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e59add11cd
|
fix(anthropic): self-heal on missing thinking-signature errors from Bedrock/Vertex (#33719)
* fix(anthropic): self-heal on missing thinking-signature errors from Bedrock/Vertex Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): narrow thinking signature error marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): stabilize prompt caching fixture size Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: re-trigger CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tin Chi Lo
|
cd3ac05a1f |
fix(mcp): forward the caller's MCP credentials from every gateway surface
The /v1/messages handler resolved only the auth object and the trace id, so tool listing and tool execution ran without the caller's MCP auth headers. That fails quietly rather than loudly: the tool still executes, just with no credentials, so every server behind interactive OAuth, a bearer token or per-user env vars returns nothing while the model reports it has no access. Only a no-auth server looks healthy, which is exactly what the first proof used. Threading the missing arguments would have left the real problem in place. Each gateway surface rebuilds the same context by hand (responses/main.py twice, chat_completions_handler, mcp_streaming_iterator), which is why a new surface drops fields; this adds a fifth that dropped six of eight. Resolve it once into a frozen MCPRequestContext and have the handlers take that, so a field cannot be forgotten at a call site. chat_completions_handler now uses it too, and the resolver reads user_api_key_auth from both metadata keys because LITELLM_METADATA_ROUTES carry it in litellm_metadata while chat uses metadata. Also stop the loop when every tool call was skipped. tool_results is empty then, and the tool_result message built from it has empty content, which Anthropic rejects; the caller saw a 400 from mid-loop instead of the model's own answer. Tests pin both: dropping the headers from either listing or execution fails, and so does removing the empty-results guard. |
||
|
Tin Chi Lo
|
ae952ce971 |
feat(mcp): support MCP servers on the Anthropic /v1/messages API
MCP tool calling worked on /v1/chat/completions and /v1/responses but not on /v1/messages. Those are the only two surfaces with an MCP gateway entry point, so a litellm_proxy MCP reference reached Anthropic verbatim inside tools and the API rejected the request with "Input tag 'mcp' found using 'type' does not match any of the expected tags". The playground never surfaced this because it dropped the reference before sending, and disabled the MCP selector for the endpoint. Add the third entry point in anthropic_messages_handler, ahead of the provider branch so it covers the native path and both bridges from one place. The gateway expands the reference against the caller's own credentials and access control, which is the whole point of routing it through litellm rather than handing the url to the provider. /v1/messages needs Anthropic's own tool shape, so transform_mcp_tool_to_anthropic_tool joins the OpenAI chat and Responses transforms alongside it. The tool loop speaks tool_use and tool_result rather than OpenAI tool_calls, and reuses the existing FakeAnthropicMessagesStreamIterator to re-stream the result, the same pattern the websearch interception already uses on this route. Argument extraction moves into the shared extractor: an Anthropic tool_use block carries its arguments under `input`, and reading only `arguments` failed silently, executing the tool with every argument dropped. On the frontend the request builder declared selectedMCPTools and never read it, so no tools key was ever sent. Wire it through a shared block builder and add the endpoint to MCP_SUPPORTED_ENDPOINTS, which is what greys the selector out. Resolves LIT-4517 Resolves LIT-4518 |
||
|
devin-ai-integration[bot]
|
ebc6fdb4c2
|
fix(cli/anthropic): unblock lite autoroute proxy deps, adaptive thinking, and thinking+signature streaming (#33507) | ||
|
devin-ai-integration[bot]
|
bbd52984b1
|
fix(anthropic): stop 500 on combined thinking+signature streaming chunk (#33505) | ||
|
yucheng-berri
|
587b8aca9b
|
feat(guardrails): add Compresr guardrail for query-aware context compression (#33295)
* feat(guardrails): add Compresr guardrail for query-aware context compression Adds a first-class guardrail that compresses bulky message content (tool outputs, RAG chunks, search results) through the Compresr API before the request reaches the LLM, via the apply_guardrail / structured_messages hook so it covers /chat/completions, /v1/messages, and /v1/responses (the latter through the texts channel, mirrored only when the replacement is unambiguous; anything ambiguous is left uncompressed). Distinct from whole-conversation compressors: - Query-aware: each message is compressed against the intent that produced it (a tool output against its originating tool call's name + arguments, resolved via tool_call_id; otherwise the last user message). - Recoverable: each compressed message carries a hash marker and the request gains a compresr_retrieve tool, so the model can pull the original content back through the agentic loop when the compressed version is not enough. Originals are cached in-process, scoped to the caller's virtual-key hash plus the request's litellm_call_id, with a TTL and a per-call byte cap; recovery is skipped when no caller scope is available so one caller can never read another's originals. The store is per-process, so multi-worker deployments need sticky routing (or enable_retrieval=false). Fail-closed by default (fail_open configurable), SSRF-validated api_base (alternate IP-literal encodings included), cross-tenant-isolated recovery store, and upstream errors redacted from client-facing responses. The outbound client follows redirects and re-resolves DNS per request, so the api_base host/IP checks are defense-in-depth, not a full SSRF guarantee; this is documented as a known limitation. Requests where nothing was actually compressed are returned untouched (same object identity) so handlers skip the write-back. Auto-discovered via the guardrail_hooks registry. * fix(guardrails): cap Compresr recovery store total memory The recovery store bounded bytes per call and entry count, but had no aggregate cap: 256 tracked call ids at the 10 MiB per-call default could retain ~2.5 GiB per worker. A flood of requests with distinct x-litellm-call-id values and large compressible tool outputs could exhaust a shared proxy worker. Add a global byte budget (_MAX_TOTAL_STORE_BYTES, 256 MiB) across all entries. A running total is maintained on every insert/eviction so the cap is enforced without re-encoding the whole store on the request path; oldest entries are evicted once the budget is exceeded, always keeping the most-recent entry so recovery still works for the request populating the store. +2 regression tests. * fix(guardrails): gate and bound Compresr recovery loop Two hardening fixes to the compresr_retrieve agentic loop: 1. Only run the loop when a retrieve call resolves to recovery state this guardrail actually created for the request. Previously the gate checked only that the caller-supplied tool list contained a compresr_retrieve function and that the model emitted a call, so a caller could define their own same-named tool and force an extra provider round-trip with nothing to recover. The plan now returns run_agentic_loop=False when no requested hash resolves. 2. Bound the follow-up against retrieval amplification: each distinct hash is expanded at most once (repeats get a short marker) and at most _MAX_RETRIEVALS_PER_LOOP calls are honored, so prompting the model to call compresr_retrieve many times with the same marker cannot balloon the follow-up. _retrieve_original now returns None on miss. +3 regression tests; two existing security tests updated to assert the stronger veto behavior (forged/cross-tenant hashes now stop the loop entirely instead of returning a not-found follow-up). * fix(guardrails): warn when Compresr recovery is skipped without auth scope When enable_retrieval is on (the default) but the proxy has no per-key auth, the request has no caller scope, so recovery is silently disabled: content is compressed but the compresr_retrieve tool is never injected and the originals are dropped, with no runtime indication. Emit a one-shot call-time warning so operators can see recovery is being suppressed and configure virtual-key auth. +1 regression test. * style(guardrails): tighten Compresr guardrail comments Condense the verbose multi-line inline comments and the api_base docstring to concise form. No behavior change. * fix(guardrails): keep injected tool on Responses API + bound recovery markers by byte cap Two fixes for reviewer-flagged defects in the Compresr guardrail: - Responses API: _merge_tools_after_guardrail iterated only over the request's original tools, dropping any tool a guardrail appended (the compresr_retrieve recovery tool) whenever the request already had tools. Keep the appended tools so recovery works on /v1/responses. - Recovery markers: markers + originals were built for every compressed target before the per-call byte cap trimmed the store, so an evicted original left a marker the model could never retrieve. Attach recovery only while the store (existing entries under the same key + this call's originals) stays within the cap, so a shipped marker is always retrievable -- including on a later turn that reuses the store key. Adds regression tests for both paths. * refactor(guardrails): extract _existing_originals to keep apply_guardrail under the complexity gate The byte-cap fix added a branch to apply_guardrail, tipping it past the C901 complexity ceiling. Move the store lookup into a small helper; no behavior change. * fix(guardrails): harden Compresr SSRF blocklist, re-arm no-scope warning, tolerate odd tool shapes * fix(guardrails): rerun input guardrails on Compresr retrieval follow-up * chore: remove unrelated deepkeep files committed by mistake --------- Co-authored-by: charafkamel <charafkamel@live.com> |
||
|
mateo-berri
|
0f9d593d29 |
fix(anthropic-adapter): drop empty content_block_delta events
An empty upstream delta (e.g. Bedrock Converse's empty reasoning delta
mid-thinking-block) falls through the translate fallback as
text_delta {"text": ""} at the open thinking block's index, crashing
Anthropic SDK clients like Claude Code with "Content block is not a
text block". Payload-less deltas carry no information, so never emit
them.
|
||
|
Krrish Dholakia
|
477ef3a7e2
|
fix(anthropic): use native output capability (#33235)
* fix(anthropic): route native structured output Use model capability metadata so new native structured-output models do not require transformation allowlist changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): pass provider to capability Co-authored-by: Cursor <cursoragent@cursor.com> * test(anthropic): cover dotted model IDs Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): handle remote capability lag Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
devin-ai-integration[bot]
|
71dffc1e9a
|
fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models (#33244)
* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models * test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping * fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models Narrow the fix to the temperature reconciliation; the reasoning_effort budget cap is reverted because the live translation grid relies on budget_tokens >= max_tokens to reject unsupported effort tiers (xhigh/max) on budget-mode models, so capping turned those 400s into 200s. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Krrish Dholakia
|
39e0efa11d
|
fix(auto-router): correct Responses API tool_choice shape and propagate alias litellm_params (#32974)
* fix(anthropic-messages): send bare-string tool_choice to Responses API, propagate router-alias litellm_params
The Anthropic /v1/messages -> Responses API adapter always wrapped
tool_choice in an object ({"type": "auto"}, {"type": "required"}), but
the Responses API's tool_choice schema for these cases is a bare
string ("auto"/"required"/"none"). Sending the object shape to an
OpenAI-compatible backend (e.g. vLLM) fails Pydantic validation with a
400. The "none" case also fell through to "auto" instead of mapping to
"none".
Separately, litellm_params configured directly on a router-alias
deployment (auto_router/complexity_router, adaptive_router,
quality_router, or semantic auto_router) - e.g.
cache_control_injection_points, drop_params - were silently dropped
for every request through that alias. async_pre_routing_hook swaps
`model` from the alias name to the selected tier/route's model before
the deployment lookup runs, so the outbound call only ever merged in
the tier deployment's own litellm_params, never the alias's. Register
non-routing-config litellm_params from the alias deployment and apply
them to the request whenever a pre-routing hook substitutes the model.
* fix: satisfy ruff-strict-budget UP006 and router coverage checker
Use builtin dict[...] generics instead of typing.Dict for the two new
annotations introduced in the previous commit, since they pushed
UP006 over the codebase ceiling in ruff-strict-budget.json. Add a
direct unit test for _register_pre_routing_alias_overrides so the
text-based router_code_coverage.py checker sees it exercised by name.
* fix(router): replace alias-param denylist with a tight allowlist
_PRE_ROUTING_ALIAS_RESERVED_PARAMS excluded router-init-only keys from
the alias's litellm_params before forwarding the rest as request
kwargs, but GenericLiteLLMParams also holds deployment-management
fields (tpm, rpm, weight, tags, max_budget, budget_duration,
use_in_pass_through, litellm_credential_name, ...) on the same object.
Any of those left off the denylist would get silently forwarded as if
they were request kwargs.
Replace the denylist with a tight allowlist of exactly the two
request-shaping params this feature exists for - drop_params and
cache_control_injection_points - so unrelated management fields never
reach the outbound call regardless of what else GenericLiteLLMParams
grows to hold.
* fix(router): re-register adaptive-alias overrides on set_model_list reload
set_model_list() unconditionally clears pre_routing_alias_overrides on
every call (e.g. /config/reload), but _finalize_adaptive_router_if_configured()
skips rebuilding an AdaptiveRouter whose model_name already exists in
self.adaptive_routers - so _register_pre_routing_alias_overrides() never
ran again for an auto_router/adaptive_router alias after a reload,
silently dropping its drop_params/cache_control_injection_points.
Build the Deployment unconditionally and re-register its overrides even
on the skip-existing-router path; only the (expensive) AdaptiveRouter
construction itself stays skipped.
* style: ruff format after merging litellm_internal_staging
* fix(router): drop the alias-param allowlist, exclude only model
Per review discussion: instead of a router.py-local allowlist of exactly
which litellm_params an alias (auto_router/complexity_router,
adaptive_router, quality_router, semantic auto_router) can forward to
the request it routes, _register_pre_routing_alias_overrides now
forwards everything except `model` (the alias marker itself, e.g.
auto_router/complexity_router, never a real provider model).
Router-init-only fields (complexity_router_config,
complexity_router_default_model, auto_router_config,
auto_router_config_path, auto_router_default_model,
auto_router_embedding_model, adaptive_router_config,
adaptive_router_default_model, quality_router_config,
quality_router_default_model) now flow into request_kwargs unfiltered
too. That's safe because litellm.completion()/acompletion() already
strips anything in litellm.types.utils.all_litellm_params before
building the provider request - added these 10 keys there, alongside
the deployment-management fields (tpm, rpm, weight, ...) already listed.
Verified live: without that addition, complexity_router_config lands in
extra_body and ships raw to the provider; with it, it's stripped.
This moves the "which fields aren't real LLM params" list from a
router.py-local allowlist to the single existing global list every
completion() call already depends on, instead of maintaining two.
* refactor(router): look up alias litellm_params on demand instead of caching them
_register_pre_routing_alias_overrides cached each alias's litellm_params
into self.pre_routing_alias_overrides at deployment-init time, which
required keeping that cache in sync with set_model_list() reloads - the
exact bug the previous adaptive-router-reload fix was patching around
(AdaptiveRouter survives a reload, but the cache didn't always get
refreshed to match).
Delete the cache and the registration method entirely. async_pre_routing_hook
now looks up the alias's own litellm_params directly from self.model_list
via self.model_name_to_deployment_indices at request time, the same
model_list that's already correctly rebuilt on every set_model_list()
call. No second piece of state to invalidate, so the reload staleness
bug class isn't possible anymore, and it's less code than before.
|
||
|
Abhimanyu Kapur
|
831dbbc4df |
fix(anthropic): translate raw adaptive thinking for chat completions on pre-4.6 models
Clients that pass thinking={"type": "adaptive"} directly (not via the
reasoning_effort alias) on the /chat/completions interface had it forwarded
unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the
translation already applied on the native /v1/messages passthrough (#32867):
translate to legacy thinking={type: enabled, budget_tokens}, capped below
max_tokens, dropping thinking when max_tokens can't fit even the minimum
budget. Hoists the shared budget-capping helper onto AnthropicConfig so both
paths use one implementation.
|
||
|
mateo-berri
|
41b599b2d8 | fix(anthropic): thread real provider through capability probes instead of pinning anthropic | ||
|
mateo-berri
|
6fa088224b | fix(fallback-generalizations): cover bare Claude majors in baseline and routing, require claude- prefix in adaptive gate | ||
|
mateo-berri
|
1ccc3382d9 | feat(fallback-generalizations): widen adaptive-thinking gate to any claude family at major 5+ | ||
|
Mateo Wang
|
3a62e5428f
|
fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support (#32867)
* fix(anthropic): translate adaptive thinking/effort to pre-4.6 model support
AnthropicMessagesConfig now reshapes the 4.6+ adaptive-thinking interface
(thinking:{type:adaptive} + output_config:{effort:...}) to whatever the routed
model supports. Thinking-capable non-adaptive models (e.g. Haiku 4.5, Sonnet 4.5)
get the effort translated to a legacy thinking budget_tokens. Models with no
reasoning support have thinking/effort dropped under drop_params. And because
adaptive thinking carries no budget while the legacy form must satisfy Anthropic's
max_tokens > budget_tokens rule, the translated budget is capped below max_tokens,
dropping thinking when max_tokens can't fit the minimum budget. 4.6+ models pass
through untouched.
This matters because clients like Claude Code speak native Anthropic /v1/messages
and send the adaptive interface unconditionally, regardless of the routed model.
The native passthrough previously only capability-gated the OpenAI-style
reasoning_effort alias and forwarded native output_config/adaptive thinking raw, so
a pre-4.6 model rejected it with "This model does not support the effort parameter"
and the request failed. Claude Code already gets drop_params auto-set, so its
requests now succeed.
* test(anthropic): gate undersized-max_tokens thinking drop on drop_params; add edge tests
Addresses review feedback on the max_tokens-too-small branch. Previously a
thinking-capable model whose max_tokens could not fit the minimum thinking budget
had thinking silently dropped regardless of drop_params, while a residual
output_config field in the same call still raised when drop_params was off. Gate
both consistently on drop_params: raise a clear error (naming max_tokens for the
undersized case) when drop_params is off, drop otherwise. Claude Code gets
drop_params auto-set, so it still succeeds.
Adds tests for the undersized-max_tokens raise, the residual output_config raise,
and the no-adaptive-interface passthrough on a non-adaptive model.
* fix(anthropic): make adaptive-effort translation silent to avoid breaking provider strip contracts
The previous raise-when-not-drop_params behavior broke existing bedrock and vertex
messages tests: those providers already silently strip unsupported output_config
for pre-4.6 models (issue #22797) with no drop_params required, and the shared
parent transform raising pre-empted that. It also conflicted with the goal of
keeping requests working rather than failing them.
Make the reshape silent: translate effort to legacy thinking for thinking-capable
models, drop thinking for non-reasoning models, and remove only the consumed effort
key from output_config, leaving any residual (e.g. format) for provider subclasses
(bedrock/vertex) to handle. No raise, no drop_params gating. This also resolves the
review note about inconsistent drop_params handling by making every path uniform.
Updates the tests to assert the silent behavior and residual output_config
preservation.
* fix(anthropic): handle output_config-capable but non-adaptive models (Opus 4.5)
Greptile caught a real bug: the early-return guard treated supports_output_config
as equivalent to supporting adaptive thinking. Claude Opus 4.5 advertises
supports_output_config (it accepts output_config.effort) but is not adaptive, so it
rejects thinking:{type:adaptive} with "adaptive thinking is not supported on this
model". The guard early-returned for Opus 4.5 and forwarded the adaptive thinking
block raw, reproducing the exact failure the fix is meant to prevent.
thinking:{type:adaptive} and output_config.effort are independent capabilities.
Only early-return for adaptive-thinking models. For a model that supports
output_config.effort but is not adaptive, keep the native effort and drop only the
unsupported adaptive thinking block. Verified live against Opus 4.5: the Claude Code
payload now returns 200 instead of 400.
Adds regression tests for Opus 4.5 with and without adaptive thinking.
* fix(anthropic): translate adaptive thinking for effort-capable pre-4.6 models
Claude Opus 4.5 advertises supports_output_config but not adaptive thinking,
so the early-return guard forwarded thinking.type=adaptive raw and Anthropic
rejected it. The guard now only skips true adaptive models; effort-only
requests on effort-capable models still pass through untouched. The
_map_reasoning_effort call is wrapped to surface unrecognized effort values
as a clean 400, matching _translate_reasoning_effort_to_anthropic
* fix(anthropic): fall back to legacy thinking when effort level unsupported
Opus 4.5 accepts output_config.effort but only low/medium/high; Claude Code
defaults to xhigh on newer models, so preserving that level raw gets rejected
by Anthropic. Gate the native-effort passthrough on _validate_effort_for_model
and fall through to the budget translation for unsupported levels
* fix(anthropic): keep effort-only requests untouched for provider normalization
The xhigh fall-through consumed effort-only requests on effort-capable
models, breaking bedrock invoke's own normalization which clamps xhigh to
the model's ceiling after the base transform runs
(test_bedrock_messages_normalizes_output_config_effort_for_opus). Restrict
the fall-through to requests that carry adaptive thinking; effort-only
requests pass through so provider subclasses keep owning level clamping
---------
Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
|
||
|
Mateo Wang
|
4baf326a39
|
fix(anthropic): strip @version suffix in _model_map_lookup_candidates (#32833)
vertex_ai/claude-opus-4-8@default (and sibling @default models) were misclassified as non-adaptive because _model_map_lookup_candidates only stripped provider prefixes but never the @<suffix> portion. The lookup produced candidates like ["vertex_ai/claude-opus-4-8@default", "claude-opus-4-8@default"], neither of which exists in model_cost, so _is_adaptive_thinking_model returned False. LiteLLM then sent thinking.type=enabled to a @default Vertex AI endpoint that requires thinking.type=adaptive, resulting in a 400. _strip_version_suffix now removes @<suffix> from each candidate, adding the bare model name (e.g. "claude-opus-4-8") to the lookup chain. Also adds supports_adaptive_thinking: true to the three @default model_cost entries that were missing it as belt-and-suspenders. Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|
Mateo Wang
|
9076c33347
|
fix(batches): price anthropic passthrough message batches correctly in batch cost job (#32307)
* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
|
||
|
Mateo Wang
|
7e43b3fac7
|
fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop (#32159)
* fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop * fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors The bytes branch of _is_message_stop_chunk used a plain substring match, so a content_block_delta whose partial_json contained the literal text message_stop would look like a real terminal event and suppress the synthetic incomplete-stream error. Match the SSE event header line instead. Also treat a provider-emitted error event as terminal so a stream that ends with an upstream error is not followed by a second, contradictory synthetic incomplete-stream error. * test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
yucheng-berri
|
07b9ea8c3b
|
fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool (#32093)
* fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool The advisor_20260301 interceptor honored a caller-supplied api_base once allow_client_side_credentials was enabled, even without a caller-supplied api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials plus the conversation history got sent to a caller-chosen destination _resolve_advisor_credentials() now only honors api_base alongside a non-empty caller-supplied api_key, requires the https scheme, and validates api_base via validate_url() before use, mirroring check_complete_credentials in auth_utils.py. https is required because validate_url only DNS-pins the connection for http; for https with TLS verification on it returns the URL unchanged and relies on certificate validation to block DNS rebinding * fix(anthropic): also reject advisor api_base when ssl_verify is disabled validate_url only DNS-pins the connection for http, or for https with litellm.ssl_verify disabled; the previous https-only check missed the ssl_verify=False case, where validate_url's rewritten URL was still being discarded, per Greptile's review of this PR. Reject api_base outright when ssl_verify is False so the discarded rewrite can no longer matter |
||
|
Mateo Wang
|
2e1d8d2928
|
fix(anthropic): keep context_management working when drop_params is enabled (#32020)
* fix(anthropic): keep context_management working when drop_params is enabled drop_params (proxy-wide or per-request) silently disabled the in-gateway context_management polyfill on the /v1/messages -> chat completions adapter path, even though context_management is a LiteLLM-supported param (native on Anthropic, polyfilled elsewhere). Gate the polyfill on an explicit additional_drop_params: ["context_management"] opt-out instead, which also makes that escape hatch actually work on the adapter path. * test(anthropic): cover sync adapter polyfill gate for global drop_params and additional_drop_params |
||
|
Mateo Wang
|
6d828e5759
|
feat(messages): passthrough /v1/messages to native endpoints via supported_endpoints (#31685)
* feat(messages): passthrough /v1/messages to native endpoints via supported_endpoints
The unified /v1/messages proxy endpoint always translated inbound Anthropic
requests down to /v1/chat/completions (or the Responses API for openai) when the
deployment's provider lacked a native Anthropic-messages config, dropping
Anthropic-only features like cache_control and thinking. Some customers run
OpenAI-compatible servers (self-hosted vLLM, DeepSeek's Anthropic endpoint, etc.)
that also natively expose /v1/messages and want the raw Anthropic payload
forwarded untranslated, while keeping provider openai so /v1/chat/completions to
the same deployment stays native.
Opt in per deployment via model_info.supported_endpoints containing
/v1/messages. When present, the gate routes to a generic, provider-agnostic
OpenAILikeAnthropicMessagesConfig that POSTs the Anthropic payload to
{api_base}/v1/messages with Bearer auth, instead of translating. Default
behavior is unchanged. Generalizes and supersedes the hosted_vllm-only,
env-var-toggled PR #28745.
* fix(messages): preserve standard-cased caller headers in native passthrough
The OpenAI-like Anthropic passthrough config only checked for lowercase header
names before injecting Bearer auth, anthropic-version, and content-type
defaults. A caller sending standard-cased Authorization, Anthropic-Version, or
Content-Type was treated as missing those headers, so LiteLLM added duplicate
lowercase variants and overwrote the caller's credential/version at the HTTP
layer. Header presence is now checked case-insensitively and the merge no longer
mutates the caller dict.
Also moves the feature docs out of the main repo (docs live in litellm-docs).
* fix(openai_like/messages): delegate to parent transform and inject anthropic-beta headers
The passthrough config bypassed the parent transform and skipped header beta injection. Both gaps cause native /v1/messages features (context management, advisor tool, fast mode, structured outputs, reasoning_effort, advisor stripping) to silently degrade on opted-in deployments. Reuse the parent's pipeline and call _update_headers_with_anthropic_beta after merging defaults
* fix: normalize anthropic-beta header key case before beta injection
* style: collapse anthropic-beta header normalization to single line
ruff format --check requires the comprehension on one line (it fits within
the 120 char limit); fixes the lint job failure on the bugbot autofix commit
* fix(messages): forward anthropic-beta to native passthrough upstream
The shared anthropic_messages HTTP handler ran update_headers_with_filtered_beta
with the deployment's custom_llm_provider after validate. For the native
/v1/messages passthrough that provider is openai, which has no beta-header
mapping, so every anthropic-beta value (caller-supplied or feature-derived for
speed/context_management/etc.) was stripped to empty before the upstream
request, breaking beta passthrough to the Anthropic-compatible endpoint.
Beta filtering only makes sense on cross-provider translation paths where the
upstream cannot understand Anthropic betas. Gate it on a new
should_filter_anthropic_beta_headers() that defaults to True (bedrock, vertex_ai,
native anthropic unchanged) and is overridden to False by
OpenAILikeAnthropicMessagesConfig, whose upstream is a native Anthropic endpoint,
so betas pass through verbatim.
* chore: remove accidentally committed local QA logs and config
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
|
||
|
Mateo Wang
|
26ee5dd597
|
fix(passthrough): drop top-level additional_drop_params on /v1/messages (#31645)
* fix(passthrough): drop top-level additional_drop_params on /v1/messages On the Anthropic Messages pass-through path, additional_drop_params only stripped nested dotted paths, so plain top-level keys like `thinking` and `context_management` were forwarded to the provider. Bedrock rejects these with "Extra inputs are not permitted", returning a 400 to Claude App/CLI even when the user configured `additional_drop_params: ["thinking"]`. delete_nested_value already handles plain top-level fields, so route every drop param through it and remove the nested-only filter. Fixes #25931. * fix(passthrough): drop thinking for bedrock inference-profile ARNs on /v1/messages Opaque Bedrock Application Inference Profile ARNs contain neither "anthropic" nor "claude", so is_anthropic_claude_model returned False and the thinking param was rewritten to reasoning_effort before additional_drop_params ran. That made additional_drop_params: ["thinking"] a no-op for the converse-ARN form, and the Bedrock Converse transform re-expanded reasoning_effort back into additionalModelRequestFields.thinking, so the request 400'd. Extend the thinking-translation gates to also accept bedrock ARNs via the existing is_bedrock_arn_model helper, mirroring the cache_control path, so thinking is preserved as thinking and additional_drop_params can drop it. |
||
|
Sameer Kankute
|
2cf565ae28
|
test(batches): add 1:1 test file scaffold for batches component paths (#30529)
* test(batches): add 1:1 test file scaffold for batches component paths Co-authored-by: Cursor <cursoragent@cursor.com> * Add harness test for create batch endpoint * Add retrieve endpoint harness tests * Add list endpoint harness tests * Add cancel endpoint harness tests * Add cancel endpoint harness tests * Add test for litellm/batches/main.py * Add test for litellm/tests/test_litellm/batches/test_batch_utils.py * Add handler and transformation tests for all providers * Fix: run batches tests in cicd * fix(tests): remove azure/__init__.py that shadowed azure namespace package Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to insert tests/test_litellm/llms/ into sys.path[0], making our empty azure/ dir shadow the real azure-identity namespace package. Any test that patched azure.identity.* would then fail with AttributeError. * style(tests): apply ruff format to test_batch_utils.py Base migrated the formatter from black to ruff format (#31317); reformat the batches scaffold test file to match. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
b76a858826
|
feat: declarative fallback generalizations for unknown models (#29718)
* feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match |
||
|
Mateo Wang
|
64d8d7f8cb
|
fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke (#31364)
* fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke
* style(bedrock): use builtin generics in new Invoke helpers to clear UP006 gate
* fix(bedrock): honor explicit thinking budget_tokens=0 in clear_thinking conversion
The clear_thinking_20251015 -> adaptive conversion resolved the thinking
budget with `thinking.get("budget_tokens") or BEDROCK_MIN_THINKING_BUDGET_TOKENS`,
which treats a caller-supplied `budget_tokens=0` as missing and silently
substitutes the Bedrock minimum. Resolve the budget with an explicit
`is not None` check so an explicit 0 is honored.
* fix(bedrock): gate Fable 5 into clear_thinking adaptive injection on Invoke
_ensure_thinking_for_clear_thinking_context_management returns early when
_supports_extended_thinking_on_bedrock(model) is False, so the adaptive-thinking
injection never runs for models absent from that gate. Opus 4.8 slips through on
the incidental "opus-4" substring, but Fable 5 had no matching pattern, so a
clear_thinking_20251015 request on Fable 5 reached Bedrock with an unsupported
context-management edit and no thinking field; the exact 400 this path exists to
prevent. Add the fable-5 patterns to the gate so Fable 5 (mapped ids and unmapped
aliases) gets thinking.type=adaptive + output_config.effort like the other
adaptive models.
Extend the adaptive-injection regression test to cover Fable 5 (a mapped id and
an unmapped alias) so it fails without the gate entry, and add focused coverage
for the budget->effort tiers, the disabled/enabled/adaptive thinking branches,
output_config.effort preservation, and list/dict system-role normalization.
Also normalize the Invoke transformation module and its test to line-length 88
so ruff format --check (CI format-check) passes.
* refactor(anthropic): make supports_adaptive_thinking flag authoritative for thinking detection
Replace the per-version name helpers (_is_claude_4_6/4_7/4_8_model,
_is_claude_fable_5_model) with cost-map-flag-first detection. _is_adaptive_thinking_model
now reads supports_adaptive_thinking from the model cost map and falls back to a single
generalized family-version regex (_claude_version_at_least(model, 4, 6)) only when a model
is unmapped, instead of hard-coding each new Claude release.
Wire supports_adaptive_thinking through ProviderSpecificModelInfo and ModelInfo so the cost
map flag actually surfaces at lookup time. Reroute the Bedrock Invoke extended-thinking gate
and the two anthropic/chat/transformation.py call sites through _is_adaptive_thinking_model.
Known gap left to the fallback_generalizations work (#29718): unmapped Fable 5 aliases have
no parseable minor version, so they defer to the cost map and are not detected until a mapped
entry or a generalization rule exists. Covered by an explicit regression test.
* refactor(anthropic): drop name-based version fallback; resolve adaptive thinking from cost map only
The prior commit kept a regex (_claude_version_at_least) as a fallback when an id
resolved to no cost-map entry. Remove it: _is_adaptive_thinking_model now reads
supports_adaptive_thinking and nothing else, so "which Claude versions think
adaptively" lives entirely in the model cost map, and a new adaptive release is a
JSON edit rather than a Python edit.
To keep the flag authoritative across the id forms the Bedrock Invoke and anthropic
paths actually see, backfill supports_adaptive_thinking=true on every adaptive Claude
entry that was missing it (Opus 4.6/4.7 and Sonnet 4.6 across region/provider aliases)
in both the root and bundled cost maps, and generalize _model_map_lookup_candidates to
normalize an id to its base cost-map key: strip a Bedrock version suffix (-v1:0 fully,
or just the :0 inference-profile minor so the -v1-keyed 4.6 entries resolve), strip a
dated-release suffix (-20260219), and rewrite a dotted family version (4.6 -> 4-6).
This is id normalization feeding the lookup, not capability-by-name.
Tests load the PR-local cost map (the flags are not on main until merge) and cover each
normalization path plus the unmapped-alias deferral to fallback_generalizations (#29718).
* refactor(reasoning_effort): single-source effort<->thinking-budget mappings
Route every reasoning_effort <-> thinking-budget conversion through the DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants so the numbers stay in sync across providers. The five constants are now 2000/5000/10000/20000/40000
Add reasoning_effort_from_thinking_budget() in litellm_core_utils/reasoning_effort_utils.py and route the three OpenAI-style forward maps (anthropic adapters, responses adapters, hosted_vllm) through it. The bedrock invoke and experimental messages adaptive maps now reference the constants directly; the only behavior change is the xhigh threshold moving from 24000 to 20000. Reverse maps and the cross-provider test grid read the same constants
* test(reasoning_effort): lift budget-mode max_tokens above the new high budget
The single-sourced DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET thresholds moved
high from 4096 to 10000. The live reasoning_effort grid sends budget-mode
requests with max_tokens=8192, so reasoning_effort=high now produces
budget_tokens=10000 > max_tokens and every provider returns 'max_tokens must be
greater than thinking.budget_tokens'. Derive a shared BUDGET_MODE_MAX_TOKENS
(2x the high budget) for the spec and the request builder so the ceiling always
clears the largest 200-expected tier. Also resolve the inherited base
test_reasoning_effort assertion off the same high-budget constant instead of the
stale 4096 literal so it tracks the source of truth.
* fix(reasoning_effort): keep effort<->budget thresholds at pre-PR values
The single-sourcing refactor moved the shared effort<->budget thresholds up
(low 1024->2000, medium 2048->5000, high 4096->10000, xhigh 8192->20000,
max 16384->40000). That silently changes the effort->budget direction: a caller
who sets reasoning_effort together with a max_tokens that used to sit above the
old per-tier budget but below the new one now trips the provider's
"max_tokens must be greater than thinking.budget_tokens" 400. It spans every
backend that derives a budget from an effort (Anthropic, Gemini/Vertex,
hosted vLLM), not just Bedrock.
Restore the constants to their pre-PR values while keeping every backend reading
from the shared DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, so the
mapping stays single-sourced without the behavior change. Tests that pinned the
raised thresholds now derive their boundaries from the same constants.
* test(reasoning_effort): derive high effort->budget assertions from the shared constant
The cross-provider translation tests pinned reasoning_effort="high" to a literal
budget_tokens=10000, the raised value. Point them at
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET so they track the single source
instead of a magic number.
* fix(anthropic): resolve adaptive flag for combined dated+versioned Bedrock ids
The model-map candidate normalization applied each suffix strip independently to
the original id, so the real Bedrock shape "<base>-<YYYYMMDD>-v1:0" never reduced
to its base cost-map key: stripping the version left the date, and the
dated-suffix regex is anchored to the end so it could not fire while the version
was still present. An adaptive Claude model invoked by its full dated+versioned
id (e.g. us.anthropic.claude-sonnet-4-6-20251101-v1:0) therefore resolved to
supports_adaptive_thinking=null and was treated as non-adaptive, reaching Bedrock
with the rejected thinking.type=enabled shape, the exact 400 this path prevents.
Add a composed normalization that rewrites the dotted family version, then peels
the -vN:rev version suffix, then the -YYYYMMDD dated suffix, so the combined form
resolves to its base key. Regression tests pin the combined suffix on sonnet-4-6
and opus-4-8 across provider/region prefixes.
* fix(reasoning_effort): align budget<->effort tests with reverted constants and format common_utils
The constant revert restored the effort<->budget thresholds to their pre-PR
values (1024/2048/4096/8192/16384) and single-sourced the reverse
budget->effort ladder through reasoning_effort_from_thinking_budget, but
several tests still pinned the briefly-raised values and the old hardcoded
reverse buckets, so the "All Other Providers" shard failed
Derive the anthropic chat effort->budget assertions from the shared
DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, and update the
experimental pass-through and responses adapter expectations to the
single-sourced reverse ladder (budget 1024 -> low, 5000 -> high)
Also run ruff format --line-length 88 over anthropic/common_utils.py so the
CI format-check, which checks the whole changed file, passes
|
||
|
Sameer Kankute
|
133da06aa3
|
chore: litellm oss staging (#31185)
* fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped
The Total Tokens Over Time and Total Requests Over Time AreaCharts on the
Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow
once totals pass the hundred-million mark — leading digits of labels like
"100.00M" / "4500.00M" got clipped against the chart edge. The requests
chart was worse: it formatted with toLocaleString(), so billion-scale
request counts produced "1,000,000,000" (13 chars) and overflowed
immediately.
Fix in two places so neither alone has to carry the whole margin:
- activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and
switch the requests chart to the shared valueFormatter so it uses the
same compact k/M/B suffixes as the tokens chart.
- value_formatters.tsx: add a >= 1e9 branch to valueFormatter /
valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping
every formatted label at most 7 chars.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* docs(readme): add Deploy on AWS/GCP with Terraform section
Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.
Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): add 1-click deploy buttons for AWS + GCP
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.
AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): move AWS + GCP deploy buttons next to Render button
* docs(readme): unify deploy button sizes and badge styles
* docs(readme): bump deploy button height to 48 to match Render/Railway
* docs(readme): bump AWS/GCP badge height to compensate for SVG padding
* docs(readme): bump AWS/GCP badge height to 72
* docs(readme): bump AWS/GCP badge height to 84
* fix(readme): make deploy buttons same height (48px)
https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc
* docs(readme): flag GCP project ID substitution in image_registry
* docs(readme): equalize deploy button heights and fix Cloud Shell button font
GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.
Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
* docs(readme): collapse Railway deploy anchor to a single line
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
* Add Claude Fable 5 cost map entries as a data-only hotfix
Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano
Three bugs in model_prices_and_context_window.json:
1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens
were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K
max output, but the values were set as max_input=128000,
max_tokens=272000. This caused token limit errors when sending
prompts over 128K tokens to GPT-5 Pro.
2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was
272000, but GPT-5.4 Mini shares the same 1,050,000 token context
window as GPT-5.4. This was inconsistent with the azure/ variants
which already correctly had 1,050,000.
3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini,
max_input_tokens was 272000 instead of 1,050,000.
Source: OpenAI model documentation and contextwindows.dev which
aggregates official context window sizes.
Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini
should be 400K; their 272K values are correct per OpenAI docs)
* fix: also correct max_output_tokens for gpt-5-pro (272000→128000)
Per reviewer feedback, max_output_tokens was left at 272000 while
max_tokens was corrected to 128000, causing an internal inconsistency.
Both should be 128000 per OpenAI docs.
* fix(cost): price gpt-image generated output tokens as image tokens (#31147)
The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return
usage with no output token breakdown — litellm's `ImageUsage` has no
`output_tokens_details` field — so generated-image OUTPUT tokens were priced at
the text rate (`output_cost_per_token`) instead of the image rate
(`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x
undercount on the dominant cost component (image output is ~74% of spend). This
also affects azure gpt-image, which shares this calculator.
The OpenAI gpt-image cost calculator re-implemented usage handling instead of
reusing `calculate_image_response_cost_from_usage`, the shared helper that
azure_ai/gemini/vertex_ai already use. That helper classifies generated output
tokens as image tokens when the provider does not itemize output, and splits
text/image when it does.
Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage`
(pre-transformed chat Usage objects are still costed directly). Adds a regression
test for the no-breakdown ImageUsage case (gpt-image-2).
* fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098)
A bare application-inference-profile ARN passed as bedrock/arn:... fell
through to the invoke route, which cannot derive a provider from the
opaque profile id and raised 'Unknown provider=None'. The converse route
needs no provider, so detect these ARNs in get_bedrock_route and route
them to converse, matching the behavior of the already-documented
bedrock/converse/arn:... workaround.
Explicit invoke/ prefixes still win, and they remain a dead end for these
ARNs by design (no provider derivable). System-defined inference-profile
ARNs that embed a known model, and other opaque ARN types
(provisioned-model, imported-model, custom-model-deployment) that are
frequently invoke-only, are deliberately left on their current routes;
tests guard both boundaries.
* fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060)
_add_tool_choice_required_message appended the "select a tool" prompt to
the caller's messages list in place, so transform_request corrupted the
caller's conversation history and appended a duplicate prompt on every
retry. Build and return a new list instead so the call stays idempotent.
Adds a regression test asserting the input messages list is unchanged
across repeated transform_request calls.
Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com>
* fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996)
gpt-4o-transcribe and compatible ASR backends return a diarized_json
response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8.
TranscriptionUsageDurationObject typed seconds as int, so parsing the
response raised a pydantic ValidationError (int_from_float). That error
surfaces as an APIConnectionError which the router treats as retryable, so
it keeps re-calling the upstream (200 every time) until the upstream
rate-limits and returns 429 to the caller.
OpenAI specs this field as a float (see openai SDK UsageDuration.seconds),
so widen seconds to float. With the parse succeeding there is no exception
left to retry, which removes the loop.
Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com>
* fix(deepseek): drop non-function tools before chat completions call (#30910)
* fix(deepseek): drop non-function tools before chat completions call
DeepSeek's /chat/completions only accepts tools of type "function".
Requests bridged from /v1/responses can carry responses-API-native tool
types, for example a Codex CLI tool typed "namespace", which DeepSeek
rejects with "unknown variant 'namespace', expected 'function'" so the
whole request fails (issue #30722).
Filter unsupported tool types in the DeepSeek request transform so the
function tools still go through; when nothing callable remains, also drop
the now-dangling tool_choice and parallel_tool_calls
Fixes #30722
* test(deepseek): cover async tool filtering and document tool_choice assumption
Add an async_transform_request regression test so the sync and async tool
filtering paths cannot silently diverge, and document in _drop_unsupported_tools
that only non-function tools are dropped, so a function-named tool_choice always
references a surviving tool
* feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840)
* feat(ui): surface team budget on key overview when key has no own budget (#30801)
* feat(ui): surface team budget on key overview when key has no own budget
* fix(ui): replace IIFE with derived variable and use find() for team budget display
* fix(anthropic): emit replayable streaming thinking blocks (#31022)
* feat(proxy): read cold-storage prompts back in the logs detail view (#30364)
* feat(proxy): read cold-storage prompts back in the logs detail view
When a deployment offloads prompts and responses to cold storage instead of
Postgres, the spend-log row holds only "{}" placeholders plus a
metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed
nothing. The detail endpoint only read the placeholder columns and never
fetched the object back.
Resolve the payload per row based on actual content, not a config flag: if
Postgres has content, return it; otherwise read the exact stored object key and
fetch from the configured cold storage backend through ColdStorageHandler.
Reading the persisted key is a single GET. The key embeds a microsecond
timestamp that cannot be reconstructed from the millisecond-precision startTime
column, and listing the day's prefix to match on request_id would be too
expensive for this per-open path.
Also teach the detail drawer's pretty-view parser to accept a bare messages
array. The cold storage payload carries the prompt as a top-level messages list
with no proxy_server_request, so without this the output rendered while the
input stayed blank.
ColdStorageHandler gains an optional injected logger so the resolver can be unit
tested without monkeypatching. Postgres-stored prompts are unaffected: the fast
path returns the existing columns and the request-body object still renders the
same way.
* Update litellm/proxy/spend_tracking/spend_management_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure
Add unit tests for ColdStorageHandler (injected logger, graceful None when no
logger is configured, and resolution of a configured logger from the callback
registry) and a regression test asserting a cold storage backend exception
degrades to the Postgres values instead of surfacing a 500.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068)
* fix(mavvrik): advance metricsMarker after upload + fix scheduler startup
Two bugs fixed:
1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a
successful GCS upload, so metricsMarker stayed at 0 and every daily run
re-exported the same dates in an infinite catch-up loop.
Fix: add _update_metrics_marker(date_epoch) called at the end of deliver()
after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS
file is already committed). A 410 raises consistent with the rest of the
destination.
2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call
has triggered lazy instantiation of MavvrikFocusLogger, so it found no
logger instance and silently skipped registering the daily export job.
Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call
_init_custom_logger_compatible_class to force instantiation before
the APScheduler job is registered.
* fix(mavvrik): catch up from earliest window when metricsMarker=0
When the connector is freshly registered, metricsMarker=0 parses to None.
The catch-up block was guarded by `if last_ingested and ...` which skipped
it entirely for None, so only yesterday was exported instead of the full
_MAX_CATCHUP_DAYS window.
Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup).
The existing > 7 day warning only fires for non-None markers that are old.
* fix(mavvrik): use now as end_time for yesterday's export window
LiteLLM_DailyUserSpend rows for a given date get their updated_at
bumped by the spend flush job throughout the next morning. The core
database query filters on updated_at, so capping end_time at midnight
(yesterday + 1 day) missed any spend rows flushed after midnight.
Fix: pass now (cron fire time) as end_time for the daily "yesterday"
window so all fully-settled rows are captured regardless of when the
flush job ran.
Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per
row in the exported FOCUS CSV.
* fix(mavvrik): also use now as end_time for catch-up windows
* fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class
Calling it with only logging_integration raised TypeError at proxy startup
because internal_usage_cache and llm_router have no defaults. Also fix test
name to reflect the actual status code (5xx not 4xx) used in the mock.
* ci: retrigger CI run
* feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757)
* Add optional `instruction` passthrough to the rerank API
vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction`
field (folded into the model's chat_template_kwargs and consumed by the
chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently
dropped it: RerankRequest / OptionalRerankParams had no such field, so the
outgoing body was rebuilt without it.
Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(),
get_optional_rerank_params, and the hosted_vllm transformation into the
request body, only when non-None. When callers omit it, model_dump(exclude_none)
drops the field and the outgoing request is byte-for-byte unchanged — fully
backward-compatible. (DeepInfra already forwards `instruction` via
non_default_params; this formalizes the field in the shared types.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: thread `instruction` as a typed param + cover rerank_utils
Per PR review (greptile P2 + codecov):
- Make `instruction` a typed, named argument on the rerank provider interface
instead of recovering it from the opaque `non_default_params` blob. Adds
`instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params`
and every provider override, and forwards it explicitly from
`get_optional_rerank_params`. hosted_vllm now reads the named param directly.
It is still also surfaced in `non_default_params` so providers that read it
there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction`
as a named param rather than leaving it in **kwargs.
- Add get_optional_rerank_params unit tests (present + absent) to cover the
previously-uncovered threading line flagged by codecov.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scan rerank `instruction` through request guardrails
The rerank guardrail translation (CohereRerankHandler.process_input_messages)
only scanned `query`, so the newly added `instruction` field reached the
backend model unscanned. Since instruction-aware rerankers (hosted vLLM /
Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller
could place content there to bypass configured rerank request guardrails.
Generalize the handler to scan every user-controlled text field (`query` and
`instruction`) in one apply_guardrail call and write each sanitized value back
by index. Query-only requests are unchanged (single-element list at index 0);
non-string fields are left untouched. Adds tests covering instruction
scanning, PII masking write-back, and the non-string case.
Addresses the Veria AI security review on PR #30757.
* test: narrow Optional results before len() to satisfy basedpyright budget
The lint gate (basedpyright delta-vs-base budget) flagged one new
reportArgumentType: len(result.results) where results is
List[RerankResponseResult] | None. Assert results is not None first to
narrow the type before len()/indexing.
* fix: read rerank `instruction` from kwargs to satisfy basedpyright budget
The basedpyright delta-vs-base gate flagged one new reportArgumentType: the
Router forwards rerank calls via an untyped `**kwargs` unpack
(`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a
typed named param on the public `rerank`/`arerank` entrypoints made pyright
check that key against `str | None`, adding an error at router.py with no real
safety gain. Read `instruction` from kwargs in `rerank` instead.
It remains fully typed where it matters - threaded as a typed argument through
`get_optional_rerank_params` and each provider's `map_cohere_rerank_params`
(the original Greptile P2 ask). Whole-repo reportArgumentType is back to the
base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(github_copilot): synthesize empty choices at the provider seam (#30929)
Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with
choices=[], either carrying Anthropic-native content blocks or, for the
max_tokens=1 probe Claude Code sends, no content at all. github_copilot
is dispatched through the OpenAI SDK handler, which calls
convert_to_model_response_object directly and never invokes
GithubCopilotConfig.transform_response, so the empty-choices guard there
surfaced as a 500
Instead of synthesizing choices inside the shared
convert_to_model_response_object (which would silently turn empty choices
into a fabricated success for every provider), add a no-op
transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig
overrides it to synthesize choices from Anthropic-native content, reusing
its existing parsing, and the OpenAI SDK handler routes its parsed
response through the hook before generic conversion. The core utility
keeps treating empty choices as an error for all other providers
Fixes: https://github.com/BerriAI/litellm/issues/30927
Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com>
* fix(router): stop fallback lookups from mutating the router fallbacks config (#30624)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in
|