* fix(guardrails): skip streaming guardrail rounds that re-scan cleared output
Streaming guardrails scanned the finished answer twice at end of stream
whenever the chunk count landed on a multiple of the sampling rate, ran
sampled rounds whose payload was identical to the previous one, and on
/v1/messages could scan an empty text before the first content chunk.
Every redundant round is a paid guardrail provider call.
Each endpoint handler now exposes a scan key describing what a round
would hand to apply_guardrail (the text so far, plus tool calls once the
stream has ended), and the unified streaming hook skips a sampled or
end-of-stream round whose key equals the last scanned one or carries
nothing to scan yet. Rounds that carry tool calls are never skipped.
* test(guardrails): expect one end-of-stream scan when the terminal chunk is sampled
Update sampled cadence expectations and use tuple-backed scan state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_image_sources had no test asserting what it extracts. The existing image tests
live on the Bedrock side and all use base64 without a media_type, which is the one
path the fix left unchanged, so both behaviors it does change went unverified: the
url shape reaching the guardrail at all, and base64 arriving as a data URI.
Against the pre-fix extractor the url case sees [] and the media_type case sees
['AAAA'] instead of ['data:image/png;base64,AAAA'].
The remaining three assert behavior the fix deliberately preserves -- bare base64
passed through, a file source yielding nothing, a malformed source dropped rather
than handed on for a consumer to choke on.
Each message carries a text block because a message with no text never reaches the
guardrail, which would make every source shape look equally dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf(streaming): add shared JSONFragmentAccumulator for Vertex and Anthropic
Vertex's handle_accumulated_json_chunk and Anthropic's
_handle_accumulated_json_chunk each independently accumulated SSE fragments
into a JSON envelope with self.accumulated_json += fragment. Because the
attribute holds a live reference, CPython copies the whole prior buffer on
every fragment, making buffer assembly O(n^2) in total payload size.
Anthropic additionally had no completeness heuristic at all and called
json.loads on the whole buffer after every fragment, and could wedge forever
on two concatenated envelopes.
Add JSONFragmentAccumulator in litellm_core_utils/: fragments append to a
list in O(1), a could_close_json heuristic lets callers skip the join+parse
entirely until a value could plausibly be complete, and pop_next_value peels
one JSON value off the front of the buffer at a time using
json.JSONDecoder().raw_decode, keeping any unconsumed remainder instead of
failing on concatenated values. Migrate both providers onto it; Anthropic's
__next__/__anext__ end-of-stream handlers now delegate to
_handle_accumulated_json_chunk(is_final=True) instead of duplicating the
parse-and-reset logic inline.
Fixes#31861.
* test(streaming): close diff-coverage gaps in JSONFragmentAccumulator migration
Codecov flagged 11 uncovered lines in the migration: the accumulated_json
setter, __next__/__anext__'s end-of-stream drain branches in both providers,
and the pop_next_value "not found" path when a buffer's newest fragment ends
in "}" but is genuinely incomplete (an inner object closed, the outer one
didn't). Add targeted tests for each.
* fix(streaming): make JSONFragmentAccumulator's completeness heuristic O(1)
could_close_json rescanned every preceding blank fragment on each call, so a
hostile upstream that sends malformed JSON (never closing) followed by many
blank keepalive fragments could drive that scan, and the join+parse it
gates, to O(n^2) total. Track the last non-blank fragment's trailing byte
incrementally in append/pop_next_value/set instead of rescanning the buffer.
Reported by automated review on PR #36610.
* fix(streaming): make JSONFragmentAccumulator.pop_next_value O(1) per call
pop_next_value previously rebuilt the full remaining string and sliced a
new remainder on every call, so draining N concatenated JSON values
already sitting in one buffer cost O(n^2) total. Replace the rebuild-and-
slice with a materialize-once cursor: pending fragments are joined into
the buffer only when new ones have arrived since the last pop, and
consumed values are dropped by advancing an offset instead of copying
the remaining string.
* test(streaming): make JSONFragmentAccumulator drain regression test CI-stable
The 80k-value drain test used an absolute ms budget that flaked on a
busier CI runner (233.5ms vs a 150ms budget calibrated on a quiet
machine). Replace it with a doubling-ratio check: draining twice as many
concatenated values should take roughly 2x as long for O(n), not
~4x for O(n^2), and that ratio holds regardless of machine speed.
* test(streaming): suppress TQ002 on the append-laziness spy test
A test-quality gate (TQ002: don't assert only that a mock was called)
landed upstream since this branch's last rebase and now flags
test_append_never_calls_raw_decode. The test verifies append() defers
all decoding to pop_next_value, which has no caller-observable proxy
other than spying on the stdlib call it must avoid making.
* test(vertex): spy on raw_decode instead of json.loads in accumulator regression tests
Post-migration to the shared JSONFragmentAccumulator, Vertex's decode
path goes through json.JSONDecoder.raw_decode, not json.loads. The two
O(n^2)/partial-fragment regression tests still patched json.loads, which
that path never calls, so both passed unconditionally regardless of
whether the underlying implementation regressed. Verified by simulating
an eager-reparse regression: both tests now fail against it and pass
against the correct implementation.
* fix(streaming): widen JSONFragmentAccumulator's whitespace skip to match str.strip()
pop_next_value's whitespace skip only matched json.decoder.WHITESPACE's
ASCII set, narrower than str.strip() (Unicode-aware) which the O(1)-cursor
rewrite replaced. A non-ASCII separator like U+00A0 between two
concatenated JSON values on one SSE line made raw_decode fail on it, and
the buffer never advanced past that byte again, permanently stranding
everything after it for the rest of the stream. Use str.isspace() to
match str.strip()'s tolerance.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
* test: enforce PT012 so a pytest.raises block cannot hide dead assertions
`with pytest.raises(...)` stops at the first statement that raises. Anything
sequenced after it inside the block never runs, so an assertion written there is
never checked and the test still reports green.
Two sites were doing exactly that, and both assertions turned out to be wrong
once they started running. tests/llm_translation/test_prompt_factory.py asserted
the bedrock rejection names "requires at least one non-system message", which
holds. tests/proxy_unit_tests/test_proxy_server.py asserted the prisma startup
failure mentions "httpx.ConnectError", which never appears: the failure is an
httpx.ConnectError whose message is "All connection attempts failed", so that
test now asserts the type. Its DATABASE_URL override moves to monkeypatch, since
the old restore sat below the assertion and leaked the invalid URL into every
later DB test the moment the assertion started being able to fail.
The remaining 72 sites are rewritten without changing what they exercise: setup
that cannot raise moves above the block, a nested `patch` moves outside it, and
bodies with real control flow (a stream drain, an if/else on sync_mode, a
retry loop) move into a local closure the block calls.
Fixing PT012 unmasked two B017s, since ruff only reports a blind
pytest.raises(Exception) once the block holds a single statement.
tests/proxy_unit_tests/test_auth_checks.py narrows to the ProxyException
can_key_call_model actually raises. tests/local_testing/test_completion_cost.py
was asserting vertex_ai/medlm-medium has no cost entry, which stopped being true
at some point; that dead first half is gone and the rest of the test, which
checks medlm pricing resolves above zero, now runs instead of being skipped.
* chore(ci): ratchet TQ004 to 768 after the prisma test moved to monkeypatch
Adds a chat_completions route module to litellm-core, mirroring the messages
route, plus Anthropic Messages and Bedrock Converse provider configs. The
per-model `rust: true` opt-in now covers /chat/completions for both providers.
The core accepts an allowlisted subset (text conversations, non-streaming) and
returns CoreError::Unsupported for anything else, so tool calls, multimodal
content and streaming fall back to the Python path transparently.
Resolves LIT-5698
The iterations branch in AnthropicConfig.calculate_usage summed
cache_creation_input_tokens but never aggregated the per-iteration
cache_creation 5m/1h breakdown, leaving cache_creation_token_details
as None. As a result all cache-creation tokens fell back to the flat
5m write rate, underbilling 1h cache writes by up to 2x.
Aggregate the ephemeral_5m/ephemeral_1h split across iterations so 1h
writes are priced at the 1h rate.
Fixes LIT-4868
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Gate the OpenAI handler's tools forwarding behind scan_only_tool_results,
matching the Anthropic handler, so a tool-results-only scan can no longer
evaluate or rewrite trusted function definitions.
When a guardrail returns a replacement structured_messages list, substitute
the returned messages back into the positions their scoped originals came
from instead of installing the scoped list as the whole conversation, so
out-of-scope messages (system prompt, prior turns) survive redaction on
both the OpenAI and Anthropic paths.
Guardrails silently skipped three surfaces on the Anthropic Messages
path, so an agent loop driven by /v1/messages ran unguarded:
- The Anthropic input translation never walked tool_result blocks, so
content returned by a local tool (a curl, a file read, an MCP call)
reached the model unscanned in both the string and list content
shapes, images inside a tool_result included.
- tool_permission only understood ModelResponse, so an Anthropic
non-streaming response or a raw SSE stream carrying tool_use blocks
passed through with no rule ever evaluated.
- ContentFilterGuardrail scanned inputs["texts"] but never
inputs["tool_calls"], so the arguments a model proposes for a tool
call went unchecked.
Tool call arguments are parsed as JSON before filtering so a MASK
action rewrites the value and leaves the payload valid JSON; non-JSON
arguments fall back to scanning the raw string. Denied tool_use blocks
are dropped from the Anthropic content array and replaced with a text
block, and stop_reason resets to end_turn when nothing tool-shaped
survives.
* fix(guardrails): add /v1/messages support for Straiker plugin
- Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py)
- Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs
* fix(guardrails): gate cross-surface message resolution and cover streaming request data
Resolve request messages only for surfaces that have a mapped translation
handler. The unguarded fallback tried every registered handler in turn, which
raised AttributeError out of the guardrail's error handling on list-shaped
`input` bodies, and synthesized a chat message that was never sent for bodies
it happened to parse.
Prepare request data on the mid-stream Anthropic branch as well, matching the
terminal branch and the OpenAI handler, so guardrails that scan before
end-of-stream still receive identity metadata.
Read usage from Anthropic dict responses so non-streaming /v1/messages reports
token counts instead of null.
Add regression coverage for the streaming request data on both the terminal and
mid-stream branches; reverting either now fails.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
* feat(guardrails): add only_scan_new_messages for per-session incremental scanning
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): use fixed TTL constant and revert unrelated test formatting
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): run only_scan_new_messages in the unified apply_guardrail path
The initial wiring lived in BedrockGuardrail.async_pre_call_hook, but the proxy
routes Bedrock through the unified apply_guardrail interface, so the flag had no
effect live. Move incremental selection into apply_guardrail: filter the flat
texts list against per-session scanned hashes, skip the Bedrock call when nothing
is new, and mark hashes only after a successful (non-blocked) scan. Full-context
fallback is preserved when there is no session id, the cache is unavailable, or a
masking guardrail is configured.
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover session-id fallbacks and mark_texts_scanned guards
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* fix(guardrails): fall back to full scan when incremental guardrail masks content, use shared cache
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover generic agent multi-turn incremental scan
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover incremental scan cache resolver fallbacks
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
* test(guardrails): cover flag interactions and /v1/messages incremental scan semantics
* feat(guardrails): make GUARDRAIL_SCANNED_MESSAGES_CACHE_TTL_SECONDS env configurable
* test(guardrails): prove skip_system/skip_tool are enforced upstream of incremental scan
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: Yucheng Zhu <yucheng@berri.ai>
* feat(guardrails): add Compresr guardrail for query-aware context compression
Adds a first-class guardrail that compresses bulky message content (tool
outputs, RAG chunks, search results) through the Compresr API before the
request reaches the LLM, via the apply_guardrail / structured_messages hook
so it covers /chat/completions, /v1/messages, and /v1/responses (the latter
through the texts channel, mirrored only when the replacement is
unambiguous; anything ambiguous is left uncompressed).
Distinct from whole-conversation compressors:
- Query-aware: each message is compressed against the intent that produced
it (a tool output against its originating tool call's name + arguments,
resolved via tool_call_id; otherwise the last user message).
- Recoverable: each compressed message carries a hash marker and the request
gains a compresr_retrieve tool, so the model can pull the original content
back through the agentic loop when the compressed version is not enough.
Originals are cached in-process, scoped to the caller's virtual-key hash
plus the request's litellm_call_id, with a TTL and a per-call byte cap;
recovery is skipped when no caller scope is available so one caller can
never read another's originals. The store is per-process, so multi-worker
deployments need sticky routing (or enable_retrieval=false).
Fail-closed by default (fail_open configurable), SSRF-validated api_base
(alternate IP-literal encodings included), cross-tenant-isolated recovery
store, and upstream errors redacted from client-facing responses. The
outbound client follows redirects and re-resolves DNS per request, so the
api_base host/IP checks are defense-in-depth, not a full SSRF guarantee;
this is documented as a known limitation. Requests where nothing was
actually compressed are returned untouched (same object identity) so
handlers skip the write-back. Auto-discovered via the guardrail_hooks
registry.
* fix(guardrails): cap Compresr recovery store total memory
The recovery store bounded bytes per call and entry count, but had no
aggregate cap: 256 tracked call ids at the 10 MiB per-call default could
retain ~2.5 GiB per worker. A flood of requests with distinct
x-litellm-call-id values and large compressible tool outputs could
exhaust a shared proxy worker.
Add a global byte budget (_MAX_TOTAL_STORE_BYTES, 256 MiB) across all
entries. A running total is maintained on every insert/eviction so the
cap is enforced without re-encoding the whole store on the request path;
oldest entries are evicted once the budget is exceeded, always keeping
the most-recent entry so recovery still works for the request populating
the store. +2 regression tests.
* fix(guardrails): gate and bound Compresr recovery loop
Two hardening fixes to the compresr_retrieve agentic loop:
1. Only run the loop when a retrieve call resolves to recovery state this
guardrail actually created for the request. Previously the gate checked
only that the caller-supplied tool list contained a compresr_retrieve
function and that the model emitted a call, so a caller could define
their own same-named tool and force an extra provider round-trip with
nothing to recover. The plan now returns run_agentic_loop=False when no
requested hash resolves.
2. Bound the follow-up against retrieval amplification: each distinct hash
is expanded at most once (repeats get a short marker) and at most
_MAX_RETRIEVALS_PER_LOOP calls are honored, so prompting the model to
call compresr_retrieve many times with the same marker cannot balloon
the follow-up. _retrieve_original now returns None on miss.
+3 regression tests; two existing security tests updated to assert the
stronger veto behavior (forged/cross-tenant hashes now stop the loop
entirely instead of returning a not-found follow-up).
* fix(guardrails): warn when Compresr recovery is skipped without auth scope
When enable_retrieval is on (the default) but the proxy has no per-key
auth, the request has no caller scope, so recovery is silently disabled:
content is compressed but the compresr_retrieve tool is never injected and
the originals are dropped, with no runtime indication. Emit a one-shot
call-time warning so operators can see recovery is being suppressed and
configure virtual-key auth. +1 regression test.
* style(guardrails): tighten Compresr guardrail comments
Condense the verbose multi-line inline comments and the api_base docstring
to concise form. No behavior change.
* fix(guardrails): keep injected tool on Responses API + bound recovery markers by byte cap
Two fixes for reviewer-flagged defects in the Compresr guardrail:
- Responses API: _merge_tools_after_guardrail iterated only over the
request's original tools, dropping any tool a guardrail appended (the
compresr_retrieve recovery tool) whenever the request already had tools.
Keep the appended tools so recovery works on /v1/responses.
- Recovery markers: markers + originals were built for every compressed
target before the per-call byte cap trimmed the store, so an evicted
original left a marker the model could never retrieve. Attach recovery
only while the store (existing entries under the same key + this call's
originals) stays within the cap, so a shipped marker is always retrievable
-- including on a later turn that reuses the store key.
Adds regression tests for both paths.
* refactor(guardrails): extract _existing_originals to keep apply_guardrail under the complexity gate
The byte-cap fix added a branch to apply_guardrail, tipping it past the
C901 complexity ceiling. Move the store lookup into a small helper; no
behavior change.
* fix(guardrails): harden Compresr SSRF blocklist, re-arm no-scope warning, tolerate odd tool shapes
* fix(guardrails): rerun input guardrails on Compresr retrieval follow-up
* chore: remove unrelated deepkeep files committed by mistake
---------
Co-authored-by: charafkamel <charafkamel@live.com>
Clients that pass thinking={"type": "adaptive"} directly (not via the
reasoning_effort alias) on the /chat/completions interface had it forwarded
unmodified to pre-4.6 Anthropic models, which reject the shape. Mirrors the
translation already applied on the native /v1/messages passthrough (#32867):
translate to legacy thinking={type: enabled, budget_tokens}, capped below
max_tokens, dropping thinking when max_tokens can't fit even the minimum
budget. Hoists the shared budget-capping helper onto AnthropicConfig so both
paths use one implementation.
* feat: declarative fallback generalizations for unknown models
Unknown or newly-released models previously degraded (missed cost lookups,
wrong supports_* flags, broken provider routing) and were patched with one-off
hardcoded regexes scattered across Python. This adds a single data-driven source
of truth: a fallback_generalizations block in model_prices_and_context_window.json
holding ordered, case-insensitive regex rules that map a model name to the
metadata to apply when it has no exact entry.
A new fallback_generalizations module owns the rules and a compiled-regex cache
that is built once and invalidated on reload, so the O(n) scan runs only on a
cache miss. get_llm_provider now routes an otherwise-unknown model via the first
matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and
_matches_claude_model_pattern. _get_model_info_helper falls back to a matching
rule's model_info after the exact lookups miss, so get_model_info and the
supports_* helpers resolve unknown models from the same rule. get_model_cost_map
extracts the block out of the returned map, and the integrity check now counts
real model entries (excluding reserved meta keys) so the new key cannot mask a
genuinely shrunk upstream file.
The top level of the file stays a flat map of models so existing litellm releases
that fetch the live file keep working and keep receiving updates; the block ships
in both the root file and the bundled backup. An anthropic-claude rule reproduces
the old future-claude routing and additionally supplies capability flags and a
context window
https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo
* refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations
Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring
matchers with a single _claude_version_at_least predicate that parses the Claude
family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x
without a code change (the old matchers missed 4.8 entirely) while keeping an explicit
supports_adaptive_thinking flag authoritative when present, so there is one source of
truth. The two direct call sites in the chat transformation now route through
_is_adaptive_thinking_model instead of the deleted matchers.
Also address review feedback on the generalizations module: return a copy of the
matched model_info so a future caller cannot mutate the compiled-rule cache, document
that patterns are matched with re.search and must anchor with ^ and $, and reindent
the fallback_generalizations block to the file's 2-space style in both JSON files.
https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo
* fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse
supports_adaptive_thinking shipped in the model cost map but was never declared
on ModelInfo nor copied during construction, so get_model_info (and the supports_*
factory) silently dropped it for every provider-prefixed or generalized name; only
a bare base entry resolved. Wire it through ModelInfo like the other capability
flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across
providers so the data, not code, declares the capability. The anthropic-claude
fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so
an unmapped future Claude degrades to adaptive thinking without a code change.
Tighten the Claude version parser so an eight-digit date suffix
(claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor
4.20250514. The cost map stays authoritative; the version check is only a fallback
for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to
no mapped entry and so cannot be reached by an exact lookup or the bare-name rule.
https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo
* fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate
Reconcile adaptive-thinking detection after merging litellm_internal_staging.
Keep the cost-map resolver (_supports_model_capability) as the source of truth and
add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for
provider-prefixed ids the cost map cannot resolve (e.g.
bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an
eight-digit date suffix from being misread as a minor version, so the dated Claude
4.0 release stays non-adaptive
Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or
newly released Claude is over-costed rather than billed as free
Drop the module-level global state in fallback_generalizations (PLW0603) in favor of
a small registry object, and switch its annotations plus the new utils helper to
builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling
* refactor(anthropic): drive adaptive-thinking version gate from a declarative rule
Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit
The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor
* refactor(anthropic): dedupe adaptive-thinking rule via declarative extends
The version-gated anthropic-claude-adaptive-thinking rule duplicated the
broad anthropic-claude rule's entire Opus-tier price block because rules do
not merge: first match wins and returns one rule's whole model_info, so the
adaptive rule had to be self-contained.
Add a declarative extends field to fallback_generalizations: a rule names a
parent and inherits its model_info, with its own keys overriding. Inheritance
is resolved once at install time against each rule's raw model_info, so the
adaptive rule now carries only its delta (supports_adaptive_thinking) and
inherits pricing from the broad rule. Runtime matching, provider routing and
gating are unchanged; the broad rule stays anchored and first-match-wins still
holds.
* docs(anthropic): add ignored description key documenting each generalization regex
* fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule
Per review feedback, the base rule no longer carries input/output/cache costs, and the
adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an
unmapped model at a guessed tier reports a confidently-wrong cost without the caller
knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated
number) so a missing price stays visible. The rules still supply provider routing,
context window, and capability flags, so a brand-new Claude can still be called and its
capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests
updated to match
* fix(bedrock): normalize Messages system role and adaptive-thinking for Claude Invoke
* style(bedrock): use builtin generics in new Invoke helpers to clear UP006 gate
* fix(bedrock): honor explicit thinking budget_tokens=0 in clear_thinking conversion
The clear_thinking_20251015 -> adaptive conversion resolved the thinking
budget with `thinking.get("budget_tokens") or BEDROCK_MIN_THINKING_BUDGET_TOKENS`,
which treats a caller-supplied `budget_tokens=0` as missing and silently
substitutes the Bedrock minimum. Resolve the budget with an explicit
`is not None` check so an explicit 0 is honored.
* fix(bedrock): gate Fable 5 into clear_thinking adaptive injection on Invoke
_ensure_thinking_for_clear_thinking_context_management returns early when
_supports_extended_thinking_on_bedrock(model) is False, so the adaptive-thinking
injection never runs for models absent from that gate. Opus 4.8 slips through on
the incidental "opus-4" substring, but Fable 5 had no matching pattern, so a
clear_thinking_20251015 request on Fable 5 reached Bedrock with an unsupported
context-management edit and no thinking field; the exact 400 this path exists to
prevent. Add the fable-5 patterns to the gate so Fable 5 (mapped ids and unmapped
aliases) gets thinking.type=adaptive + output_config.effort like the other
adaptive models.
Extend the adaptive-injection regression test to cover Fable 5 (a mapped id and
an unmapped alias) so it fails without the gate entry, and add focused coverage
for the budget->effort tiers, the disabled/enabled/adaptive thinking branches,
output_config.effort preservation, and list/dict system-role normalization.
Also normalize the Invoke transformation module and its test to line-length 88
so ruff format --check (CI format-check) passes.
* refactor(anthropic): make supports_adaptive_thinking flag authoritative for thinking detection
Replace the per-version name helpers (_is_claude_4_6/4_7/4_8_model,
_is_claude_fable_5_model) with cost-map-flag-first detection. _is_adaptive_thinking_model
now reads supports_adaptive_thinking from the model cost map and falls back to a single
generalized family-version regex (_claude_version_at_least(model, 4, 6)) only when a model
is unmapped, instead of hard-coding each new Claude release.
Wire supports_adaptive_thinking through ProviderSpecificModelInfo and ModelInfo so the cost
map flag actually surfaces at lookup time. Reroute the Bedrock Invoke extended-thinking gate
and the two anthropic/chat/transformation.py call sites through _is_adaptive_thinking_model.
Known gap left to the fallback_generalizations work (#29718): unmapped Fable 5 aliases have
no parseable minor version, so they defer to the cost map and are not detected until a mapped
entry or a generalization rule exists. Covered by an explicit regression test.
* refactor(anthropic): drop name-based version fallback; resolve adaptive thinking from cost map only
The prior commit kept a regex (_claude_version_at_least) as a fallback when an id
resolved to no cost-map entry. Remove it: _is_adaptive_thinking_model now reads
supports_adaptive_thinking and nothing else, so "which Claude versions think
adaptively" lives entirely in the model cost map, and a new adaptive release is a
JSON edit rather than a Python edit.
To keep the flag authoritative across the id forms the Bedrock Invoke and anthropic
paths actually see, backfill supports_adaptive_thinking=true on every adaptive Claude
entry that was missing it (Opus 4.6/4.7 and Sonnet 4.6 across region/provider aliases)
in both the root and bundled cost maps, and generalize _model_map_lookup_candidates to
normalize an id to its base cost-map key: strip a Bedrock version suffix (-v1:0 fully,
or just the :0 inference-profile minor so the -v1-keyed 4.6 entries resolve), strip a
dated-release suffix (-20260219), and rewrite a dotted family version (4.6 -> 4-6).
This is id normalization feeding the lookup, not capability-by-name.
Tests load the PR-local cost map (the flags are not on main until merge) and cover each
normalization path plus the unmapped-alias deferral to fallback_generalizations (#29718).
* refactor(reasoning_effort): single-source effort<->thinking-budget mappings
Route every reasoning_effort <-> thinking-budget conversion through the DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants so the numbers stay in sync across providers. The five constants are now 2000/5000/10000/20000/40000
Add reasoning_effort_from_thinking_budget() in litellm_core_utils/reasoning_effort_utils.py and route the three OpenAI-style forward maps (anthropic adapters, responses adapters, hosted_vllm) through it. The bedrock invoke and experimental messages adaptive maps now reference the constants directly; the only behavior change is the xhigh threshold moving from 24000 to 20000. Reverse maps and the cross-provider test grid read the same constants
* test(reasoning_effort): lift budget-mode max_tokens above the new high budget
The single-sourced DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET thresholds moved
high from 4096 to 10000. The live reasoning_effort grid sends budget-mode
requests with max_tokens=8192, so reasoning_effort=high now produces
budget_tokens=10000 > max_tokens and every provider returns 'max_tokens must be
greater than thinking.budget_tokens'. Derive a shared BUDGET_MODE_MAX_TOKENS
(2x the high budget) for the spec and the request builder so the ceiling always
clears the largest 200-expected tier. Also resolve the inherited base
test_reasoning_effort assertion off the same high-budget constant instead of the
stale 4096 literal so it tracks the source of truth.
* fix(reasoning_effort): keep effort<->budget thresholds at pre-PR values
The single-sourcing refactor moved the shared effort<->budget thresholds up
(low 1024->2000, medium 2048->5000, high 4096->10000, xhigh 8192->20000,
max 16384->40000). That silently changes the effort->budget direction: a caller
who sets reasoning_effort together with a max_tokens that used to sit above the
old per-tier budget but below the new one now trips the provider's
"max_tokens must be greater than thinking.budget_tokens" 400. It spans every
backend that derives a budget from an effort (Anthropic, Gemini/Vertex,
hosted vLLM), not just Bedrock.
Restore the constants to their pre-PR values while keeping every backend reading
from the shared DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, so the
mapping stays single-sourced without the behavior change. Tests that pinned the
raised thresholds now derive their boundaries from the same constants.
* test(reasoning_effort): derive high effort->budget assertions from the shared constant
The cross-provider translation tests pinned reasoning_effort="high" to a literal
budget_tokens=10000, the raised value. Point them at
DEFAULT_REASONING_EFFORT_HIGH_THINKING_BUDGET so they track the single source
instead of a magic number.
* fix(anthropic): resolve adaptive flag for combined dated+versioned Bedrock ids
The model-map candidate normalization applied each suffix strip independently to
the original id, so the real Bedrock shape "<base>-<YYYYMMDD>-v1:0" never reduced
to its base cost-map key: stripping the version left the date, and the
dated-suffix regex is anchored to the end so it could not fire while the version
was still present. An adaptive Claude model invoked by its full dated+versioned
id (e.g. us.anthropic.claude-sonnet-4-6-20251101-v1:0) therefore resolved to
supports_adaptive_thinking=null and was treated as non-adaptive, reaching Bedrock
with the rejected thinking.type=enabled shape, the exact 400 this path prevents.
Add a composed normalization that rewrites the dotted family version, then peels
the -vN:rev version suffix, then the -YYYYMMDD dated suffix, so the combined form
resolves to its base key. Regression tests pin the combined suffix on sonnet-4-6
and opus-4-8 across provider/region prefixes.
* fix(reasoning_effort): align budget<->effort tests with reverted constants and format common_utils
The constant revert restored the effort<->budget thresholds to their pre-PR
values (1024/2048/4096/8192/16384) and single-sourced the reverse
budget->effort ladder through reasoning_effort_from_thinking_budget, but
several tests still pinned the briefly-raised values and the old hardcoded
reverse buckets, so the "All Other Providers" shard failed
Derive the anthropic chat effort->budget assertions from the shared
DEFAULT_REASONING_EFFORT_*_THINKING_BUDGET constants, and update the
experimental pass-through and responses adapter expectations to the
single-sourced reverse ladder (budget 1024 -> low, 5000 -> high)
Also run ruff format --line-length 88 over anthropic/common_utils.py so the
CI format-check, which checks the whole changed file, passes
