* feat(messages): apply modify_params rewrites on rust
* feat(messages): mint anthropic workload identity tokens on rust
* feat(messages): apply prompt caching injection on rust
* feat(messages): serve the easy python-only detectors on rust
* feat(messages): retry rejected thinking signatures on rust
* feat(messages): serve response cache reads on rust
* feat(messages): retry rejected strict tools on rust
* refactor(messages): delete the python path for anthropic messages
Provider anthropic on /v1/messages is now RUST_REQUIRED, so it runs on Rust
without LITELLM_RUST or litellm.rust(True) and never falls back to Python.
A call the rules would send to Python (a MESSAGES_PYTHON_FEATURES feature or a
detector that raises) gets UnsupportedParamsError naming the feature, raised
from one place in litellm/messages/dispatch.py. Other providers keep the
Python fallback. A call whose arguments the handler signature rejects still
gets Python's TypeError.
Deleted Python code that only served anthropic Messages:
- workload identity in AnthropicMessagesConfig.validate/avalidate (subclasses
were never eligible; the Rust host mints the token itself)
- the anthropic Foundry beta-row and affinity-base branch in
async_anthropic_messages_handler, and the "forward" mode of the beta filter
that only that branch used
The proxy MCP tools detector now uses the light tool precheck so a function
tool never imports the gateway handler, since a raising detector now refuses.
* feat(messages): forward message threads on rust
The Rust path for anthropic /v1/messages dropped the thread body field, so
a create or continue turn reached Anthropic without its thread or the
message-threads-2026-08-12 beta. It now matches the Python handler from
#50976:
- the host forwards thread with the other body params
- a non-null thread adds the message-threads beta
- a well-formed continue turn keeps advisor blocks in history and derives
no compaction, per-turn-control or tool-change beta from its messages
- the recorded httpx_response carries the request that was sent, for JSON
and streamed calls, so the proxy's thread ownership hook resends the
create turn's betas on a continue as it does for Python
Upstream error.details already reached the exception message unchanged, so
thread_not_found relays and keeps the deployment out of cooldown on both
paths. Parity cases cover create, continue, streaming, sync, retries,
thread_not_found and the ownership hook round trip.
* feat(messages): serve mock responses on rust
* feat(messages): send caller clients and unsupported tls through python on rust
* feat(messages): run pre-request hooks on rust
* feat(messages): run agentic loop hooks on rust
* feat(messages): run the safeguards classifier on rust
* feat(messages): run registered interceptors on rust
* fix(messages): match python stream cache gate, logging and interceptor order on rust
Store a streamed Messages response in the cache only when Python would: a
message_stop event line, no provider error frame and at least one frame with
output. Leave dict results untouched by mark_rust_response so hidden params
never reach result dicts, logging objects or hook digests. Ask interceptors
whether they handle the call after the cache lookup, not at decode time, so a
cache hit never calls can_handle. Emit the track_llm_api_timing service event
for the provider exchange. Bill and log a stream superseded by a router
fallback as the failure of its error frame, with the partial usage warning.
Hand litellm_logging_obj to deployment pre-call hooks as @client does, so the
autorouter baseline cache sees retried requests.
* fix(messages): forward python-only json values to anthropic as python does
Accept any max_tokens, sampling scalar and messages collection that
Python main accepts, and carry values serde_json cannot hold (ints beyond
64 bits, NaN and infinities, strings with lone surrogates) as validated
raw JSON leaves written back verbatim on the wire, so the upstream body
matches Python's json.dumps
* fix(messages): match python's body reads, error bodies, encoded heads and tls warnings on rust
* fix(messages): log python's plain provider header names on rust streams
* fix(messages): match python's request params, logged params and validation on rust
Read a non-bool stream by Python truthiness and send it as given, log the
caller's stream flag as Python's handler does, raise the raw Python errors
main raises for malformed messages, tools, model_info and hook-set
litellm_params or providers, read the tool custom-field drop from
drops_unsupported_params, log the caller's nested fields that
additional_drop_params strips and the Python values of raw JSON leaves,
keep a missing stop_reason absent, and map provider-config rejections
(missing max_tokens, unknown reasoning effort) as Python's HTTP handler
maps them, with Python's text, effort order, model and provider
* fix(messages): match python's tls curve names, redirects, auth token, header encoding and timeouts on rust
* fix(messages): match python on hooked streams, empty cache writes and mcp cache hits
* fix(messages): join already buffered stream reads and trim the env api key as python does on rust
* fix(messages): deliver stream bytes read before a cut, keep wide reply numbers and follow python's redirect limit on rust
A stream cut by a transport error now hands the caller the bytes already
buffered before raising, as aiohttp does for a caller that reads them before
the event loop sees the cut, so the frames and their usage reach the caller and
the failure log. Provider replies keep integers wider than 64 bits exactly and
read numbers that overflow a float as infinity, through the response, the
response cache and the logged result. The provider client follows up to 20
redirects as httpx does, and running out raises Python's text. A
provider_specific_header value the ascii codec cannot encode fails before the
send like a caller extra header. The deployment success hook gets the stream as
AnthropicMessagesStreamingResponse, as Python's handler hands it.
* feat(messages): serve proxy mcp tools on rust
* test(messages): cover hook, agentic and interceptor edge cases in the parity suite
* fix(messages): keep provider raw-json marked strings, drop non-string tool_result texts and key clamped streams as python does on rust
* fix(messages): match python's api key encoding, malformed api_base and string provider header errors on rust
A header value Python's client cannot encode as ASCII now fails before the send
whichever header carries it, so a caller api_key with a non-ASCII character
raises Python's InternalServerError instead of a raw UnicodeEncodeError after
the reply. A caller api_base without an http(s) scheme, or with whitespace,
control or non-ASCII characters, is sent through Python's client at the URL
Python builds, so its relative-URL and invalid-URL errors match, and a base
urlsplit cannot split raises Python's bare ValueError before the send. A
provider_specific_header of the wrong shape is scoped through Python's
ProviderSpecificHeaderUtils first, so a plain string raises Python's
AttributeError.
* fix(messages): keep integer temperature and top_p digits a float cannot hold and raise python's typeerror for a string max_tokens while capping the thinking budget on rust
temperature and top_p now deserialize through recognized::exact_float, which keeps an integer
where float(n) != n (beyond 2**53) as an unrecognized value with its exact digits, so it is sent
upstream and logged as the integer python forwards, and comparisons still go through scalar.rs
cap_thinking_budget_to_max_tokens compares max_tokens <= 1024 first, so a max_tokens python
cannot order against an int (a string, list or dict) raises TypeError "'<=' not supported between
instances of 'str' and 'int'" in both the reasoning effort translation and the adaptive effort
reshape. rust now returns ErrorDetail::UnorderableTypes there, the bridge raises it as a python
TypeError marked as a late rejection, and map_failure hands it to the caller unmapped
* test(messages): format the non-string tool_result parity helper
* fix(messages): flag and log the call before python's pre-send raises on rust
The bridge now hands settings() and map_failure() whether the call is async.
A provider_specific_header of the wrong shape sets the aanthropic_messages
flag before re-raising, as Python's handler has already flagged the call. A
base urlsplit cannot split also logs the resolved model, provider and optional
params first, as Python raises it only after its logger is updated. The
TypeError raised while capping the thinking budget for a string max_tokens
sets the flag too
* feat(messages): serve the web search short-circuit override on rust through the intercept host call
* fix(messages): give the web search entry kwargs its mutable-ok reason
* test(messages): drop the body hook gap the rust short-circuit closes
* fix(messages): name the partially unknown handler types on the short-circuit host call
Python stores a unary messages response from a background cache write task that runs after the router stamps _hidden_params, so the entry carries model_id and the router headers and a later hit returns x-litellm-cache-key. The native path stored the entry inside the native call, before the router saw the response. The async unary store is now deferred until the route encodes its public response and is handed to Python's own cache write task
* feat(redis): route the router, rate limiters and auth through the native store families
* style(redis): satisfy the python lint gates
Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): give the registry's default usage store a memory-only handle
RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): build stores from native handles and wait for router tasks
Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality
sub-routers now take a RoutingContext instead of the Router, and read the router only through
registry(), settings(), health() and the context's forwarding methods. Router-free helpers move
from router_utils to litellm_core_utils, outside callers of router state switch to entry points,
and the surface allowlist shrinks to the endpoint methods plus the entry points
* feat(rust-bridge): scope catalog rules with a contextvar
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): select native integration routes with scoped rules
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): run the Rust extension suite without LITELLM_RUST
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): use scoped catalog rules in dispatch tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): keep catalog rules a static global and override them once per test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust-bridge): pin Messages and Responses callback contracts on the native path
Each test runs the native route and asserts what the Python path does today, so a
known gap is a strict xfail that names its reason and the exception it fails with
Messages and Responses share one Surface and hook recorder in support/callbacks.py,
and each surface folder lists its own expected counts and gaps.
Body-level hooks from the function body (pre-request hook, websearch short-circuit,
Messages interceptors, planned and legacy agentic loops, streaming deployment hook)
are xfail until native runs them
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust-bridge): give every route folder the same one-concern-per-file layout
Route data, the route Surface and the server fixture move into each route folder, and support/ keeps only
route-neutral helpers. test_inference.py splits into chat_completions/ and responses/, ocr/test_requests.py
splits into requests, failures and settings, and Messages and Responses get streaming and body hook files.
Messages request tests that duplicate inference-messages request.rs are removed, and drop_params read
from litellm.drop_params is now covered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust-bridge): expect the required-route decline error for unsupported Responses requests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): import route data from the route folders in the secrets unit tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): import NoReturn in the OCR dispatch tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): move route-neutral helpers out of support/callbacks.py
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): cover deployment rejection and concurrent callbacks
* test(rust-bridge): assert native serving via the x-litellm-rust marker in the route folder tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(python-bridge): ship a signature base and read the resolved call
NativeCall carries base (positionals by name plus signature defaults) instead
of the fully bound dict. resolved lays kwargs over base, which is what bound
held, so every pre-hook read and every route host keeps seeing the same values.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(bedrock): build the transcription NativeCall with an empty base
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(dispatch): read the resolved call instead of bound
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(host-python): rename effective to effective_py_args and note the shallow copy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): expose public names for private proxy helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep original class names behind public aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve internal callback filtering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep _PROXY_ class names for managed files hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names in package exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve recursive auth helper name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align MCP limiter tests with server enforcement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): match main's MCP limiter tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep old private names bound in importing modules
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve compatibility imports through strict lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): use exact pyright suppression in password helper test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): add reasons to compatibility import noqa comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): update IN-list baseline for renamed helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): separate Messages settings from capability inputs
* refactor(rust-bridge): unify Messages OCR and Responses call inputs
* refactor(rust-bridge): share NativeCall across inference entrypoints
* fix(rust-bridge): preserve Responses URL aliases and public test inputs
The native response-cache runtime attached through Cache._native_cache is
unreachable since the V2 cache replaced it. Drop the Python branches and
wrapper, the _ResponseCacheRuntime pyclass and its backend/activation/
semantic modules, the python-bridge deps only they used, and the tests and
fixtures dedicated to that path.
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): group cache tests under cache/ and fold test_ocr.py into ocr/
The two failure cases in test_ocr.py duplicated the upstream-500 and timeout rows
of PUBLIC_FAILURES, so only the file-input encoding case moves to ocr/test_requests.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(rust): split the response cache suite into one file per backend
test_response_cache.py grew to 2400 lines. Each backend now has its own file, shared
fixtures live in cache/conftest.py and shared helpers in support/cache.py. The helpers
alias the private native test handles once, dropping the per-call reportPrivateUsage hits
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* split tokenizer test
* test(core): consolidate route integration tests under tests/ with rstest and wiremock
Moves the public-API OCR route tests out of src/ocr/route.rs and document.rs into
tests/ocr/, split per provider plus lifecycle, machine, and document tests, merging
the duplicated pairs. Messages, audio transcription, and chat completions share one
wiremock-based upstream and recording secret source in tests/support, and gain
table-driven cases for auth, routing, upstream errors, streaming, and declines.
Tests of litellm-llms items move to that crate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(messages): keep the stream relay test independent of the stream head contents
The stream head carries no headers on main, so the relay test asserts the
open-then-deliver order and the relayed body instead of header hand-off.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
