Commit graph

19 commits

Author SHA1 Message Date
zoveki
68b5a737a1
fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params (#49102)
* fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params

The native /v1/messages path sent tools with `strict` straight through, so
when Claude's strict-tool limits rejected them with a 400 the caller got that
400 even with drop_params set. Under drop_params, a 400 that matches Claude's
strict-limit phrases on a request with a strict tool is now sent once more
with `strict` removed from every tool. Without drop_params nothing changes.

The Claude strict-limit matcher moves from bedrock/common_utils.py to
anthropic/common_utils.py so Bedrock converse and the base messages config
share it, and the Rust bridge routes strict tools under drop_params to Python.

* fix(anthropic): retry without strict only when a tool sends strict true

A tool with only strict false no longer triggers a resend of an identical body, the warning no longer
blames tool schemas when the rejected schema may be the output format, and requests with only
additional_drop_params stay on the Rust messages route since the Python retry ignores them

* fix(anthropic): keep sync messages calls with strict tools on Rust

Python has no sync /v1/messages handler, so routing a sync call with a
strict tool to Python raised ValueError where Rust served it before.
The strict-tools detector now applies to async calls only. The retry
warning also names the strict tool count without a plural.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 17:18:59 -07:00
zoveki
310c96162b
fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions (#48893)
* fix(anthropic): drop the non-spec custom object from /v1/messages tool definitions

Anthropic and Vertex AI reject a tool carrying a custom object with tools.N.custom.custom: Extra inputs are not permitted. Drop it on the shared Anthropic request path, keep the hoist of a boolean custom.defer_loading to top-level defer_loading, reuse the helper on Bedrock, and stop the chat adapter from merging custom into the tool's JSON schema

* fix(messages): hoist custom.defer_loading only on Claude routes and strip custom on count_tokens

Non-Claude Messages providers such as DeepSeek now get custom dropped without a defer_loading hoist, since they have no tool search. /v1/messages/count_tokens on Anthropic, Azure AI, and Vertex AI drops custom too, and the local token counter no longer counts it as schema.

* fix(anthropic): drop tool custom object only under drop_params and hoist defer_loading for Claude models

Without drop_params the tool is forwarded as sent, so the provider's 400 stands. A boolean custom.defer_loading still becomes the top-level flag for Claude models and on Bedrock Invoke and Mantle. Bedrock count_tokens now normalizes invokeModel tools the way Bedrock Messages sends them.

* fix(bedrock): drop tool custom object on the Mantle count_tokens fallback

Mantle Messages drops custom unconditionally, so its count_tokens call now counts the tools the same way.

* fix(messages): keep custom-tool Messages calls on Python and match Mantle count_tokens to Mantle Messages

* test(vertex): annotate count_tokens tool test locals with Final

* fix(messages): pass tools without custom through untouched so Bedrock fallbacks keep filled tool names

* fix(messages): keep Bedrock's in-place tool cleanup so fallbacks still work, and stop hoisting defer_loading on the OpenAI-like passthrough

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-29 03:36:20 -07:00
Mateo Wang
14193435b8
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages

Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): keep Final variables out of loops in the rename shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): match the warning text in the rename shim test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): mark the alias loader spec restore for the type discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): format the renamed package shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): keep the rebind-ok marker on the spec restore line

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): carry the new package search path on alias specs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): declare the shim test locals Final

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 10:08:25 -07:00
yucheng-berri
dd511a2b22
fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)
* fix(logging): fire one CustomLogger hook per async call outside the async flag list

aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are
missing from is_sync_litellm_request's flag list, so the sync success and
failure handlers called a CustomLogger's sync hook on top of the async hook the
async handler had already called for the same request.

The sync handlers now skip a CustomLogger's log_success_event and
log_failure_event when that run is paired with an async handler run
(for_async_request) and the async handler gives the logger its own async hook.
The async client wrapper, dispatch_failure_handlers, the proxy-only failure
dispatch and the native bridge's sync failure step for an async call mark their
sync failure run as paired.

Resolves LIT-9486

* fix(logging): keep the sync CustomLogger hook when no async success runs

The async client wrapper skips async success logging for calls carrying a
fallbacks kwarg, and the native bridge skips it for internal and fallbacks
calls, yet the paired sync success run still dropped a both-hooks logger's
sync hook, so that logger got no success event at all.

The sync success run now suppresses the sync hook only when the async success
handler was actually scheduled for the call: the client wrapper passes
async_handler_runs=not is_completion_with_fallbacks, and the native host
records per call whether it enqueued or deferred the async success.

Tests that pinned the doubled failure hook on native aocr, messages and
responses calls and on proxy-rejected requests now expect the async hook
alone. The integration test adds /v1/messages and /v1/responses cells and
runs the key-auth rejection on every route.

* fix(logging): type dispatch_success_handlers kwargs

The new async_handler_runs keyword on success_handler made the untyped
**kwargs that dispatch_success_handlers forwards an unknown argument for it,
which grew the reportUnknownArgumentType count past its merge-base ceiling.
Typing the forwarded kwargs as Any, as _success_handler_body already does,
removes those diagnostics without changing what is forwarded.

* test(integration): cover CustomLogger hook counts across modes, chaos and router strategies

Moves the hook count rig into tests/integration/_support/hook_counts.py and
extends the cells it drives:

- streamed chat, Messages and Responses requests, success and failure
- loggers the async handler never reaches (sync-only overrides, sync event
  lists only) keep one sync hook per request on every surface outside the
  async flag list, key-auth rejections included
- a concurrent burst while the logger's destination stalls and then fails,
  and a burst while a worker is killed, never give a request two hooks
- least-busy keeps counting a request in flight while async speech requests
  finish around it, and usage-based routing v2 counts an async image
  request's tokens once

* test(integration): Datadog and LangSmith deliver one record per async request

Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local
sink and checks every async surface outside the flag list: one Datadog log
entry and one LangSmith run per request (two on the merge base), and never two
OpenTelemetry spans, which its own emit-once guard already ensured.

* fix(logging): give a both-hooks logger its sync hook when the async run never reaches it

The sync hook was skipped whenever the logger sat in the async callback list, so a
logger the async run never reached got no hook at all: a sibling stalled past the
logging worker timeout, a batch poll that skips default logging, or an async
failure run cut off. The sync run now holds the hook and the async run hands it
back unless it reached the logger

* fix(logging): run the held sync success hooks when the proxy drops a blocked response's success

A post-call guardrail block drops the parked async success, which left the sync
success hook held for it unrun. Base gave a both-hooks logger that hook on these
call types, so the drop now ends the pairing and the held hook runs as before

* fix(logging): run held sync hooks only after every sync run of the request has closed

A held hook released by the async run could run on another thread while the sync
run was still walking its other callbacks. The pairing now waits until the async
run has ended and every sync run has closed, and the last sync run to close runs
the held hooks itself

* fix(logging): end the async run when its coroutine is dropped before it starts

A logging queue that drops the async success coroutine, or a deferred success
that fails to schedule, left the sync hooks held for it unrun. The enqueued async
run now ends when its coroutine is discarded, and the proxy ends it when firing
the deferred success raises

* fix(logging): leave a still-queued async run to the exit flush

The discard finalizer ran at interpreter exit before the logging worker's exit
flush, so a queued request could get its held sync hook and then its async hook

* test: keep the router and file-stream tests in step with main

Main gives each owned proxy a private Redis by default, so the router cells
share the worker's Redis to read its counters, and the file-stream stub takes
the hook pairing the stream now passes

* fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up

A during_call guardrail that raised after the LLM call parked its deferred
success skipped the proxy's flush, and async runs the exit flush never started
were never ended, so a both-hooks logger lost its sync hook in both cases

* fix(logging): skip the sync failure hook when the async failure run logs to the logger

An async call outside is_sync_litellm_request's flag list ran both failure
handlers, so a CustomLogger with both failure hooks got both. The sync run
of an async request now skips log_failure_event for a logger that overrides
async_log_failure_event and is registered for async failures. The four sync
runs of async failures (dispatch_failure_handlers, the @client async wrapper,
the proxy-only failure thread and the Rust host) pass for_async_request=True.

Drops the HookPairing fallback. Successes are left to #47192, which already
gives such a logger only async_log_success_event.

---------

Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-20 15:19:14 -07:00
yujonglee
730db4d228
refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines (#46714)
* refactor(rust-bridge): detect Python-only features in the catalog instead of Rust declines

* refactor(rust-bridge): split and document route-specific detectors

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 15:06:25 -07:00
yujonglee
0ccae7ab35
refactor(rust-bridge): decide native routing in the catalog, drop the bypass predicates (#46750)
* refactor(rust-bridge): decide native routing in the catalog, drop the bypass predicates

* refactor(rust-bridge): simplify catalog compatibility checks

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-20 15:06:24 -07:00
Mateo Wang
16645cc963
fix(router)!: bound fake-streamed calls by the request timeout, not stream_timeout (#47162)
* fix(router): bound fake-streamed calls by the request timeout, not stream_timeout

A stream=True call that the provider serves as one non-streaming request (Bedrock Converse for json_schema on models without native structured output, tools on some Llama models, ai21) gets no byte until generation ends, so stream_timeout acted as a total-duration cap. The Router now passes timeout and stream_timeout separately to chat completion calls, and completion() applies stream_timeout only when it reads a live stream.

* test(timeout): cover stream_timeout on live streams only and the router's timeout split

* fix(router): cover every fake-stream site and keep the target group's stream_timeout under prompt management

* fix(router): never shorten a fake stream below stream_timeout, and split timeouts on every surface

A faked stream now takes the longer of timeout and stream_timeout, so a caller
who raised stream_timeout to cover a slow fake stream keeps that headroom.
Azure o-series dispatch is mirrored when deciding whether a stream is faked.
text_completion, /v1/messages and /v1/responses through the Router now get
timeout and stream_timeout separately, like chat calls.

* fix(router): report the effective stream timeout, split adapter streams, and cover M365 and Rust messages

* fix(router): report and log the timeout each stream was bounded by

Native /v1/messages and /v1/responses now record the timeout they applied, so
x-litellm-timeout and the cooldown guard read the real limit instead of the raw
request value. A prompt-managed model also passes on only the caller's or its
own timeout, so the target deployment's timeout still wins.

* fix(router): keep the caller's request_timeout on a prompt-managed handoff and report the applied stream limit

* fix(router): give streams answered in one piece the longer limit and accept text timeouts on responses

* fix(router): give /v1/messages and /v1/responses streams answered in one piece the longer limit

* test(integration): cover faked, live and converted stream timeout limits per endpoint

* test(router): regression tests for faked-stream timeouts
2026-10-17 22:08:02 +00:00
yujonglee
a1c9fa951a
test(rust-bridge): relocate dispatch tests before routing changes (#46926)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-16 12:29:33 -07:00
yujonglee
7d79ccb7cc
test(rust-bridge): select native routes per test without LITELLM_RUST (#46620)
* feat(rust-bridge): scope catalog rules with a contextvar

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): select native integration routes with scoped rules

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): run the Rust extension suite without LITELLM_RUST

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): use scoped catalog rules in dispatch tests

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* refactor(rust-bridge): keep catalog rules a static global and override them once per test

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 09:12:08 -07:00
yujonglee
0fc11653e2
test(rust-bridge): pin Messages and Responses callback contracts on the native path (#46443)
* test(rust-bridge): pin Messages and Responses callback contracts on the native path

Each test runs the native route and asserts what the Python path does today, so a
known gap is a strict xfail that names its reason and the exception it fails with
Messages and Responses share one Surface and hook recorder in support/callbacks.py,
and each surface folder lists its own expected counts and gaps.

Body-level hooks from the function body (pre-request hook, websearch short-circuit,
Messages interceptors, planned and legacy agentic loops, streaming deployment hook)
are xfail until native runs them

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(rust-bridge): give every route folder the same one-concern-per-file layout

Route data, the route Surface and the server fixture move into each route folder, and support/ keeps only
route-neutral helpers. test_inference.py splits into chat_completions/ and responses/, ocr/test_requests.py
splits into requests, failures and settings, and Messages and Responses get streaming and body hook files.
Messages request tests that duplicate inference-messages request.rs are removed, and drop_params read
from litellm.drop_params is now covered

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(rust-bridge): expect the required-route decline error for unsupported Responses requests

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): import route data from the route folders in the secrets unit tests

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): import NoReturn in the OCR dispatch tests

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): move route-neutral helpers out of support/callbacks.py

Co-Authored-By: Yujong Lee <yujong@berri.ai>

* test(rust-bridge): cover deployment rejection and concurrent callbacks

* test(rust-bridge): assert native serving via the x-litellm-rust marker in the route folder tests

Co-Authored-By: Yujong Lee <yujong@berri.ai>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 17:07:20 -07:00
yujonglee
8410b5ba5e
test(rust): assert native serving via the x-litellm-rust response marker (#46531)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 22:31:47 +00:00
yujonglee
6830a5c338
fix(python-bridge): preserve callback keyword deletions in native requests (#45692)
* fix(messages): preserve callback keyword deletions in native requests

* fix(python-bridge): share hook-aware request view across native route hosts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-10 02:36:49 +00:00
yujonglee
1bc85fad40
refactor(python-bridge): ship a signature base and read the resolved call (#45450)
* refactor(python-bridge): ship a signature base and read the resolved call

NativeCall carries base (positionals by name plus signature defaults) instead
of the fully bound dict. resolved lays kwargs over base, which is what bound
held, so every pre-hook read and every route host keeps seeing the same values.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(bedrock): build the transcription NativeCall with an empty base

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(dispatch): read the resolved call instead of bound

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(host-python): rename effective to effective_py_args and note the shallow copy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-08 17:20:29 -07:00
yujonglee
2aaa0b5d5c
refactor(rust-bridge): unify native call inputs and Messages settings (#45126)
* refactor(rust): separate Messages settings from capability inputs

* refactor(rust-bridge): unify Messages OCR and Responses call inputs

* refactor(rust-bridge): share NativeCall across inference entrypoints

* fix(rust-bridge): preserve Responses URL aliases and public test inputs
2026-10-07 15:05:43 -07:00
devin-ai-integration[bot]
081f73f021
feat(rust): hand upstream response headers to the native Messages stream (#43178)
* ci: drop the ocr_testing job now that tests/ocr_tests is gone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(ocr): restore the live OCR matrix and the ocr_testing job

The public litellm.ocr / aocr / Router interface is unchanged by the Rust
migration, so the live provider matrix still applies. Drops the stale VCR skip
list for the deleted test_rust_bridge.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(messages): show streamed upstream headers never reach the native stream

The Python handler puts the upstream response headers on the stream's
_hidden_params before the first chunk so the proxy can forward them as
llm_provider-* headers. The native route drops them, and this test fails
on the Rust path while passing on Python.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* feat(messages): hand upstream response headers to the native stream before its first chunk

The Messages route fills MessagesStreamHead from the upstream response and
yields it on Open. The Python driver converts it through the protocol host
and hands it to Stream and SyncStream as their _hidden_params, so a
streamed native call carries additional_headers the same way the Python
handler does and the proxy can forward them as llm_provider-* headers.

The relay contract lives in the core crate test, the hand-off in the
host-python driver test, and the header projection in the route host test,
so the recording-server test that showed the gap is dropped.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* wip

---------

Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 10:03:15 -07:00
devin-ai-integration[bot]
bdf854c3ea
feat(rust): shape Anthropic Messages requests natively (#42982)
* test(rust): encode anthropic response serialization shape as rstest cases

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(rust): shape Anthropic Messages requests natively

The Rust Messages route only relayed the body. It now runs the request shaping
the Python handler does for the direct Anthropic provider: history sanitizers
(empty blocks, tool ids, replayed web search results, provider_specific_fields,
encrypted reasoning, advisor blocks), reasoning_effort and adaptive/legacy
thinking translation against the model's capability flags, the sampling and
speed gates under drop_params, the metadata allowlist, additional_drop_params,
reasoning auto summary, OAuth and ANTHROPIC_AUTH_TOKEN credentials,
provider_specific_header merging and anthropic-beta injection. Capability flags
and LiteLLM settings reach Rust through route_host.shaping(). A request the
route rejects before the call now maps to BadRequestError instead of
APIConnectionError

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* test(rust): port Anthropic Messages shaping tests and pin comment contracts as cases

Every Python unit test that exercises the ported shaping for the direct
Anthropic provider now has a named rstest counterpart, and every comment that
stated a behavior contract is deleted in favor of a case that pins it. Measured
with cargo-mutants over the touched files, all viable mutants are caught

Porting the tests surfaced parity gaps, fixed here to match Python: every
casing of a forwarded anthropic-beta header is merged, replayed web search
results are rewritten from their own block (an empty result keeps its slot and
a server_tool_use with a non-string query stays), an empty output_config.effort
falls back to medium, speed and reasoning effort errors quote values the way
Python does, additional_drop_params apply after metadata validation and the
auto summary and never touch model or messages, and a non-string
metadata.user_id is rejected before the call

* fix(rust): resolve Messages credentials through the secret source and scope headers by resolved provider

The native Messages route read ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN and the
base URL straight from the process environment, so a key or base held in a
configured secret manager was never found. Each provider config now declares
its secret names and the route resolves them through the same SecretSource the
OCR route uses, with the Python bridge passing in litellm's configured manager

provider_specific_header entries were scoped by the explicit
custom_llm_provider only, falling back to anthropic, so an azure_ai/ model
lost its azure_ai scoped headers. Scoping now happens in the route after the
provider is resolved from the model, as Python's handler does

The Azure config now adds the same anthropic-beta feature headers Python's
Azure route adds, and the metadata allowlist, reasoning auto summary and
history sanitizers move from the core route into the llms crate, mirroring
their home in Python's messages handler

* test(rust): escape the dot in the metadata.user_id match pattern

* refactor(rust-bridge): project Messages capabilities without mutable dicts

The capability flags and effort tiers were built as dict comprehensions,
which the type-discipline gate counts as mutable construction, and the
asdict call carried a mutable-ok suppression that suppressed nothing. The
flags are now passed one by one and the effort tiers are a frozen dataclass,
which asdict projects to the same map the native side reads

---------

Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 19:08:32 +00:00
devin-ai-integration[bot]
571ada0b0f
feat(rust_bridge): mark native streams with the x-litellm-rust header (#42758)
Non-streaming responses served by the Rust core already carry
x-litellm-rust: true through _hidden_params.additional_headers, which the
SDK exposes and the gateway renders as a response header. Native streams
did not, because the lifecycle Stream and SyncStream objects had nowhere
to hold hidden params and the marker writer skips objects without them.

Give both stream classes the same _hidden_params bag every other litellm
response has, so the existing marker attaches without wrapping the stream
or changing its identity.

Co-authored-by: Yujong Lee <yujong@berri.ai>
2026-09-24 09:36:25 -07:00
devin-ai-integration[bot]
4677f1028e
refactor(rust): align the cache crates with Python and wire every native backend (#42530)
* refactor(rust): align the cache crates with Python and activate every backend

The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.

- Split `disconnect` and `test_connection` out of `BaseCache` into optional
  capabilities, implemented only where the Python class defines them, and give
  every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
  responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
  make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
  `async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
  pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
  write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
  `PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
  runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
  add the shared `litellm-cache-testing` contract suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: freeze native cache request kwargs and batch entries for type discipline

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: declare semantic lookup methods in the native stub

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(rust): align the cache crates with Python and activate every backend

The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.

- Split `disconnect` and `test_connection` out of `BaseCache` into optional
  capabilities, implemented only where the Python class defines them, and give
  every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
  responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
  make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
  `async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
  pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
  write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
  `PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
  runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
  add the shared `litellm-cache-testing` contract suite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix: freeze native cache request kwargs and batch entries for type discipline

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: declare semantic lookup methods in the native stub

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(rust): opt the native Messages and tokenizer suites into Rust explicitly

#42517 made the Messages, token counter and tokenizer routes Python-only, so
tests/test_litellm_rust silently exercised the Python path or failed outright.
Each suite now prepends a RUST_OPT_IN rule for its route, keeping native
coverage without changing the shipped default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(rust): pop one at a time in the Redis 6 lpop pipeline and drop explanatory comments

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-22 13:13:02 -07:00
Yujong Lee
1a52bae779 add streaming message 2026-09-18 15:45:08 -07:00