* feat(rust): add litellm-s3 crate, AwsCredentialsProvider and cost map fetch
* fix(model-catalog-refresh): time each read instead of the whole cost map download
The 5 s fetch timeout covered the full body, so a slow but steady link failed every attempt where httpx and boto3 would finish. Each read now gets its own 5 s, matching main.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Reworks the redis-client, connect, resolve and testing crates, adds python-compat json loads and
TTL seconds, replaces the redis-store memory tier with a DualCache plus a shared failure log, and
gives the Python side the Redis timeout log throttle and a request OTEL parent span for service events
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(gateway): forward HTTP requests to the Python proxy
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(gateway): preserve required forwarding headers and separate upgrades
* test(gateway): clarify hash compatibility and relocate forwarding docs
* test(gateway): define URL fragment forwarding behavior
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds the litellm-router-core crate, its step-driven PyO3 bridge
(RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/),
served behind LITELLM_RUST through the ROUTER route rule. The host
call-out is the only boundary: every attempt goes through Host::invoke,
there is no native executor path, and RustCore never builds, wraps or
forwards to PythonCore, so router_python_core.py stays deletable.
Attempt preparation lives in litellm/router_attempt.py so it outlives
router_python_core.py.
RustCore matches PythonCore's behavior, including its known bugs (fixed
in follow-up PRs). Routing decisions (strategy picks, routing groups,
PTU, pre-routing, pre-call checks, fallback walk, health routing, access
groups, weights, cooldowns, allowed fails, usage) live in the crate;
work that needs live Python objects (request shaping, endpoints,
streaming, model info facts, callbacks, alerting, prompt management)
lives in the host-side litellm/router_rust/ package (steps/, invoke/,
surface/) driven by Rust steps.
The crate owns the registry. It runs the deployment build loop (secret
references, ids, organization fan-out, install order) and asks the host
for what needs Python, one deployment at a time so errors surface where
Python raised them: secrets, validate, provider and added steps. It
holds the model list, the index maps and the pattern routers with the
Python core's history rules (host_index.rs), and answers the registry
lookups (lookup.rs). The Python views of that state in
litellm/router_rust/surface/ (model_list, index(), names(), patterns())
read the handle, and every write to them becomes a changeset. Model
info, deployment model info, configured limits and model group info are
merged in the crate (model_info.rs, group_info.rs) from catalog rows and
facts the host answers on a model_info step asked at query time, so
price reloads and register_model stay visible. The usage cache key
format is defined once, in the crate.
The crate records each attempt's outcome itself rather than through
litellm callbacks. A streamed attempt is deferred: it counts its
request when the stream opens, and the host's stream wrappers report
the success with the final usage, or a failure after the first chunk,
when the stream ends, as the Python core's callbacks do.
tests/unit/test_router/core characterizes PythonCore's public behavior
and runs every test on both cores, with nothing pending; only the names
of the Python core's own outcome callbacks are asserted on Python alone.
litellm.Router is now final: subclassing it raises TypeError at the class statement, and the
machinery that kept subclass overrides working is gone. The Rust core never honoured subclass
overrides, so they would have silently stopped running there. Custom routing goes through
register_plugin(), settings through apply() and outcomes through observe(). This is a breaking
change for anyone subclassing Router.
Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries,
max_fallbacks, allowed_fails, retry_after, timeout, the tag settings,
routing_strategy_args and the fallback lists as given (the crate reads
them as numbers, update_settings casts as Python does); a deprecated
`model_list = [...]` write reaches the Rust core; a failed set_model_list
is all or nothing on both cores (the deployments served before the call
are put back and the error raised); config_deployments() is a generator and `del` of a plain
attribute works on the Rust core. Least-busy pick reservations from
main now work on the Rust core: its selectors share one reservation
store, routed calls reserve and free their unstarted picks, and
select() and batch cancel do not reserve. On the Python core a dropped
upsert takes its name back off deployment_names, and
get_deployment_order reads every numeric spelling as one int so mixed
orders no longer raise TypeError.
Rulings on the remaining differences, the Python core changed where it
was the odd one out: sync picks and picks by deployment id apply the
team filter as async picks do, so another team's deployment is refused;
the fallback lists and default_litellm_params in RouterSettings are
copies, so an in-place edit is ignored and settings change only through
apply(); one health record with an unreadable timestamp drops out alone;
an empty fallback entry is skipped; a bool cooldown_time is refused at
construction, a bool or string cooldown_time on a deployment or a call
falls back to the router default, and a failure status that no HTTP
status can be is recorded as 500; an infinite or NaN cost and a whole
number wider than 64 bits (ptu_shares) are refused on both cores with
the same error (litellm/router_utils/deployment_numbers.py); and a
/v1/messages stream whose provider iterator raises cools the deployment
and retries like a non-stream call, with the Rust core waiting for that
cooldown before its retry picks.
Differences the proxy parity fuzz and its regression tests found on the
Rust core: a file upload or a batch on an uploaded file falls back only
within its own model group, as on the Python core, since the file lives
with one provider (the host sends same_group_fallbacks_only and the walk
skips targets in other groups); Decisions calls drop deployments whose
provider has no Decisions config, and deployments that refuse the audio
or videos a call carries, unless that would leave none, as the Python
core does, while the retry count still reads the whole group; a file
upload that fails on one order level is created on the next level's
deployments rather than the first level again; and a response that
cannot be converted (a text_completion body without a message) reports
only its own traceback, so a rate limit handled by the attempt before it
no longer turns the error into a 429, and both cores return 500.
* build(rust): upgrade reqwest to 0.13 and move MCP out of litellm-http
The workspace moves to reqwest 0.13, so litellm-http no longer carries a
second reqwest just for rmcp. The mcp feature, its pool and its proxy
conversion go away, and the gateway hands rmcp a pooled NoRedirect client
that already carries the TLS, proxy and timeout settings
The Client test helpers collapse into for_test, a no-redirect client
without TLS or proxy settings, and wrap_for_test for the few tests that
need a hand-built reqwest client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(http): honor transport settings for Rust WebSockets
The Responses WebSocket connected through tokio-tungstenite with its own
native-roots TLS, so it ignored the configured CA bundle, verify flag,
client certificate, proxies and timeouts. The handshake now goes through
a pooled reqwest client (ClientVariant::WebSocket, HTTP/1.1, no
redirects) and upgrades the response, so it shares the settings every
other outbound request uses
The upgrade response is checked for the Sec-WebSocket-Accept key, the
Upgrade and Connection headers and a subprotocol that was offered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(http): build outbound requests as reqwest::Request
OutboundRequest was a hand-rolled copy of reqwest::Request. Routes now build
a reqwest::Request directly and RequestSigner reads the body bytes back out
of it, so the signer still covers exactly what is sent. SigV4 signing moves
onto HeaderMap and takes the method, which drops the BTreeMap plumbing and
the header casing table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(http): make litellm_http::Error the one opaque error
transport::Error and media::Error fold into litellm_http::Error, an
Arc<Repr> newtype whose private variants keep the typed reqwest, io,
rustls and tungstenite causes and own every message. Callers can no
longer build an error with a message of their own. They ask through
dispatched, is_request, status, upstream_response and tls_source
One From<reqwest::Error> now classifies connect, network and timeout
and strips the URL. The WebSocket handshake reports each failed check
as its own variant and owns its connect timeout. OCR drops the media
variants it mirrored, and inference::Error maps a header the signer
computes to Unsupported for every route, not only chat
The error rules move out of AGENTS.md into the rust-error skill, which
also records the decisions behind this design
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust): split rust-error skill into thin SKILL.md with references
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): render causes once and ask predicates at the boundaries
The four WebSocket handshake variants collapse into Handshake(HandshakeFault), and media download failures no longer answer dispatched() since they happen before the provider call
ErrorDetail in llms and auth-types, the inference error and the OCR error now store their cause in a field named cause with no #[source], so to_string() carries it exactly once and source() is None. Tests that walked the chain now destructure the variant and downcast the cause
The gateway asks is_unsupported() and is_missing_credential() on the inference and OCR errors instead of matching through nested enums, and litellm_auth::Error answers is_missing_credential() for both
The rust-error skill gains axum and pyo3 references, boundaries.md describes the real predicate order and the known divergences, the testing example asserts dispatched() correctly, and shape, classification, conversions, messages-and-causes and recipes agree with the code again
* docs(rust): focus the rust-error gateway page on the error flow and add a rust-gateway skill
The former axum page toured gateway-inference: its status match, the envelope
table, an extractor, the unsupported route and the Sse closure. Those details
belong to the gateway crates and go stale with them. The page is now gateway.md
and follows an error from the provider failure through each From into
IntoResponse, where it is classified once and rendered by surface and framing.
The gateway-structure material moves to a new rust-gateway skill: envelopes per
API surface, extractors and path registration, SSE relay, and the router test
harness.
* test(inference): pin provider URL redaction on every route's transport error
Chat, Messages, Responses and transcription each get a route-level test that
fails if a reqwest error reaches the caller with its URL. The Messages stream
decoder also takes body errors through litellm_http::Error like every other
reqwest error
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust-error): adopt std source chains and add litellm-error Report
Messages describe their own layer and the cause is the source, boundaries
render the chain once with litellm_error::Report. Corrects skill claims
the audit found stale against the code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): keep causes as sources across http, auth, secrets, cache and utility crates
Messages describe their own layer and every cause is a typed source,
predicates match exhaustively, and boundaries render with
litellm_error::Report. Media downloads no longer count as dispatched,
Error::upstream bounds the provider body itself, and a forwarded
header SigV4 cannot sign is the caller's fault.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): keep causes as sources in llms and the gateway crates
ErrorDetail loses its string variant, OCR provider variants become data,
host faults stop counting as the caller's, a tunnelled transport error
classifies the same on every stream path, and each gateway crate has one
Error with one status() and extractors that reject in its envelope.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): nest llms errors in inference and ask predicates in route tests
litellm_inference::Error nests litellm_llms::Error instead of mirroring
its variants, host driver faults get their own variant, every predicate
matches exhaustively, and route tests ask predicates instead of matching
transport variants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): map bridge and gateway-inference errors once and render them with Report
RustUpstreamError has one constructor that always sets headers, argument
checks raise typed errors mapped once, OCR asks upstream_response(), the
Responses stream error event comes from the error type, gateway-inference
loses its string variants, and auth faults classify through one
litellm_auth predicate on every route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust-error): carry OCR provider headers as a HeaderMap and keep outages off the caller
The OCR provider error and get_error_class take reqwest's HeaderMap, the
bridge turns it into Python's (name, value) pairs once. Credential
acquisition and provider SDK failures are no longer the caller's fault,
the UI hides auth and backend causes, and chat keeps a provider answer
as is.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* docs(rust-error): correct skill claims against the aligned code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): fix the remaining sweep findings in gateway, llms, auth and cache crates
Auth rejections hide their backend cause on every inference renderer,
path parameters reject in the surface envelope, Textract's unsupported
document and Azure DI's unparseable poll URL get typed variants, the
credential chain exposes its last failure as the source, gateway-mcp
asks predicates over rmcp errors, and the S3 and Azure blob transports
classify through litellm_http::Error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(rust-error): fix the remaining sweep findings on the Python side
host-python keeps a native error's chain and returns blocking-pool
cancellation instead of panicking, Vault transport failures convert
through litellm_http::Error so no URL reaches Python, the bridge asks
predicates instead of matching three levels into Vault errors, and the
last string-built PyErrs become typed errors mapped once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build(deps): refresh compatible dependencies flagged by OSV
* refactor(rust): remove redundant client construction comment
* test(rust): cover configured MCP proxy and CA bundle
* fix(http): preserve native WebSocket trust and optional protocols
* fix(http): allow plain WebSockets without native roots
* fix(http): normalize provider URLs before building outbound requests
* fix(rust): preserve OCR download status and reduce error size
* fix(rust): preserve upstream bodies and bound diagnostics only
* test(rust): preserve bridge serialization boundary in body regression
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http): honor transport settings for Rust WebSockets
The Responses WebSocket connected through tokio-tungstenite with its own
native-roots TLS, so it ignored the configured CA bundle, verify flag,
client certificate, proxies and timeouts. The handshake now goes through
a pooled reqwest client (ClientVariant::WebSocket, HTTP/1.1, no
redirects) and upgrades the response, so it shares the settings every
other outbound request uses
The upgrade response is checked for the Sec-WebSocket-Accept key, the
Upgrade and Connection headers and a subprotocol that was offered
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(http): preserve native WebSocket trust and optional protocols
* fix(http): allow plain WebSockets without native roots
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): upgrade reqwest to 0.13 and move MCP out of litellm-http
The workspace moves to reqwest 0.13, so litellm-http no longer carries a
second reqwest just for rmcp. The mcp feature, its pool and its proxy
conversion go away, and the gateway hands rmcp a pooled NoRedirect client
that already carries the TLS, proxy and timeout settings
The Client test helpers collapse into for_test, a no-redirect client
without TLS or proxy settings, and wrap_for_test for the few tests that
need a hand-built reqwest client
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* build(deps): refresh compatible dependencies flagged by OSV
* refactor(rust): remove redundant client construction comment
* test(rust): cover configured MCP proxy and CA bundle
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add circuit-breaker crate and guard redis-client with it
* fix(circuit-breaker): ignore stale probe drops
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Update certifi, Rust trust roots, and the Docker CA bundle to remove emSign ECC Root CA - G3
Require the fixed certifi release in litellm-core and retain libssl3 in Docker runtimes for Prisma compatibility
Co-authored-by: shin-berri <shin-laptop@berri.ai>
* refactor(host-python): run the Python driver as one async body
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(host-python): give the driver a PythonCaller with named suspensions
PythonTask becomes PythonCaller with await_python, open_stream and yield_chunk, so the call body no longer builds Suspension values or maps stream replies itself
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
litellm-router-types mirrors litellm/types/router.py: LitellmParams holds the
model, the credentials, one flattened group per credential family from
auth-types (aws, vertex), the deployment settings a config spells, and every
other key in extra, as Python's extra="allow" keeps it. Config parses it
directly, so config::LiteLlmParams and its untyped additional_fields are gone
and a mistyped provider param is a parse error. fields() and specs() derive
from the families' param specs for hosts that project kwargs and fold module
globals. Spelled<T> is the one untagged shape for a typed value or the text a
config spells, organization takes the list Python's router expands, and the
callback shorthand keeps a value-level one-or-many. No credentials wrapper
group: serde's flatten only consumes keys for struct-shaped children.
Move the shared test helpers out of litellm-inference's test-support
feature into a publish = false litellm-inference-testing crate used only
as a dev dependency by the format crates.
Also drop the dead src/constants.rs (OPENAI_DEFAULT_API_BASE had no
users) and declare the litellm-http/litellm-llms test-support features
on the crates that actually use them instead of relying on feature
unification through litellm-inference's dev-dependencies.
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): rename litellm-core to litellm-inference
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): expose inference base API for format crates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): move shared inference test helpers behind test-support
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): track ClickHouse migrations in a checksummed ledger
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(clickhouse): harden migration execution and reuse retention SQL
* refactor(rust): track ClickHouse migrations in a checksummed ledger
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(rust): keep ClickHouse retention TTLs in the current policy list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wip
* wip
* test(traces): separate root status from diagnostic error counts
* test(traces): cover normalization precedence and fallbacks
* chore(cache): remove stray comments from trace PR
* test(traces): name lens test for shared query path
* fix(traces): place query implementation before test module
* test(traces): use unified read scope in migration tests
* ci(rust): allow feature checks to finish
* ci(mcp): allow dependency resolution to finish
* fix(traces): preserve key visibility and safe spend attribution
* feat(tracing): add SQL queries and schema-aware query help
* test(tracing): verify help requests and sync API types
* refactor(tracing): render query help with Askama
* refactor(tracing): use jinja extension for query guide
* fix(tracing): preserve query help when discovery fails
* feat(tracing): enforce team SQL scope with managed ClickHouse readers
* test(tracing): verify reads with one ClickHouse URL
* fix(tracing): revoke rotated trace reader credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tracing): streamline query help catalog assembly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tracing): run query help discovery sequentially
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tracing): update reader setup request expectations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): embed migration folders with a shared migrate! macro
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust): enable syn proc-macro feature for litellm-migrate-macros
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust): reject signed versions and symlinks in migrate!
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* wip
* feat(traces): establish shared Rust storage foundation
* fix(traces): escape ClickHouse text parameters
* test(traces): exercise response cap with bounded strings
* fix(traces): remove unnecessary lint expectation
* fix(traces): encode ClickHouse timestamp units in Rust
* test(traces): mark exception match as a regex
* refactor(traces): execute schema setup in Rust
* refactor(traces): use shared logging execution wrapper
* docs(traces): replace foundation README with boundary rules
* fix(traces): use current bridge execution facade
* fix(traces): account for protocol cast in lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): extract litellm-host-native as the shared Rust host driver
Move service and hook dispatch out of host-http into a Driver that owns the
machine and Rust handlers, returning at completion or a stream boundary and
holding the demand reply until the consumer advances. Move the in-process
runner onto the same driver. host-http now layers encoding, SSE, body polling
and lifecycle observation over it. host-python keeps driving litellm-host
directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust): interrupt the machine when the in-process stream consumer fails
Restores the pre-refactor interruption path for StreamConsumer errors via
Driver::fail and ports the generic run lifecycle tests into host-native.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): separate the machine contract from coroutine execution
* auth update
* refactor(rust): use standard flow control for host requests
* style(rust): keep host driver imports formatted
* chores
* mostly relocation
* refactor(rust): separate interceptors from queued observers
* refactor(rust): centralize legacy callback mappings and lifecycle
* docs: define Python host boundaries and migration plan
* refactor: enforce Python host and bridge boundaries
* refactor(rust): separate operations from callback composition
* refactor(rust): compose SDK policy through call hooks
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add litellm-db and litellm-db-testing workspace scaffolding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db-testing): apply the real Prisma migrations in a test and drop the sort mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): prepare inference and auth foundations
* fix(rust): keep textract operations parsing from kebab-case model names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(testkit): install and configure Claude Code, Codex and opencode against a gateway
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* refactor(testkit): derive targets from target-lexicon and split agents behind a trait
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* refactor(testkit): group sources into agent and install folders
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* feat(testkit): split agents into install, configure and drive with semver-aware launch
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* style(testkit): drop a needless borrow
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Move the generic async coroutine out of the host crate into litellm-coroutine,
with its requirements documented in the crate's AGENTS.md. RouteMachine becomes
CallMachine on top of it, host protocol ops move to host/src/protocol.rs, and
the messages, OCR, host-python, python-bridge and legacy callbacks crates adopt
the new types. Adds error definition rules to litellm-rust/AGENTS.md.
Co-authored-by: Yujong Lee <yujong@berri.ai>
* feat(logger): add shared Rust diagnostics and Python logging bridge
* feat(logger): dispatch diagnostic processing through Rust
* chore: regenerate Cargo.lock after rebase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: allowlist bounded logging tree walkers in recursive detector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(logger): skip decoding plain access arguments
* test(logger): skip embedded-python logger test when litellm deps are absent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: cargo fmt
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: expect NativeDiagnosticProcessor in the native public surface
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(stub): export NativeDiagnosticProcessor via __new__ in _native.pyi
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tracing): rename logger crate and document host sink contract
* test(logger): cover exc, stack, and nested extras in the diagnostic filter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logger): keep rendered redacted line when template scan flags a key pattern
The blanket REDACTED for a changed msg/color template discarded lines
whose rendered form was already redacted by the same pipeline, e.g.
'password=%s' became 'REDACTED' instead of 'password=REDACTED'. Only
fall back to REDACTED when the rendered form did not change either,
which is where interpolation can mangle the key pattern the scrub
would otherwise see.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(rust): install python deps so the logger bridge test runs
The end-to-end bridge test skipped silently when litellm's Python deps
were absent. uv sync --no-install-project installs them without a
maturin build, and PYTHONPATH makes them visible to the embedded
interpreter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): align the cache crates with Python and activate every backend
The cache port had drifted: lifecycle and Redis-only operations sat on
`BaseCache`, counters were pinned to `f64`, each semantic backend defined its
own embedder and prompt handling, and only the in-memory backend could be
selected natively.
- Split `disconnect` and `test_connection` out of `BaseCache` into optional
capabilities, implemented only where the Python class defines them, and give
every Redis-only operation its own capability trait.
- Decouple counters from the stored value type, so one backend can serve both
responses and counters as Python's `RedisCache` does.
- Share one `Embedder` and prompt contract in `litellm_cache::semantic`, and
make the Redis and Valkey semantic backends generic over their codec.
- Port the Python operations that were missing: `async_refresh_ttl`,
`async_rpush_and_trim`, `async_set_cache_pipeline_with_ttls`, the DualCache
pipeline, sadd, bulk delete and TTL reads, and the semantic-similarity
write-back.
- Take the HTTP client from the host pool in the GCS, S3 and Azure backends.
- Activate all nine backends through the Rust catalog, whose rules all stay
`PYTHON_ONLY`, and route the `Cache` facade's storage calls to the native
runtime when one is selected.
- Give every crate the same layout, move all tests to `tests/` on rstest, and
add the shared `litellm-cache-testing` contract suite.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* fix: freeze native cache request kwargs and batch entries for type discipline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: declare semantic lookup methods in the native stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): opt the native Messages and tokenizer suites into Rust explicitly
#42517 made the Messages, token counter and tokenizer routes Python-only, so
tests/test_litellm_rust silently exercised the Python path or failed outright.
Each suite now prepends a RUST_OPT_IN rule for its route, keeping native
coverage without changing the shipped default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(rust): pop one at a time in the Redis 6 lpop pipeline and drop explanatory comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: benchmark and gate an installed release wheel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: simplify installed-wheel benchmark check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(rust): add native tokenizer codec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): route Python tokenization through the Rust extension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(lint): format tokenizer call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(packaging): restore runtime dependencies and native images
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve Python SDK behavior with Rust tokenizers
* fix(tokenizer): restore compatibility paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tokenizer): count custom tokenizers directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tokenizer): preserve caller-supplied Python tokenizer counts
* fix(tokenizer): reuse packaged vocabularies in the native wheel
* refactor(rust_bridge): route token counting through the catalog as RUST_OPT_IN
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): compare tokenizer groups by value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): re-resolve filelock under the <4.0 pin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(llms): align transformation override signatures with base configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(rust): use fat LTO to keep the native wheel under the 35 MB limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): preserve Python defaults with opt-in Rust dispatch
* test(proxy): tolerate missing litellm.utils.Tokenizer when patching it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): patch the tokenizer dispatch function instead of the removed alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(tokenizer): give the Rust wrappers the tiktoken and tokenizers surface
Callers of litellm.encoding and litellm.create_tokenizer must see the same
read-only API whichever backend the catalog selects.
- OpenAIEncoding mirrors tiktoken.Encoding: n_vocab, max_token_value,
token_byte_values, encode_single_token, encode_with_unstable,
encode_to_numpy, decode_with_offsets, is_special_token, repr; the Rust
tiktoken crate keeps a Vocabulary beside each CoreBPE and reports the
requested encoding name (gpt2 stays gpt2).
- HuggingFaceTokenizer mirrors the read-only tokenizers.Tokenizer surface
(token_to_id, id_to_token, get_vocab, get_vocab_size,
get_added_tokens_decoder, num_special_tokens_to_add, padding, truncation,
encode_special_tokens, from_buffer); HuggingFaceEncoding gains the
char/word/token lookups, pad, truncate, set_sequence_id and merge.
Mutators stay on the Python tokenizer.
- from_json/from_pretrained claim the fork gate only when the huggingface
feature is compiled in; the surrogate fallback matches on the Codec.
- Tokenizer caching is keyed on the same catalog Context the dispatch runs
on; rust_tokenizer reads the encoding name without loading an encoding;
LITELLM_RUST parsing is cached.
- Drop the unused tiktoken_encoding_for_model export and Error::Download.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(tokenizer): close the exhaustive matches with assert_never
CodeQL reads a `match` over a Literal with no default arm as an implicit
`None` return. `assert_never` makes the exhaustiveness explicit for both the
HuggingFace tokenizer loader and the Rust token-counter factory.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* feat(tokenizer): derive the fast counter from the shared tokenizer
The count-only counter (`fast` feature) and the codec each parsed the same
artifact: TokenCounter took the Anthropic JSON and the tiktoken rank files
from Python while Tokenizer loaded them again. One parse now serves both.
- FastTokenizer builds from a model another loader holds: `from_shared`
takes the Arc<tokenizers::Tokenizer> the HF codec keeps, and
`from_*_pairs` take the ranks the tiktoken vocabulary already parsed.
- `FastCounter::fast_counter` in the core crate derives it from either codec;
encodings the fast scanner does not reproduce are refused.
- Native `Tokenizer.count(text, fast=False)` opts into that counter, built
once per tokenizer on first use; `TokenCounter.from_tokenizer(tokenizer,
fast=False)` replaces the JSON and rank-file constructors.
- The Python route counts over the native tokenizers the codec path shares
(`native_encoding`, `native_anthropic`) and no longer reads rank files;
the packaged Anthropic tokenizer has one loader, `tokenizer_dispatch.anthropic`.
- Public wrappers gain `count(text, fast=False)`.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
