Move each route's bridge module under litellm/rust_bridge/<route>/ so a folder
means a Rust implementation exists while the catalog row says whether it is
used. OCR now keeps the Python implementation in litellm/ocr/main.py and the
Rust selection in litellm/ocr/rust.py, removing litellm/ocr/legacy.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A registered gateway DCR client can now POST /token with
grant_type=urn:ietf:params:oauth:grant-type:token-exchange and an IdP
JWT as subject_token. The gateway proves the JWT the way its JWT auth
does, resolves the user and team, and answers with the proxy-API
credential and a refresh token, so a fresh laptop with only an IdP login
gets a gateway key without a browser round trip.
"/token" joins mcp_inference_routes so the default JWT team allowlist
reaches the exchange, and the JWT auth builder accepts any header
mapping so the request headers pass through unchanged.
Adds integration contracts for MCP lifecycle, protocol errors and OAuth
configuration, A2A wire versions, the OpenAI consumer path, persisted
toolsets, callback delivery, guardrail effects, configured prices, the
filtered spend ledger, and a CircleCI-owned browser flow for project
detachment, with the ASGI, browser-state, client and MCP helpers they use.
Consolidates the eleven commits previously stacked on
litellm_integration_providers onto its rebased tip
Since #40994 the proxy relays the upstream error body on a 400, so the
public error type is now whatever the upstream sent instead of the
status-derived name. The fake upstream answered every scripted failure
with type api_error, which made the public-error contract in
test_retry_counts_and_public_errors_match_actual_provider_attempts fail
on main. The fake now sends the type a real OpenAI-compatible upstream
sends for the status, so the assertion holds whether the proxy relays
or maps the type
The request path used get_end_user_object, which falls back to a database
lookup on a cache miss. Read the LiteLLM_EndUserTable row auth already cached
instead, with the default budget already attached, and leave misses to the
periodic refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Without a master key the auth layer echoes whatever key the caller presented as the authenticated key, so the passthrough's strip-by-value matched the caller's own Anthropic key and dropped it: a bring-your-own-key request that returned 200 on main answered 401 telling the caller to send the key they had just sent. Only the auth module's own no-auth dev-mode definition, shared through is_no_auth_dev_mode, decides that nothing was authenticated, and only when no custom auth is installed; JWTs, OAuth2 tokens, and custom-auth credentials are still stripped there. The sk- prefix heuristic goes with it.
The Vertex credential-less test now sets a master key, since a virtual key can only authenticate under one: the auth layer returns before any key lookup when the master key is unset.
The metric attribute filter now removes every attribute whose value is
None, and the content and inference-details events pass their attributes
through drop_none before emitting, so a call with no provider label or no
model name never hands the OTLP exporter a NoneType attribute. This closes
the gen_ai.request.model report on #36759 the same way the gen_ai.system
one was closed, and the regression tests cover both keys.
Adds owned local TCP peers, a Redis process helper and SigV4 helpers to
tests/integration, and integration contracts for Anthropic and Bedrock wire
shapes, Bedrock role configuration, S3 wire, observed routing, Redis
recovery and stream contracts. Consolidates the four commits previously
stacked on litellm_integration_accounting onto its main-merged tip
A stream counted before its usage is known increments TPM by zero, so the
worker that served it never refreshed its local TPM value from Redis and
the first byte headers reported the token count another worker had already
consumed. Both pipeline operations now always run, matching the pre-change
callback, so the returned values refresh both worker local keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Parse the switch with pydantic TypeAdapter(bool) so 1/true/yes/on and 0/false/no/off all work, and treat an unparseable value as unset instead of off. PYTHON_ONLY and RUST_REQUIRED still ignore both switches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The compat cells drove the CLI with a fresh HOME per invocation and the
pytest process's own working directory, and both reach the request body.
The system prompt names a memory directory built from
$CLAUDE_CONFIG_DIR/projects/<cwd slug>, so a per-invocation config
directory rewrote every body, and the CLI adds a git block for its
working directory, so inheriting the checkout rewrote every body once
per candidate. The device id churned for the same reason: the CLI mints
it once and persists it in .claude.json, which we threw away each call.
Nothing here was load-bearing. All three ride in metadata.user_id, whose
job is abuse detection, not quota, caching or continuity. So pin the
config directory and the working directory at fixed paths, seed the
device id, and pin the session id.
HOME stays fresh and empty per invocation, so the isolation is no weaker
than before, and the CLI's own state no longer outlives the pod either.
The working directory is deliberately not the checkout, so a
model-directed Read now sees an empty directory rather than the
repository.
A pinned session id needs --no-session-persistence beside it: the CLI
refuses a session id another live process holds, and the matrix runs its
cells across xdist workers. Without the flag, six of eight concurrent
invocations die on "Session ID is already in use".
Replace the stateful transcription loader with NativeBinding pairs and call
runtime.run/arun from the Bedrock dispatch class so the RUST_REQUIRED catalog
row is load-bearing: missing native and admission declines are terminal, and
there is no Python replay. Cover the remaining runtime, OCR lifecycle and
configuration branches, and make decide() exhaustive.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The per-route capability test hardcoded vendor facts, including function calling support on the gemini route, which the live model card says is not supported. Keep the backup-matches-main invariant and the routing tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The budget-tier reset read every customer linked to an expiring tier into
one result set before the write, then invalidated their caches one key at
a time. Both of those scale with the customer count, so a large enough
deployment can OOM the proxy pod on the read, and the tail of the
population sits on a stale spend counter while the per-key invalidations
drain
PR #40639 moved the reset write itself to a link-based UPDATE, so that
pre-commit read no longer feeds the write. It only fed cache invalidation
and the service-logging counts, which means it can move after the commit.
This replaces it with a keyset walk over litellm_endusertable ordered by
user_id, taking RESET_BUDGET_JOB_BATCH_SIZE rows per page, the same shape
_reset_windows_for_source already uses, with no per-run page cap for the
same reason that walk has none: the cursor cannot survive the run, so a
cap would restart at the first customer on every tick and never reach the
tail
Each page's counter and cache keys now go out as one batched delete
through a new DualCache.async_delete_cache_keys, which drops the
in-memory entries and chunks the Redis DELETE at
DEFAULT_MAX_REDIS_BATCH_CACHE_SIZE
num_endusers_found and num_endusers_updated now report the customers
whose caches were invalidated after the commit rather than the rows read
before it, so both read 0 when the cascade write fails
The pre-call hook passed the proxy's whole per-request metadata dict into the
request identity, so proxy-owned siblings such as requester_ip_address were
promoted alongside the caller's keys. Only the requester_metadata mapping is
read now, keyed under its wrapper, which keeps the default allowlist behaviour
unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
