Piece 4 replaced V1PerUserTokenStore with the v2-native chain at the composition root, leaving the
adapter with no callers, so remove it and its test. The shared v1 read/refresh core
(resolve_user_oauth_access_token and friends) stays - delegate's egress in server.py still uses it -
and comes out with the delegate migration.
The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic
SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh),
release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's
RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired /
not-held so a cache blip causes an extra refresh, never a crash on the resolve path.
The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET
NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the
token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The
lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read
and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis
SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis.
The cross-replica TokenCacheBackend implementation that plugs into the foundation's
CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's
shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a
token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected;
a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss.
The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so
encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token
and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived
refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token
always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss.
The refresh_token grant for the authorization_code mode: POSTs the RFC 6749 refresh_token grant to
the server's token endpoint, persists the rotated triple, and returns the new typed OAuthToken for
RefreshingTokenStore to cache. HTTP post and persist are injected so the grant + response parsing
are testable without a live IdP/DB. Also extends the TokenRefresher seam with (user_id, server_id),
which the foundation's refresh(token) lacked but the grant (server config) and persist (key) need.
Reads the user's persisted authorization_code credential and returns a typed OAuthToken (access
token, epoch expiry, refresh token), validating the decoded blob at this boundary so no Any leaks
past it. The raw inner store that RefreshingTokenStore/CachedOAuthTokenStore wrap; the DB read +
decode collaborator is injected so it stays testable. Not yet wired - V1PerUserTokenStore is still
the composition-root store until the refresher and cross-worker cache land.
raise_user_oauth_challenge emitted the header lowercase while the sibling raise_public and every
resource_metadata (RFC 9728) emitter use the canonical WWW-Authenticate; align it. HTTP header names
are case-insensitive on the wire so this is cosmetic for compliant clients, but it keeps the challenge
builders consistent and matches RFC 6750.
The discovery-phase 401 no longer calls v1's _get_user_oauth_extra_headers_from_db to decide
whether a migrated server has a token; it asks the v2 resolver via a new has_user_oauth_token
manager method (to_server_spec + to_subject + resolve_credentials, Ok means a token exists). With
this, every authorization_code resolution runs through the v2 resolver: the call_tool egress, the
listing connection, and the discovery challenge. Delegate servers short-circuit before the check
(the client completes PKCE with the upstream). The challenge itself still emits the RFC 8414
authorization_uri form; the format unification stays a follow-up.
_resolve_oauth2_headers_for_tool_call steps aside (builds no header) when to_server_spec maps the
server, so the v2 resolver drives the token-present case instead of being shadowed by a token v1
places in extra_headers. Non-migrated oauth2 (delegate, client_credentials) and BYOK still build
their header on v1. With this, v2 owns the authorization_code egress end to end: inject the
refreshed per-user token when present, raise the per-server fail-closed 401 when absent.
When an authorization_code server has no usable per-user token, the arm returns a semantic
unauthorized and the graft builds the 401 where the full MCPServer is in hand: a relative,
per-server RFC 9728 resource_metadata pointer (/.well-known/oauth-protected-resource/mcp/{name})
that names the server's own authorization server, instead of the resolver's earlier root pointer
which resolved to the gateway's generic PRM. Relative, so it is correct behind a reverse proxy
without request context. The listing-phase 401 still emits the RFC 8414 authorization_uri form;
both now target the same server, so the remaining difference is cosmetic and unifies in a later PR.
to_server_spec maps an oauth2 server to AuthorizationCodeConfig when it relies on per-user tokens
(needs_user_oauth_token and not delegate_auth_to_upstream); client_credentials (M2M), delegated
upstream OAuth, token exchange, and SigV4 still defer to v1. The manager injects V1PerUserTokenStore
(resolving through v1's shared egress core) into the credential provider. The v2 path is live but
still defers to a token v1 places in extra_headers; the cutover that makes v1 step aside lands next,
alongside the unified challenge.
Extract v1's per-user OAuth egress (Redis cache, else DB read with the refresh_token grant, then
re-cache) from _get_user_oauth_extra_headers_from_db into resolve_user_oauth_access_token in db.py;
the v1 header builder is now a thin wrapper over it and its callers are unchanged.
V1PerUserTokenStore (the v2 OAuthTokenStore adapter) resolves through that same core via an injected
server lookup, so the authorization_code arm injects exactly the token v1 would, with the same silent
refresh, rather than a Redis-only read that can never refresh. One resolution implementation, two thin
adapters (header dict and OAuthToken). Behavior-preserving: the existing v1 egress tests pass
unchanged, and the arm is not wired into the live path yet (that lands with to_server_spec + the
manager).
V1PerUserTokenStore reads the user's stored access token through v1's mcp_per_user_token_cache
(Redis-backed, encrypted) and wraps it in an OAuthToken. v1 holds only the access token (its
cache TTL is the lifetime), so no expires_at/refresh_token yet; the v2 cache holds it for its
default TTL and the OAuth challenge drives re-auth once v1's cache drops it. Additive: nothing
wires it yet, so no behavior change. Step 1b swaps it for a v2-native token store behind the
OAuthTokenStore seam.
Resolve a user's authorization_code token through the injected OAuthTokenStore: present ->
Authorization: Bearer <access_token>; absent -> the RFC 9728 WWW-Authenticate OAuth challenge;
store unavailable -> the same challenge (not a 500), since a transient outage is not a definite
absence. UpstreamCredentialProvider gains the oauth_token_store collaborator (fail-closed null
default); per-subject isolation comes from keying the fetch on subject_id. Not live until
to_server_spec maps authorization_code and a v1-backed token source is wired (next steps).
Make CachedOAuthTokenStore's storage and RefreshingTokenStore's single-flight injectable so a
cross-replica deployment can back them with Redis without touching the resolver. The defaults preserve
today's behavior exactly: InMemoryTokenCacheBackend (the bounded per-process dict) and
InProcessRefreshCoordinator (the asyncio single-flight). A distributed deployment injects a shared
DualCache-backed backend and a SET NX PX coordinator. invalidate() is now async (the backend may be).
The cache stores via the backend with a TTL derived from the token's expiry; the coordinator threads a
reread callback for the cross-replica case (losers re-read the persisted token) that the in-process
default ignores.
The refresh seam took only the OAuthToken, but a refresher needs the server's
config (token endpoint, client credentials, scopes) to run the grant and the
(user_id, server_id) key to persist the minted token, neither of which is
derivable from the token. Widen TokenRefresher.refresh to (user_id, server_id,
token) and pass them through from RefreshingTokenStore so each stacked mode PR
plugs into the final seam rather than forcing a later signature change across
the stack.
The proactive token-refresh / cache-expiry buffer defaulted to 30s, which is
an outlier among OAuth clients. Spring Security uses 60s as both its JWT
clock-skew tolerance and its refresh buffer, and 60s sits inside RFC 7519's
"a few minutes" leeway while preserving nearly all of a typical token's life;
30s was untested, so pin the default with two boundary-probe regression tests.
CachedOAuthTokenStore no longer caches the "not authorized" None result; every miss re-reads the
inner store. v1's per-user token cache never caches misses, so a token written by the OAuth flow
is visible on the next request without an invalidation hook, and uniformly across replicas since
the in-process cache holds no stale None to clear. invalidate() now only covers rotation or
revocation of a cached token. Negative caching (with distributed invalidation) can return later
if a slow DB-backed v2-native source makes per-miss reads expensive.
Add TokenRefresher (a mode-supplied seam: mint a fresh token from an expired one and persist it)
and RefreshingTokenStore: when the stored token is near expiry, the first caller refreshes while
concurrent callers await the same in-flight task and share its result, so the IdP is not
stampeded. The task self-cleans (a done-callback drops its entry), so the map is bounded by
in-flight refreshes rather than by distinct users/servers, and is detached from the caller so a
cancelled caller does not abort the refresh. An expired token the refresher cannot renew surfaces
as None so the arm challenges, never a stale bearer; it composes under CachedOAuthTokenStore.
OAuthToken's repr masks the access/refresh tokens so a stray log cannot leak them. Cross-replica
single-flight (Redis) and reactive-401 refresh are the later distributed hardening.
Lay the foundation for the authorization_code resolver arm: OAuthToken (access_token,
expires_at, refresh_token), the OAuthTokenStore Protocol seam, TokenStoreUnavailable for
outages, and CachedOAuthTokenStore, an expiry-aware cache that serves a token only while
unexpired, caches the "not authorized" None for a default TTL, and propagates a store outage
without caching it. Mirrors the BYOK store/cache pattern, adapted for tokens. Refresh and
distributed single-flight are deferred to the hardening step.
The unauthorized case becomes a structured Unauthorized (detail + optional WWW-Authenticate
header + optional structured body) instead of a bare string, and raise_public emits the header
and body when present. This lets a mode reproduce a rich 401 challenge (e.g. BYOK's
provisioning prompt) through the generic resolver edge. of_unauthorized's new params are
keyword-only and default to None, so existing callers and the summary string are unchanged.
* feat(mistral): support Mistral OCR 4 (mistral-ocr-4-0)
Add the mistral/mistral-ocr-4-0 model to the cost map and reprice
mistral/mistral-ocr-latest, which now resolves to OCR 4 server-side,
at $4 / 1000 pages. Add the include_blocks param so callers can request
OCR 4's paragraph-level bounding boxes and typed content blocks.
OCR 4's new per-page response fields (blocks, confidence_scores, tables,
hyperlinks, header, footer) already pass through transform_ocr_response
via the extra="allow" config on OCRPage; add a regression test pinning
that behavior alongside cost and param coverage.
* fix(mistral): revert unverified OCR 4 annotation_cost_per_page bump
Mistral's published OCR 4 pricing lists $4/1000 pages for the API and no
separate annotation rate; the $5/1000 figure is the distinct Document AI
(Studio) tier. The earlier 0.003 -> 0.005 bump on annotation_cost_per_page
had no cited source, and ocr_cost() never reads that field (it bills off
ocr_cost_per_page), so the value is documentation-only.
Revert annotation_cost_per_page to the existing 0.003 convention for both
mistral-ocr-latest and mistral-ocr-4-0, keeping only the verified, tested
ocr_cost_per_page: 0.004 change.
* fix(mistral): set OCR 4 annotation_cost_per_page to verified $5/1000 rate
Verified against Mistral's authoritative sources: the pricing page, the
OCR 4 announcement, and the ocr-4-0 model card all list OCR 4 at $4/1000
pages for basic OCR and $5/1000 for annotated pages (Document AI). The
$5/1000 figure is the annotated-pages rate, which is exactly what
annotation_cost_per_page encodes, mirroring the original OCR entry's
0.001 basic / 0.003 annotated split.
Restore annotation_cost_per_page to 0.005 for mistral-ocr-latest and
mistral-ocr-4-0; the earlier revert to 0.003 was based on an incomplete
reading that treated Document AI as a separate product. ocr_cost_per_page
stays 0.004, which is the value billed by ocr_cost().
* fix(mistral-rust): include_blocks in Rust OCR supported params
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(aiml): add openai/gpt-image-2 image model
Adds aiml/openai/gpt-image-2 to the cost map and teaches AimlImageGenerationConfig
to route OpenAI-style image models through the upstream OpenAI request schema
instead of the AI/ML flux schema. Without this, size, n, and response_format would
be remapped to image_size/num_images/output_format, which the gpt-image-2 endpoint
on api.aimlapi.com does not accept.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(aiml): note gpt-image-2 flat-rate pricing basis; apply ruff format
Documents in the cost-map notes that output_cost_per_image is AI/ML's
published medium-quality rate, billed as a flat per-image price like the
other aiml image entries. Reformats the touched files under the repo's
ruff formatter (migrated from black in #31317).
* fix(aiml): drop /v1/images/edits from gpt-image-2 supported_endpoints
LiteLLM only implements an image generation transformer for AIML, so
listing /v1/images/edits overclaimed support. Align with every other
aiml image entry, which lists only /v1/images/generations.
* style(aiml): format transformation.py at line-length 88
The repo formats litellm/ with ruff at line-length 88 (Makefile/CI call
sites), while ruff.toml's global 120 only governs E501/import sorting.
Reformat the transformer to 88 so make format-check / CI lint pass, and
restore the test files to their original layout since tests/ is not part
of the auto-formatted tree.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
The namespace configured under cache_params was only applied to get/set/
increment paths. Operations that take keys through other code paths (the Lua
scripts registered via async_register_script, delete, scan_iter, rpush, lpop,
get_ttl, and the sync increment_cache) hit raw keys. With a namespace set, the
rate limiter ({key}:tokens/requests/window), pod-lock release, and budget
limiters wrote keys outside the configured prefix, breaking multi-tenant key
isolation and leaving those operations reading keys the namespaced writes never
created.
check_and_fix_namespace is now applied uniformly across every key-taking
RedisCache operation. It is a no-op when no namespace is configured, so
deployments without a namespace are unaffected. The prefix is prepended ahead of
any {hash-tag}, so Redis Cluster slotting is preserved.
Resolves LIT-3374
* chore(lint): widen ruff budget slack to 10% of baseline for high-volume ANN rules and PLR0913
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(lint): drop PLR0913 from strict gate to roll out rules gradually
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(lint): ratchet-guard rising baselines even when slack is cut to mask them
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(otel): hashable scope for _emit_once when guardrail_mode is list
`_emit_once` keys `spans_logged` by `(class, id, *scope)`. When a
guardrail entry's `guardrail_mode` arrives as a `List[GuardrailEventHooks]`
(the shape Presidio expands to with `output_parse_pii: true`, and the
shape `event_hook` carries for any `mode: [...]` in config), the tuple
contains a list and `spans_logged.get(dedupe_key)` raises
`TypeError: unhashable type: 'list'`. On the post-call path this fires
inside the logging callback and is swallowed; the request returns 200 but
the OTEL `guardrail` span is silently dropped. On the blocking path the
same error surfaces as HTTP 500.
Adds `_freeze_for_dedupe`, a small recursive normalizer that turns lists
and tuples into tuples, sets into frozensets, dicts into frozensets of
`(key, value)` pairs, and falls back to `repr` for arbitrary
unhashables. Applied inside `_emit_once` before the dict lookup, so all
three callsites are protected without touching the guardrail-specific
callsite. Helper assumes acyclic input; `guardrail_mode` values are
built fresh from config (str enums, lists of str enums, TypedDict of
str/list-of-str), so no cycle can arise in practice.
Regression tests in `TestOpenTelemetrySpanDedupe` cover the list crash,
distinct-list-scope collision, dict and set scope parts, and an
end-to-end `_create_guardrail_span` exercise that confirms exactly one
`guardrail` span is emitted across repeated lifecycle entrypoints. Each
new test fails on a reverted helper (4/4 mutation kill)
* fix(otel): cap _freeze_for_dedupe recursion depth and ignore in recursive detector
CI's recursive_detector blocks new recursive functions in litellm/ unless they
are in the allowlist with a documented bound. Cap the helper at 16 levels and
return repr(value) past the cap; this is well past the realistic depth of
guardrail_mode (1-3 levels) and means a future caller passing a cyclic
container can no longer push the proxy logging path into a RecursionError.
Add a regression test that exercises the cycle path.
* refactor(otel): annotate _freeze_for_dedupe return as a HashableScope union
Per review feedback from @mateo-berri: replace the loose `-> object` annotation
with a recursive `HashableScope` union (str | int | float | bool | bytes | None
| Tuple[HashableScope, ...] | FrozenSet[HashableScope]) so the helper's contract
is visible at the signature. Replace the `try/except hash(value); return value`
passthrough with an explicit isinstance check over the hashable-scalar types so
the type checker can narrow without requiring `cast(Hashable, value)` on the
return. Symmetric: dict keys also flow through the freezer (a TypedDict key is
already a string in practice, so behaviorally identical). All 16 regression
tests still pass; mutation kill behavior preserved
* fix: avoid explicit casting
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution
MCP per-server IP access control reads the client IP from X-Forwarded-For
and trusts the leftmost entry. Behind an append-style proxy or load
balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy,
Cloudflare), a client can prepend an arbitrary value to the header, so the
leftmost entry is attacker-controllable even when the direct peer is a
trusted proxy. An attacker can therefore spoof an internal IP and reach
servers marked available_on_public_internet=false.
This adds an optional mcp_xff_num_trusted_hops general setting modelled on
Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries
from the right of the chain (where N is the number of trusted appending
proxies in front of the gateway) instead of the leftmost value, so any
entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges,
which still validates the direct peer, and only takes effect once that check
passes; without a validated direct peer the gateway keeps failing closed, so
hop counting cannot be abused by a direct-to-pod attacker. The chain must
contain at least N valid entries or resolution fails closed.
Default is unset, preserving existing behaviour.
* chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops
* fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum
A 0 or negative value is silently treated as disabled, which could leave
an operator believing they enabled append-style X-Forwarded-For hardening
while client IP resolution stays on the spoofable leftmost value. Emit a
warning, consistent with how the module already surfaces invalid CIDR
config, so the misconfiguration is visible in logs.
* fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time
Add a ge=1 bound to the ConfigGeneralSettings field so the
update_config_general_settings path rejects 0 and negative values with a
clear validation error instead of accepting them, and self-documents the
valid range. The runtime warning stays as defense-in-depth for raw-dict
config that bypasses model validation.
* style(mcp): black-format ip_address_utils.py
* fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid
A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1)
previously made _resolve_num_trusted_hops return None, which the caller
treated identically to "unset" and silently fell back to the legacy
leftmost X-Forwarded-For value. An operator who set the value to harden
client IP resolution but typo'd it would get weaker security than before,
with no fail-closed signal.
Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid,
_HopCount) so the three states are distinct: unset keeps the legacy path,
a valid count drives hop-counting, and an invalid value fails closed
(returns "") instead of reverting to the spoofable leftmost address. The
caller matches on the union exhaustively.
Add a parametrized regression test asserting get_mcp_client_ip returns ""
for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry,
and update the resolver unit tests for the new return type.
* fix(streaming): word-sliced cache replay for stream=true cache hits
* fix(streaming): align mypy and replay happy-path test with word-sliced cache replay
* fix(streaming): short-circuit whitespace-only content in cache replay splitter
* fix(streaming): emit tool_calls/function_call only on first replay slice
* refactor(streaming): drop dead delattr guard in cache replay
A non-None usage on the replay base object always lives in
__pydantic_extra__ (it is attached via setattr earlier in the same
function), so delattr can never raise here; the try/except AttributeError
that silently swallowed a failure was dead defensive code that could only
ever hide a real regression, so it is removed in both the async and sync
generators.
Also switches the new replay annotations from typing.List to the builtin
list to satisfy the strict ruff UP006 gate and drops the unused
PLR0915 noqa directives (the rule is not enabled in this repo's ruff
config, so RUF100 flagged them).
* fix(streaming): drop carried-over metadata from later cache replay slices
The word-sliced cache replay deep-copies the full ModelResponseStream per
slice, so reasoning_content, thinking_blocks, logprobs, enhancements,
annotations and the rest of the per-message metadata rode on every slice, not
just the first. Downstream handlers that accumulate streamed deltas would
collect each one once per slice, e.g. duplicating a cached reasoning trace N
times on a stream=true cache hit.
Later slices are now rebuilt as a content-only delta with choice-level logprobs
and enhancements stripped, so the whole metadata class stays on the first slice.
Adds async (logprobs) and sync (reasoning_content/thinking_blocks/logprobs/
enhancements, plus annotations) regression tests
---------
Co-authored-by: Mateo <277851410+mateo-berri@users.noreply.github.com>
An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the
user to sign in. On an unauthenticated initialize the gateway answered locally
(200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop
either connected empty or hit "OAuth probe timeout after 10000ms".
#30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers
to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth
servers, expecting the upstream to emit its own challenge. On initialize the
gateway never probes upstream, so no challenge ever reached the client.
Replace the `continue` with a preemptive 401 carrying the proxied
resource_metadata (RFC 9728) challenge, the same form passthrough servers and
MCPUpstreamAuthError already use. This keeps #29770 fixed (still no
authorization_uri) while restoring the upstream PKCE sign-in prompt.
* fix(mcp): resolve toolset tools by the server's known prefix
Toolsets store {server_id, bare tool_name} and reconcile that against the
live prefixed tool name at list time. The reconciliation chopped the live
name at the first MCP_TOOL_PREFIX_SEPARATOR with no server context, so a
server whose prefix contains the separator (a hyphenated alias, or the
UUID server_id used as the prefix when a server has no alias) had its
tools silently dropped from /toolset/<name>/mcp while listing fine
everywhere else. Strip the exact known prefix for the tool's server_id
instead of guessing the boundary, on both the resolve and filter sides
Also render toolset tools as {server-prefix}-{tool} in the dashboard
picker result and chips; this is display only, the persisted record
stays {server_id, bare tool_name}
Resolves LIT-3419
* test(mcp): add focused unit tests for strip_known_server_prefix
Cover the LIT-3419 cases directly on the helper with real MCPServer
objects: clean prefix round-trip, hyphenated alias, UUID server_id
fallback, unprefixed passthrough, and the server=None legacy fallback
* fix(mcp): warn loudly when X-Forwarded-For is present but use_x_forwarded_for is off
When a request carries an X-Forwarded-For header but use_x_forwarded_for is
unset, get_mcp_client_ip silently falls back to the direct peer's IP (the load
balancer / reverse proxy). That peer almost always sits inside
mcp_internal_ip_ranges, so the 'Internal network only'
(available_on_public_internet: false) restriction trusts every external caller
as internal and effectively exposes those servers.
Emit a one-shot loud error pointing the operator at use_x_forwarded_for instead
of hard-failing: on a deployment with no load balancer, a crafted
X-Forwarded-For header must not be able to take the service down, and a one-shot
log keeps a flood of crafted headers from spamming the logs.
* fix(mcp): re-arm XFF-disabled warning on config change and harden test assertion
Address PR review: tie the one-shot warning flag to the observed
use_x_forwarded_for value so it re-arms whenever the setting is seen enabled,
restoring the diagnostic on a later rollback to disabled. Also assert against
str(call_args) so the test survives a positional-to-keyword logger refactor.
* fix(mcp): correct misleading no-trusted-proxy warning for XFF access control
* test(mcp): assert the no-trusted-ranges warning was logged instead of relying on StopIteration
* fix(proxy): stop double-decrypting email/slack alerting env vars in get_config
proxy_config.get_config() already returns environment_variables decrypted
(the DB overlay decrypts them in _update_config_fields, and YAML values are
plaintext), so the /get/config/callbacks slack and email blocks were running
decrypt_value_helper() a second time on plaintext. That second decrypt always
failed and the helper swallowed the error and returned None, so every SMTP_*
field came back blank when the Admin UI reloaded the email settings, and the
proxy logged a misleading "Did your master_key/salt key change recently?"
error even when nothing changed.
Consume the already-decrypted values directly, matching process_callback's
handling of the same dict for langfuse/datadog/etc. Sensitive-value masking
is preserved.
Fixes#19221
* fix(proxy): preserve a cleared slack webhook instead of falling back to OS env
Use an explicit is-not-None guard rather than truthiness when deciding whether
to fall back to os.getenv for SLACK_WEBHOOK_URL. With `or`, a webhook the admin
cleared (stored as "") is falsy and would surface a stale SLACK_WEBHOOK_URL from
the OS environment; only a truly absent key should trigger the OS lookup. No
decryption is reintroduced.
Both tests were xfail(strict=True) for known proxy bugs: /team/new writing
budget_limits as a raw list (Prisma 500) and custom per-token pricing leaking into
the shared cost map for sibling deployments. Both are fixed, so the tests pass and
strict mode reports the unexpected pass as a failure. Remove the markers (as their
reasons instructed) so they run as plain regression guards; docstrings updated to
describe the regression each now pins.
* feat(proxy): add logging_endpoints package init
* feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out
* feat(proxy): register callback_logs_router
* test(proxy): add logging_endpoints test package init
* test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling
* refactor(proxy): move callback-logs request/response models to litellm/types/proxy
* refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging
* test(proxy): update callback-logs tests for class-based replayer and separated types
* fix(proxy): cover /v1/callbacks/ in backend component allowlist
The new /v1/callbacks/logs route was dropped by both component
allowlists, failing test_gateway_plus_backend_covers_full_app. It's an
admin-only spend-logging route, so it belongs on the backend (control
plane) alongside the existing /callbacks family.
* refactor(proxy): use builtin dict/list generics in callback-logs endpoint
Switch Dict/List from typing to builtin dict/list to satisfy the ruff
strict-rule budget (UP006).
* refactor(proxy): use builtin dict/list generics in callback-logs types
UP006: builtin generics over typing.Dict/List.
* chore(ui): regenerate schema.d.ts for /v1/callbacks/logs
Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/
CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the
dashboard types in sync with the proxy OpenAPI spec.
* fix(proxy): force stream=False when replaying callback logs
A replayed StandardLoggingPayload is a terminal, fully-aggregated event —
the producer (e.g. the rust realtime gateway) already collected the whole
session before POSTing. Marking the rebuilt Logging object as streaming made
async_success_handler wait for a complete_streaming_response that never
arrives, so the spend log was never written. Realtime sessions now land in
LiteLLM_SpendLogs.
* feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs
integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait
(base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger
— the first concrete logger, owning a bounded channel + background worker that
batches and POSTs to the Python proxy's /v1/callbacks/logs.
* feat(litellm-rust): RealTimeStreaming per-session log collector
1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id
per event (never buffers frames); log_messages() builds one StandardLoggingPayload
on session close and fans out to the CustomLogger callbacks. request_id == the
OpenAI realtime session id (sess_…), with the gateway id as fallback.
* feat(litellm-rust): wire realtime logging into the splice (lock-free observe)
The collector is owned on the splice task and observed via a synchronous &mut
callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic
on the per-frame hot path. On session close the bridge flushes one payload.
AppState carries the registered loggers; main spawns the proxy logger.
* docs(litellm-rust): ai-gateway realtime logging architecture
* docs(litellm-rust): document request-log egress to the LiteLLM control plane
Add a 'Request logging' guide to the ai-gateway README: how to point the gateway
at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the
admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session
behavior.
* feat(litellm-rust): make log-egress tunables env-overridable
Channel capacity, batch size, and flush interval now read from
LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS,
falling back to the DEFAULT_* consts on missing/invalid/non-positive values.
Grouped behind an EgressTunables::from_env() read once at logger construction.
* docs(litellm-rust): document log-egress tuning env vars
* docs(litellm-rust): require constants in a crate-level constants.rs
Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings
go in src/constants.rs, not inline in feature modules; env-overridable tunables
keep their DEFAULT_* value there.
* refactor(litellm-rust): move ai-gateway constants into constants.rs
Per the new rule: the log-egress defaults (proxy base, ingest path, channel
capacity, batch size, flush interval) and the realtime provider default move to
crates/ai-gateway/src/constants.rs; modules import from it.
* ci: run logging_endpoints tests in the proxy-infra coverage shard
tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading
job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on
codecov/patch despite being ~98% covered locally. Add it to proxy-infra's
test-path so the test is exercised under --cov.
* fix(litellm-rust): hash the master key before logging — never send the raw credential
Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which
fans out to spend logs and every callback (Langfuse/Datadog) and could be
recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's
hash_token); the field is named *_hash and the proxy stores it verbatim when it
isn't sk-prefixed, so the DB value is identical with zero plaintext exposure.
* fix(litellm-rust): observe realtime logging on upstream events only
Greptile P1: observe ran on the client->upstream arm too, so an authenticated
client could send a fabricated response.done and inflate its own spend log.
session.created/response.done are server->client events; observe the upstream
arm only.
* feat(proxy): bound callback-logs batch + return per-record failures
Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000,
env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and
return per-record {index, error} failures so a caller (the rust gateway) can
distinguish a transient callback error from a structurally bad payload.
* chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field
* fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant
It doesn't need to be env-configurable (only the rust egress tunables are). As an
os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires
every env key to be documented in the (separate-repo) config_settings.md. Plain
constant → not scanned → code-quality + documentation checks pass.
* docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes
* docs(litellm-rust): tighten the README request-logging section
* docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback)
* docs(litellm-rust): drop em-dashes from the request-logging section
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* test: multi-window budgets coverage
* fix: p0 issues, added types and shared functions for each test suite
* chore: add config.yml
* test: passthrough endpoints stream/non-stream e2e
* style: carry clearer status_code comparison into renamed e2e dir
* fix: rename cost breakdown function
* fix: pydantic validation for budget info, dont allow explicit type cast
* refactor: migrate to gateway client
* test: add custom pricing tests
* chore: change master key
* test(e2e): address greptile review feedback
Remove the duplicate cache/cache_params block in the gateway config so the two
can't silently diverge under future edits. Reorder the soft-budget test to assert
the call isn't a budget block before require_successful_call, since that helper
hard-fails any non-2xx and left the budget-block check unreachable; the misleading
"skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it
so a failed delete doesn't leak a budget on the shared proxy. Scope the
spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup
import so a broader "pytest tests/" run isn't left with a mutated path.
* test(e2e): drop misleading skip comment on require_successful_call
require_successful_call fails hard, it does not skip; the trailing
comment was factually wrong. The function name already states intent,
so the comment is removed in both per-model and tag budget helpers.
* test(e2e): assert budget-isolation invariant before success check
On the should-still-succeed path of the per-model and tag isolation
tests, check is_budget_block before require_successful_call. If the
isolation bug fires the unaffected model/tag is blocked, so asserting
the specific 'blocked by X' invariant first yields the diagnostic
message instead of a generic upstream-failure. Matches the ordering in
test_soft_budget_e2e.py.
* fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows
* fix(e2e): run case init() inside try so partial-init failures tear down
run_case called case.init() outside the try/finally that runs teardown(), so a
case that registers cleanups progressively (create team, then user, then key)
and then fails partway through init() would leak the already-created entities on
the long-lived shared proxy. Move init() inside the try so teardown always runs.
Add a regression test that registers a cleanup then raises mid-init and asserts
the resource is still released.
* test(e2e): mark known pricing-leak isolation test xfail(strict)
test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy
gap (a deployment's custom per-token pricing leaks into the shared cost map for
sibling deployments of the same underlying model) and was left unconditionally
failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True)
so the suite stays green while the leak persists and turns into a failure the
moment isolation is fixed, prompting the marker's removal.
* refactor(e2e): make suite pass its shipped strict basedpyright config
The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright
--project tests reported four errors in it: three reportAny on the parametrize
ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed
autouse fixture _require_live_proxy. Replace the untyped lambda with a typed
_case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and
rename the fixture to require_live_proxy so basedpyright no longer treats it as an
unused private function (it is referenced only by pytest's autouse machinery).
basedpyright --project tests now reports zero errors.
* fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory
* test(e2e): run harness unit tests without a live proxy
The autouse session fixture skipped the whole tests/e2e session when no proxy
answered, which also skipped test_lifecycle.py, a pure unit test of run_case that
never touches the proxy. A regression test that silently skips gives no signal,
so the skip now lives in pytest_runtest_setup gated on the same e2e marker the
spend-log truncate guard already uses: live tests skip when no proxy is up while
harness unit coverage always runs. The liveness probe is cached with lru_cache so
it still runs once per session
* test(e2e): clean up gateway config comment debris
Fix the typo on the header comment and drop the orphaned namespace/ttl
comment remnants left indented under cache_params; the active values are
already set above. Flagged by greptile review.
* fix: add new tests, split gateway
* test(e2e): type the redis spend-counter probe for strict basedpyright
The new cold-counter reseed test drove its redis client untyped, so the strict
tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the
file landed: scan_iter/get came back unknown and the pool.map lambda had an
untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING
import (the runtime import stays lazy so the suite still skips, not errors, when
redis is absent), which resolves scan_iter to Iterator[str] and get to str | None,
and replace the lambda with a typed inner function mirroring _burst. basedpyright
--project tests is back to zero errors.
* test(e2e): xfail the known team multi-window failure and isolate member teardown
Greptile flagged two issues in the mirrored split-gateway commit. The team
multi-window budget test documents a real /team/new write bug (budget_limits go
straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and
/team/update paths) and was left as an unconditional hard failure, which would
turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing
isolation test so the suite stays green while the bug persists and flips to a
failure the moment the write is fixed and the marker should go.
The class-scoped member fixture in test_team_member_budget_e2e.py tore down its
key, user, and team sequentially with no exception isolation, so a failed
delete_key would strand the user and team on the long-lived shared proxy. Route
cleanup through a ResourceManager: register each delete progressively and run them
LIFO best-effort in a finally, so a partial-setup failure still releases what came
before and one failed delete never blocks the rest.
* test(e2e): set fast budget-reset cadence in gateway config so staging windows reset within e2e timeouts
* test(e2e): surface real /spend/tags errors instead of masking them as missing tags
The spend-tracking e2e client swallowed every non-200 from /spend/tags into an
empty list, so a real server error or a response-shape mismatch showed up only as
the generic "tag never appeared in /spend/tags" with no diagnostics. That masking
is what made the original cluster failure undiagnosable.
spend_by_tags now raises SpendTagsError carrying the actual HTTP status and body
for any non-Success result, and poll_tag_spend fails fast on a hard server error
rather than polling it into a timeout; eventual consistency only manifests as a
200 whose payload does not yet carry the tag, so only that case waits. The tag
test now reports the last observed status and asserts the endpoint returned 200
at least once, with no weakened assertions.
Hardening surfaced the real defect in the test itself: /spend/tags returns a
top-level JSON array (List[LiteLLM_SpendLogs]), but the client validated against a
SpendTagsResponse dict wrapper that never matched, so every call fell through to
the empty-list mask. Wired spend_by_tags to the existing TagSpends RootModel and
removed the dead SpendTagsResponse model. Verified against the real Postgres that
request_tags is stored as proper JSONB arrays and /spend/tags aggregates them
correctly, so there is no encoding bug to fix here.
* test(e2e): drop flaky test_tag_spend_matches_sum_of_tagged_logs
The test wrote tagged requests and polled /spend/tags expecting read-after-write
consistency. /spend/tags itself is fine; verified live that request_tags is stored
as a JSON array and the endpoint reflects a fresh tag within seconds, so the
failures were a timing flake under full-suite load rather than a real defect.
Coverage is retained by test_request_tags_round_trip (tags persist onto the row)
and the /spend/tags route probe in test_spend_routes.py.
Also remove the now-dead tag-spend scaffolding this test was the only user of:
poll_tag_spend, spend_by_tags, TagSpendPoll, SpendTagsError, the TagSpend/TagSpends
models, and their imports.
* test(e2e): widen budget-reset wait windows to de-flake wall-clock-aligned resets
The short-window reset tests asserted the reset landed within WINDOW_SECONDS + 45
(~75s), but the 30s budget window is wall-clock-aligned, so the reset can land up
to a full window after start, then the rescheduler (~15-20s) zeroes the spend, plus
poll and DB lag. A real run measured 84s, just over the 75s bound, and which of the
short-window siblings tripped flipped run to run. Widen the wait loops to 150s and
the elapsed assertions to WINDOW_SECONDS + 90 (120s for the key test). A genuinely
stuck rescheduler is still caught by the wait-loop timeout, so this only removes the
timing flake, not the regression signal.
* test(e2e): let the spend-counter reseed test reach a cluster-mode TLS redis
The test's _redis() built a standalone, non-TLS client on the docker-compose
defaults (localhost:6380), so against the EKS serverless ElastiCache (cluster-mode
+ TLS) it could never connect and the test skipped. Honor E2E_REDIS_SSL and
E2E_REDIS_CLUSTER so it builds a TLS RedisCluster client when the deploy provides
them, and E2E_REDIS_NAMESPACE so the counter is read with a direct GET (cluster-safe)
rather than a keyspace scan that can't span shards. The local standalone path and the
graceful skip-on-unreachable behavior are unchanged.
* test(e2e): take the direct-GET spend-counter path on E2E_REDIS_CLUSTER
The gateway's cache sets no namespace, so the counter key is the bare
spend🔑<hash>. Trigger the cluster-safe direct GET on E2E_REDIS_CLUSTER (not
only on E2E_REDIS_NAMESPACE) so the cluster deploy need not set a namespace it
does not use; the namespaced key is still tried first when a namespace is given.
* test(e2e): use REDIS_HOST/REDIS_PORT and drop the unused redis knobs
The runner is a standalone test pod, so the proxy's own REDIS_HOST/REDIS_PORT
names are unambiguous - no E2E_ prefix needed. The only deployed redis it talks
to is the serverless ElastiCache (always TLS + cluster), so that is inferred from
REDIS_HOST being set rather than carried as ssl/cluster knobs. Stage sets no cache
namespace (bare counter key, read directly on the cluster) and is passwordless, so
the namespace and password env are gone; the local namespace is still handled by
the standalone SCAN.
* test(e2e): replace the vacuous failure-row test with per-model attribution
test_failure_call_writes_failure_status_row had two skip hatches (the call did
not fail, or no failure row landed) and never asserted anything on this proxy -
gemini accepts an empty message (HTTP 200), and live failure-row logging is
non-deterministic across providers. Replace it with a deterministic check: one
key calling gemini-2.5-flash and claude-haiku-4-5 gets one spend row per call,
each carrying its own model and a nonzero cost, under distinct request_ids that
match the call's response id. Verified live on stage (gemini/gemini-2.5-flash
$0.00053, anthropic/claude-haiku-4-5 $0.000038, distinct ids matching the
responses). Failure-status row construction stays covered by the unit suite.
* test(e2e): assert /spend/logs returns the key's spend without 5xx
Regression for the intermittent 500s on /spend/logs (DB query / serialization
errors under load). The existing spend_logs() helper swallows non-success
responses into an empty list, so a 500 looks identical to 'rows not flushed yet'.
This test queries the endpoint directly and asserts a Success response on every
poll, failing loudly on any 5xx, then requires the call's nonzero spend to surface.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(mcp): add v1 bridge + none/api_key resolver arms (unwired)
PR4a of the MCP v2 outbound-credential migration, stacked on the resolver skeleton.
Builds the bridge for the first live modes without wiring it onto the request path:
- resolver.py: the none arm (NoOpAuth) and the api_key shared-key arm (StaticHeaderAuth
from the config); the BYOK source and the other five arms stay not_implemented.
- adapter.py: the v1 <-> v2 edge (to_subject, to_server_spec, raise_public, should_defer).
to_server_spec maps only none + the static-header family and returns None to defer every
other mode to v1. Imports v1, kept out of the package __init__ so the resolver core stays
v1-free.
- MCPClient gains an optional resolved_auth that feeds the factory's auth= slot, taking
precedence over the SigV4 aws_auth; default None keeps current behavior.
Nothing calls these from _create_mcp_client yet, so production behavior is unchanged; the
graft lands in PR4b. Unit tests cover the two arms, the full mapping table, and the auth
plumbing.
* feat(mcp): graft v2 resolver onto _create_mcp_client for migrated modes
Wire the none + api_key static-family resolver arms from PR4a onto v1's
live request path. In _create_mcp_client's HTTP/SSE branch, to_server_spec
decides per mode: a migrated mode resolves through the injected
UpstreamCredentialProvider and feeds the resulting httpx.Auth into the new
resolved_auth slot; every other mode returns None and falls through to the
unchanged v1 construction. resolve_mcp_auth now runs only when the mode
defers, so a migrated server skips the v1 token-exchange / M2M I/O.
stdio is untouched: auth_type/auth_value never reach the upstream on the
stdio path (_get_auth_headers is HTTP/SSE only), so there is nothing to
graft there. No v1 code is deleted yet; resolve_mcp_auth's static return
still backs stdio and the not-yet-migrated modes until later PRs retire it.
* test(mcp): cover the v2-resolver graft in _create_mcp_client
Regression tests for the PR4 graft. Migrated HTTP modes resolve through the
provider into resolved_auth: none -> NoOpAuth, and the static api_key family
emits the right header per scheme (X-API-Key, Bearer, token, raw authorization,
base64 basic). Deferred modes (oauth2) and a missing static token fall back to
v1's auth_value. A stdio server with a migrated auth_type still defers to v1,
since httpx.Auth never reaches the subprocess. A resolver Error is mapped to the
public HTTP contract (401) via an injected provider, exercising the DI seam.
* fix(mcp): defer to v1 when an inbound credential would be overridden
The graft attaches the resolved static credential as an httpx.Auth, whose auth
flow writes its header after extra_headers. That silently overrode an inbound
Authorization: a per-request mcp_auth_header override, or a header supplied via a
guardrail hook / static_headers / forwarded caller header. v1 lets those win, so
the graft had inverted the credential precedence for the migrated static modes.
Mirror the v2 egress credential-isolation invariant: defer the request to v1 when
mcp_auth_header is set, or when the header the resolved credential would write is
already present in extra_headers. none writes no header, so it never defers.
* test(mcp): cover the credential-isolation defer guard
Regression tests for the precedence fix. A per-request mcp_auth_header override and an
Authorization already present in extra_headers (guardrail hook like the JWT signer,
static_headers, or a forwarded caller header) both defer a migrated static server to v1
so the inbound credential wins; none stays on v2 and does not clobber an inbound
Authorization since NoOpAuth writes nothing. The deferred cases assert resolved_auth is
None, which fails if the guard is removed.
* refactor(mcp): resolve inbound-header conflict on v2 instead of deferring
For an Authorization already supplied via extra_headers (a guardrail hook such as the
JWT signer, static_headers, or a forwarded caller header), keep the request on the v2
path and skip resolved_auth rather than deferring to v1. The inbound header still wins
since nothing overwrites it, but hooks no longer pin a v1 fallback, which is what lets
resolve_mcp_auth be retired once the remaining modes migrate.
The mcp_auth_header per-request override still defers to v1, since that value becomes
the upstream credential rather than sitting in extra_headers; that defer falls away
once the per-user modes stop writing mcp_auth_header.
* fix(mcp): clear UP037 lint gate and fix allowed-servers test under the graft
adapter.py uses `from __future__ import annotations`, so the quoted "UserAPIKeyAuth" /
"MCPServer" annotations in to_subject/to_server_spec/_shared_key_spec were unnecessary
and pushed UP037 over the strict-rule budget; drop the quotes.
test_list_tools_only_returns_allowed_servers passed a MagicMock as user_api_key_auth.
The graft now builds a Subject from the principal, and the MagicMock's non-string
org_id/user_id fail Subject validation, so the listing came back empty. Use a real
UserAPIKeyAuth instead (MagicMock for an injected dependency was the anti-pattern here).
* test(mcp): assert config token via resolved_auth, not the headers dict
test_mcp_server_config_auth_value_header_used inspected _get_auth_headers(), but the
graft now carries the static credential on the client's httpx.Auth (resolved_auth) and
writes the header at send time, so that dict is empty. Assert the header the
StaticHeaderAuth emits onto the request instead. Both config keys (authentication_token,
auth_value) stay covered.
* chore(typecheck): set reportMatchNotExhaustive slack to 0
The previous slack of 3 put the ceiling at baseline + slack = 4, so a newly
non-exhaustive match (for instance dropping an Error arm off a Result match)
could land without tripping the gate. Setting slack to 0 pins the ceiling at
the current baseline of 1, so any added non-exhaustive match now fails CI while
the one pre-existing violation in router.py stays within budget
Creating or updating a key with a specific (non-allow_all_keys) MCP
server or access group failed with a 403 when the key had no team:
Key is not in a team. Only globally available (allow_all_keys) MCP
servers can be assigned
validate_key_mcp_servers_against_team computed the allowed set as
team servers + allow_all_keys servers. For a teamless key the team
set is empty, so the allowed set collapsed to just allow_all_keys
servers and any explicitly-picked server or access group was rejected.
This was asymmetric with runtime: get_allowed_mcp_servers honors a
teamless key's own object_permission.mcp_servers verbatim, with no
team gate and no allow_all_keys filter. So the create/update path
refused to persist a grant the run path would have served.
Thread is_proxy_admin into the validator from both call sites
(/key/generate and /key/update). When a key has no team and the
caller is a proxy admin, the requested servers and access groups are
folded into the allowed set so the existing subset checks pass. A
proxy admin can already reach every MCP server, so there is nothing
to escalate. Non-admins and every team-scoped key are unchanged.
Resolves LIT-3815
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
A user provisioned with "All Proxy Models" stores the literal
"all-proxy-models" sentinel in user.models. get_direct_access_models looked
that string up as a real model_name via get_model_list, which matched no
deployment, so /v2/model/info marked every model direct_access=false and the
Models + Endpoints page rendered empty for such users when they have no teams.
The model dropdown / Playground worked because get_key_models already expands
the sentinel to the full proxy model list, hence the inconsistency in the
report.
Expand the sentinel to all non-team deployment ids via
get_model_ids(exclude_team_models=True), the same call the PROXY_ADMIN branch
in the caller already uses. This fixes both /v1/model/info and /v2/model/info
since they share _populate_team_access_on_models. Empty user.models stays "no
direct access" to match get_key_models semantics.
Fixes#22791
* fix(anthropic): drop unsupported speed param with drop_params
Anthropic fast mode (speed) is Opus 4.6/4.7/4.8 on the direct API only.
Strip speed when the model map lacks supports_speed and drop_params is set,
for both chat completions and /v1/messages passthrough.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): allow supports_speed in model map schema
The new supports_speed flag on Opus entries must pass JSON schema
validation in test_aaamodel_prices_and_context_window_json_is_valid.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(review): raise on unsupported speed without drop_params
Passthrough /v1/messages now raises UnsupportedParamsError when speed
is unsupported and drop_params is false. Emit drop warning from
map_openai_params when speed is silently skipped.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic): gate speed param by routed provider, not just model id
Vertex, Azure, and Bedrock reuse the shared Anthropic transform and strip
their provider prefix first, so a bare `claude-opus-4-8` resolved to the
direct-API model-map entry (`supports_speed: true`) and forwarded `speed`
upstream, producing the same 400 that drop_params is meant to prevent.
Gate fast mode on `custom_llm_provider == "anthropic"` so it stays on the
direct Anthropic API across both the chat completions and `/v1/messages`
passthrough paths, and collapse the duplicated drop/raise logic in
map_openai_params into the shared `_maybe_drop_speed_param` helper.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The per-user BYOK, OAuth (OBO), and env-var management endpoints resolved the
target MCP server through a DB-only lookup (get_mcp_server / get_all_mcp_servers_for_user).
A server defined in config.yaml lives only in the in-memory registry and never
gets a row in LiteLLM_MCPServerTable, so those endpoints raised 404 "MCP Server
<id> not found" (or 403 for non-admins) before any credential could be stored,
leaving config-server users unable to connect and forced to re-authorize forever.
Route all three through a single registry-aware resolver: DB first, then the
in-memory registry (built into LiteLLM_MCPServerTable via _build_mcp_server_table,
the same fallback fetch_mcp_server already uses), then the canonical
get_allowed_mcp_servers authorization the MCP gateway enforces on tool calls.
Admins get a 404 for an unknown id; non-admins get 403 for a missing-or-forbidden
server so server ids stay non-enumerable. This also closes a gap where the two
store endpoints performed no per-server authorization at all.
The bedrock/mantle chat and messages paths hardcoded the public Mantle host and ignored api_base, so private VPC/VPCE/GovCloud endpoints could not be used. Route URL construction through a shared helper that prefers api_base and aws_bedrock_runtime_endpoint before falling back to the regional public host.
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(model_prices): correct regional processing uplift assignment
gpt-4.1, gpt-4o, gpt-5, and their variants were incorrectly carrying
the 10% EU/US regional processing uplift multiplier. Per OpenAI's
pricing docs, the uplift applies only to models released on or after
2026-03-05 (gpt-5.4 series and gpt-5.5 series).
Removes the uplift from: gpt-4.1, gpt-4.1-mini, gpt-4.1-nano,
gpt-4o, gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4o-mini, gpt-5,
gpt-5-pro, gpt-5-mini, gpt-5-nano.
Adds the uplift to: gpt-5.4, gpt-5.4-mini, gpt-5.4-nano, gpt-5.4-pro,
gpt-5.5, gpt-5.5-pro.
* fix(model_prices): apply same regional uplift correction to backup file
* fix(model_prices): add regional uplift to date-versioned gpt-5.4/5.5 siblings
* test(model_prices): update data residency tests to use gpt-5.4 as the uplift model
The tests were using gpt-5 which no longer carries the regional processing
uplift after correcting which models have it. Switch to gpt-5.4 (released
2026-03-05, the cutoff date) and add a regression parametrize covering
all pre-cutoff models to pin that they stay uplift-free.
* test(batches): use gpt-5.4 for data residency uplift assertion
batch_cost_calculator's data residency uplift test still pinned gpt-5,
which no longer carries the regional processing uplift after this change.
Switch it to gpt-5.4 (the canonical post-cutoff uplift model), matching
the llm_cost_calc test update.
---------
Co-authored-by: mgalbato <37748295+mgalbato@users.noreply.github.com>
