- _build_decode_kwargs no longer suppresses the unscoped-fallback warning
when LiteLLM_JWTAuth.issuers is set: tokens whose iss does not match
any configured issuer still fall through to the global path, and that
fallback is itself unscoped when JWT_AUDIENCE/JWT_ISSUER are absent.
- _get_agent_object_permission now caches the agent_id ->
object_permission_id mapping and delegates the permission lookup to
the shared get_object_permission helper, so the agent path reuses the
same cache entries as the org / team / key paths.
The new elif passthrough cold-start branch reused is_auth_error which
matches both 401 and 403. A 403 from user_api_key_auth indicates the
LiteLLM key WAS recognized but is forbidden (e.g. over budget / rate
limited); falling through to anonymous UserAPIKeyAuth() in that case
bypasses spend and rate-limit controls on passthrough servers.
Only trigger the cold-start anonymous admission on 401, which is the
signal that the bearer is an upstream OAuth token rather than a
recognized LiteLLM key.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- _parse_mcp_server_names_from_path now delegates to
MCPRequestHandler._extract_target_server_names_from_path so the
names used by the cold-start passthrough bypass cannot drift from the
names used by downstream routing.
- JWTIssuerConfig now rejects the combination of audience and
disable_audience_validation=True at validation time instead of
silently ignoring the flag.
The aggregator for team IDs only consulted the issuer-normalized claim
for the plural (team_ids) path and fell back to the global config for
the singular path. When an operator configures team_id_jwt_field only
at the issuer level, get_team_id correctly returned the mapped value
but get_all_jwt_team_ids silently dropped it, causing membership
reconciliation to disagree with request routing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Extracted the security-sensitive logic that decides whether the caller's
Authorization header is forwarded to (or stripped from) an outgoing MCP
request into a single helper, _should_strip_caller_authorization, in
mcp_server_manager.py.
Previously the same condition was duplicated across
_call_regular_mcp_tool (mcp_server_manager.py) and
_prepare_mcp_server_headers (server.py). Keeping two copies of this
check risked future divergence and credential-leak / broken-passthrough
bugs. Both call sites now share the helper, preserving exact behavior.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Mirror the OAuth pass-through admission check from _prepare_mcp_server_headers
(list-tools path) in _call_regular_mcp_tool (tool-call path): when the server
is OAuth pass-through and the caller did not supply x-litellm-api-key,
Authorization on the inbound request may itself be the LiteLLM API key — so
strip it before forwarding instead of leaking the gateway credential upstream.
When x-litellm-api-key is present, admission is unambiguous and Authorization
continues to carry the upstream OAuth bearer (transparent pass-through).
- Normalize the WWW-Authenticate header key emitted by
_check_passthrough_upstream_auth to lowercase to match the other 401
emitters in the OAuth pass-through flow.
- Mirror the streamable HTTP handler's toolset scoping in handle_sse_mcp:
strip client-supplied x-mcp-toolset-id and apply _apply_toolset_scope
before _check_passthrough_upstream_auth so the upstream probe list is
derived from the fully-authorized server set.
- Tighten _has_client_supplied_mcp_auth signature so
mcp_server_auth_headers is Optional, matching its caller in
process_mcp_request.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
For pass-through MCP servers (auth_type=none with delegate_auth_to_upstream)
the RFC 9728 cold-start flow sends the client back with only
"Authorization: Bearer <upstream-token>" after upstream OAuth discovery.
Previously this path 1) was rejected in process_mcp_request because the
oauth2_headers fallback only covered auth_type=oauth2 targets, and 2) had
the Authorization header stripped by _prepare_mcp_server_headers when no
x-litellm-api-key was present, treating the upstream token as a potential
LiteLLM key leak.
- Extend the elif oauth2_headers fallback to also admit anonymously when
every target is a pass-through server.
- Pass user_api_key_auth into _prepare_mcp_server_headers so it can
forward Authorization for pass-through servers when admission did not
consume the bearer as a LiteLLM key (api_key is unset).
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- Move JWTIssuerConfig audience-required guard into a Pydantic model_validator
so misconfiguration fails at startup instead of on the first request.
- Replace the string-match `No matching public key found` filter in
get_public_key's multi-URL fallback with a dedicated
NoMatchingJWTPublicKeyError; only that specific exception triggers
continuation, every other error still surfaces.
Ensure exceptions from get_object_permission in _get_org_object_permission are caught and return None, preserving the original fail-safe semantics.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Replace the bespoke litellm_organizationtable.find_unique + dedicated
cache key in _get_org_object_permission with get_org_object +
get_object_permission so MCP requests share the same user_api_key_cache
entries as the rest of the proxy and no longer fragment org-row caching.
Sameer's review on #28356/#28008 flagged that the new pass-through
behaviors (preemptive 401 challenges, /.well-known/oauth-protected-
resource proxying, upstream 401/403 propagation as MCPUpstreamAuthError,
and Authorization-stripping when no x-litellm-api-key is supplied)
were implicitly enabled for every server with auth_type=none plus
Authorization in extra_headers. Existing users doing static bearer
pass-through for non-OAuth reasons would have silently regressed.
Make the detection rule explicit: extend the existing
delegate_auth_to_upstream flag (previously oauth2-only) to also gate
is_oauth_passthrough. Now requires flag + auth_type=None + Authorization
in extra_headers, per Sameer's suggested detection rule. The UI toggle
now appears for both modes (oauth2 PKCE passthrough and auth_type=none
OAuth pass-through) with mode-appropriate copy.
Update test fixtures to set the flag where the test intent is to
exercise OAuth pass-through behavior, and add negative tests covering
the new default-false case.
JWTHandler() can be constructed without update_environment() being
called (tests do this directly), in which case self.litellm_jwtauth
does not exist. Accessing it raises AttributeError before getattr can
fall back. Use the same safe pattern other call sites use.
Update the tests carried over from PR #28008 to match the assertions on
the staging branch:
- tests/test_litellm/proxy/auth/test_handle_jwt.py: unknown issuers now
fall back to the legacy JWT_PUBLIC_KEY_URL path (per
litellm_feat/v1.84.0-mcp-gateway-jwt-auth's
'\''fall back to global JWKS on unknown issuer'\''), and mapped issuer
claims that are absent no longer fail closed — they simply leave the
normalised LiteLLM internal claim absent.
- tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py:
the aggregate '\''/mcp'\'' route still triggers the delegate-auth-to-upstream
lookup once for the header-supplied server name; cold-start admission
must NOT fire on top of that. Tighten the assertion to
assert_called_once_with so a future regression that re-enters cold-start
is caught.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
- LiteLLM_JWTAuth.issuers docs now state explicitly that unlisted
issuers fall back to the global JWT_AUDIENCE/JWT_ISSUER path; the
field is additive routing, not an allow-list. Matches actual
control flow in handle_jwt.auth_jwt and the regression tests
asserting backwards compatibility with the global JWKS path.
- MCPRequestHandler._get_{org,agent}_object_permission now pass
ttl=DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL on async_set_cache,
mirroring the auth_checks.py pattern so the cache TTL is explicit
on both DualCache layers.
- _build_oauth_authorization_server_response: revert to sync (no awaits in body).
The function only does dict construction and synchronous registry lookups;
async added coroutine creation overhead per discovery call without need.
- _build_decode_kwargs: accept has_issuer_config so the global path's
'JWT auth is unscoped' warning is suppressed when LiteLLM_JWTAuth.issuers
provides per-issuer scoping. Previously the warning fired spuriously for
admins who intentionally use only the new issuers config.
Threading client_ip into _target_servers_delegate_auth_to_upstream
made get_mcp_server_by_name(name, client_ip=...) also fire from the
delegate-auth check, so the call_args_list assertion on
client_ip-in-kwargs no longer uniquely signals a cold-start lookup.
Patch _is_mcp_passthrough_cold_start and assert it is not invoked,
which is the actual contract the test is pinning.
* test(proxy_behavior): scaffold session-scoped async ASGI client + liveness smoke
Slice 2 of the management-endpoints behavior-pinning effort. New top-level dir
tests/proxy_behavior/management/ outside every existing pytest glob.
conftest.py initialises the proxy app once per session against the DATABASE_URL
the harness boots Postgres at, wraps it in httpx.AsyncClient via in-process
ASGITransport. The one smoke test asserts /health/liveliness returns 200, which
exercises the full FastAPI middleware stack against a real app — no mocks.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): connect prisma via real lifespan; key/generate de-risk
Slice 3 of the management-endpoints behavior-pinning effort. The fixture now
enters the real FastAPI lifespan (proxy_startup_event) instead of just calling
initialize() — that is where prisma_client is connected, password migration is
kicked off, and the rest of the startup wiring runs.
Tests pin the loop to the session scope so the AsyncClient created in the
session fixture and the prisma connection opened in the lifespan share the
same loop as the test bodies.
New de-risk smoke: POST /key/generate with the master key returns 200, the
returned sk- token resolves to a hashed row in LiteLLM_VerificationToken, and
the cleartext token is never stored. Proves auth + handler + helper + prisma
all wire together end-to-end against a real Postgres.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): seed 8-actor read-world for the authz matrix
Slice 4 of the management-endpoints behavior-pinning effort. New
``actors.py`` defines the actor enum + seeds an immutable world (2 orgs,
2 teams, 8 users, 8 verification tokens) under the ``behavior-pin-``
prefix so the rows are identifiable in psql and ``_wipe_world`` is
targeted.
Each actor key is created with its cleartext form generated locally and
its hashed form (via ``litellm.proxy.utils.hash_token``) stored in
``LiteLLM_VerificationToken`` — so the real ``user_api_key_auth`` accepts
the cleartext bearer token. Roles, ``team_id``, ``organization_id``, and
the service-account metadata flag are all set on the seeded rows so the
auth layer resolves the same scopes a real proxy would.
The session-scoped ``world`` fixture re-seeds at session start (idempotent
via wipe-then-create), and the smoke test confirms each of the 8 actor
keys can call ``/key/info`` on itself and receive its own row back.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): per-test scratch namespace + targeted delete_many teardown
Slice 5 of the management-endpoints behavior-pinning effort. Adds the
``scratch`` function-scoped fixture: each test gets a uuid4-derived
namespace prefix, tags writes with it (``key_alias``, ``team_alias``,
``user_id``, ``budget_id``), and the fixture teardown ``delete_many``-s
any row whose namespace column starts with that prefix.
Cleanup uses Prisma model methods only (no raw SQL, per CLAUDE.md) and
orders deletes children-before-parents to avoid FK conflicts. The Slice 3
de-risk smoke is migrated onto the same fixture so it stops accumulating
untagged tokens across repeated local runs.
Smoke proves both halves of the contract: one test writes a scratch-tagged
key and asserts it lands; a second test runs after the first's teardown
and asserts no rows in the scratch namespace survived.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): codify G3 (strict-import grep) as a pytest item
Slice 6 of the management-endpoints behavior-pinning effort. Two new tests
walk every .py file under tests/proxy_behavior/ and assert:
* no ``from litellm.proxy.management_endpoints`` import — the suite is
deliberately constrained to the HTTP boundary so it survives handler
refactors;
* no ``mock``/``patch`` on ``user_api_key_auth`` — mocking auth is the
structural failure mode of the existing 11k-line mock suite, and the
point of this harness is that the real auth layer runs.
Codifying G3 as a CI test removes the "did someone forget to check the
PR-description checklist" failure mode.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* style(proxy_behavior): apply black to G3 grep test
Follow-up to 6f588c753b — line-length fixes only, no behavior change.
* test(proxy_behavior): pin /key/generate authz matrix (18 scenarios)
Slice 7 of the management-endpoints behavior-pinning effort. Parametrized
matrix across two axes: actor (8 seeded) × target scope (self, team_alpha
in org_a, team_beta in org_b). 18 scenarios after dropping non-applicable
combos. Whole-suite wall-time stays at ~4.7s (well under the 10-min G2
budget for the eventual CI job).
While pinning, the test surfaced one seed gap: ``_get_user_in_team`` reads
``members_with_roles`` (a JSON list of ``{user_id, role}``), not the plain
``members`` String[]. Both columns are now populated in the seed to match
what the real ``/team/new`` handler would produce.
Expected status codes are intentionally heterogeneous (200, 400, 401)
because the current handler emits different statuses depending on which
check fails first (role gate, team-member-perm gate, "not assigned"
check). Pinning the *observed* codes — not what they "should" be — is
exactly the regression signal we want.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/info authz matrix (24 scenarios)
Slice 8 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target keys (own, OWNER's key in org_a, CROSS_ORG_USER's key in org_b)
covering self-read, same-team-peer read, and cross-org read.
Notable pinned behaviors (intentionally surfaced for review, not "fixed"):
* ORG_ADMIN gets 403 on individual key info even within their own org
— visibility is scoped to "your own keys" + "your team's keys", not
"your org's keys".
* Same-team peers (INTERNAL_USER, UNRELATED_SAME_ORG, SERVICE_ACCOUNT)
DO see each other's keys. Whether that is desired is for the team
to decide; this PR only pins the existing behavior so unintentional
changes flip the matrix red.
Wall-time is unchanged (~4.3s for the slice on its own).
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/list default-visibility matrix (8 scenarios)
Slice 9 of the management-endpoints behavior-pinning effort. For /key/list
the response IS the matrix: each of the 8 seeded actors calls the endpoint
with default filters and the test asserts set-equality between the returned
visible-token set (filtered to seeded tokens only, so unrelated rows can't
flap the assertion) and a pinned expected actor-set.
Pinned default visibility:
* PROXY_ADMIN sees all 8 actors' keys.
* Every other actor sees only their own key — including ORG_ADMIN
(which had broader expectations going in but currently behaves
same-as-internal-user for /key/list defaults) and TEAM_ADMIN (no
team-aggregation without include_team_keys=true).
Future changes that broaden or narrow any single actor's default
visibility will turn this matrix red — exactly the regression signal we
want. Parameter-driven views (include_team_keys, filters) are deferred to
Slice 13 / PR2 follow-up.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/update authz matrix + mutation re-read (21 scenarios)
Slice 10 of the management-endpoints behavior-pinning effort. 8 actors ×
3 target shapes (self-owned, OWNER-scoped in org_a/team_alpha,
CROSS_ORG_USER-scoped in org_b/team_beta) = 21 applicable scenarios.
Each test:
1. Master-key-seeds a fresh scratch key with the target's (user_id,
team_id) scope (so the read-world stays untouched).
2. Has the actor under test POST /key/update flipping ``models`` to
a known marker list.
3. Asserts the status code AND the DB row's ``models`` field — present
when 200, unchanged otherwise — so a handler that silently mutates
on a denied response surfaces red.
Observed gating (pinned, not endorsed):
* PROXY_ADMIN bypasses every check.
* ORG_ADMIN is blocked by an early role gate, always 401.
* Every other (INTERNAL_USER-rolesed) actor hits one of three failure
modes — 403 "user can only create keys for themselves", 403
"only proxy admins, team admins, or org admins", or 401
"team_member_permission_error" — depending on whether they own the
target and whether they're a team admin / member of its team.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/regenerate authz matrix + rotation contract (22 scenarios)
Slice 11 of the management-endpoints behavior-pinning effort. 21 matrix
scenarios (8 actors × 3 target shapes, minus the cross_org/owner combo
that exists in the seed but isn't applicable) plus one smoke for the
``/key/{key:path}/regenerate`` route registration.
On 200 outcomes the test verifies the full rotation contract:
* the regenerate response key differs from the old cleartext,
* the OLD cleartext returns 401 on a follow-up ``/key/info``,
* the NEW cleartext returns 200 on a follow-up ``/key/info``.
On denied outcomes the test verifies the OLD cleartext still works —
catching any handler that mutates the token row on a failed call.
Pinned authz divergence vs /key/update: regenerate routes most denials
through the team-member-perm 401 path rather than the role-gate 403
path. The matrices for both endpoints are now in tree side-by-side, so
any future refactor that "harmonises" the codes will turn one of the two
red.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* test(proxy_behavior): pin /key/delete authz matrix + post-delete contract (21 scenarios)
Slice 12 of the management-endpoints behavior-pinning effort. Mirrors
slices 10/11. On success: cleartext can no longer authenticate
(handles both hard-delete and soft-delete to LiteLLM_DeletedVerificationToken).
On denial: row survives and cleartext still authenticates.
Notable behavior gap with /key/update: same-team peers (internal_user,
unrelated_same_org, etc.) get 403 on /key/delete for OWNER's key — i.e.
cannot delete each other's keys — whereas they CAN read each other's
keys (Slice 8). Delete is stricter than read. Pinned as-is.
Cumulative whole-suite wall-time is 5.9s for all 128 tests on the local
runner — well under the 10-min G2 budget for the CI job in Slice 13.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* ci(proxy-mgmt-behavior): add PR-triggered workflow for the behavior suite
Slice 13 of the management-endpoints behavior-pinning effort. New
workflow ``test-unit-proxy-mgmt-behavior.yml`` fires ``on: pull_request``
for the same branch set every other proxy unit-test workflow watches
(main, litellm_internal_staging, litellm_oss_branch, litellm_**).
It delegates to the existing reusable ``_test-unit-services-base.yml``
with ``enable-postgres: true``, which already provisions a postgres:14
service container and runs ``prisma db push`` against it before pytest
collects. ``reruns: 0`` because a behavior-pinning matrix that needs
reruns is itself a regression — flakes are signal.
``timeout-minutes: 15`` gives generous headroom over the local 5.9s
whole-suite wall-time; the binding G2 budget is 10 min.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* docs(proxy_behavior): G4 regression-replay table for Key Tier-1
Slice 14 of the management-endpoints behavior-pinning effort. Documents
the regression-replay verification methodology + a 12-row table mapping
recent fix-PRs touching key_management_endpoints.py to the catching
scenarios in the PR1 matrix.
One canonical RED→GREEN cycle is captured verbatim — c7c3df2b02
"extend /key/update admin check to non-budget fields". Under the
parent-of-fix code, 6 scenarios in test_key_update.py flip from 200 to
403; under HEAD code, all 21 pass. The handler swap is the only change
between the two runs, confirming the matrix catches the behavior shift
the fix introduced.
The table also calls out 4 genuine coverage gaps deferred to PR2/PR3:
404-on-missing-key, budget-limit counter assertions, /key/regenerate
upperbound enforcement, and /key/list filter-param views.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* chore(mutmut): include the behavior suite in tests_dir + G5 triage stub
Slice 15 of the management-endpoints behavior-pinning effort. Appends
``tests/proxy_behavior/management/`` to ``[tool.mutmut].tests_dir`` so
the existing mutation-test workflow runs against both the legacy mock
suite AND the new behavior suite — the latter is where the regression
signal will actually surface.
Adds a stub at ``tests/proxy_behavior/management/mutmut_triage/pr1.md``
documenting the G5 triage protocol (zero unreviewed survivors in the 6
Tier-1 handler functions) and a placeholder baseline-metrics table to
fill in after the first manually-triggered mutmut run completes — runs
take hours and run on a manual cadence, so PR1 ships with the wiring +
protocol, not the numbers. The actual baseline is recorded in a
follow-up once ``gh workflow run mutation-test.yml`` finishes.
The kill rate stays telemetry-only, never a gate. G5 (per-survivor
classification) is the binding mutation gate.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* docs(proxy_behavior): suite README with local-repro + conventions + gates
Slice 16 of the management-endpoints behavior-pinning effort. The README
documents:
* The same three commands the CI workflow runs locally (BYO-DATABASE_URL,
no new tooling).
* Suite layout — what each test file covers, which slice it lands.
* The asyncio loop_scope convention required for session fixtures
(httpx AsyncClient + prisma connection) to share a loop with each
test body.
* G3 strict-import convention + the test that enforces it.
* Read-world vs scratch-world fixture conventions.
* Behavior-pinning philosophy: pin observed codes; flag, don't judge.
* Where each G1–G5 + PR1.M1–M3 gate's evidence lives.
Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d
* ci(proxy-mgmt-behavior): drop xdist (workers=0) to fix seed race
First run on PR #28321 failed with UniqueViolation on
``behavior-pin-budget`` plus cascading missing-membership FK errors. Both
xdist workers entered ``seed_world()`` concurrently against the shared
Postgres service container; whichever lost the race left the world in a
half-seeded state and downstream tests ran against missing
team_membership rows.
Whole-suite wall-time is ~7s sequentially, so disabling xdist here costs
nothing — and the seed itself is the wrong place to add per-worker
isolation (the world is intentionally shared so set-equality assertions
in /key/list have a deterministic expected set).
* ci(proxy-mgmt-behavior): seed scratch keys via proxy_admin actor, not master
Second CI run failed: ``/key/generate`` with explicit ``user_id`` returned
403 "User can only create keys for themselves. Got user_id=X, Your ID=None"
in every test that called ``_create_scratch_key`` with a per-actor user_id.
The bare master key's auth path was producing ``user_id=None`` in the
fresh CI Postgres, which doesn't trigger the PROXY_ADMIN bypass in
``_user_can_only_create_keys_for_themselves`` reliably. Locally the same
master key path worked, masking the issue.
Fix: every ``_create_scratch_key`` helper now takes a seeder cleartext
and the test bodies pass ``world.keys[Actor.PROXY_ADMIN].cleartext``.
That actor was seeded with ``user_role=PROXY_ADMIN`` AND a concrete
``user_id``, so the bypass fires deterministically in both environments.
No behavior shift in the matrices themselves — all 128 scenarios still
pass locally; only the setup helper's auth identity changed.
The bare-master smoke (test_smoke + test_scratch_teardown) is intentionally
left on the master key path: those tests don't pass ``user_id`` in the
body so they don't hit the user_id-mismatch gate.
* ci(proxy-mgmt-behavior): diag — run world-seed test first + bump max-failures
Third CI run failed identically: seeded PROXY_ADMIN actor's auth resolves
to ``user_id=None`` even though the DB row has the right ``user_id``. The
suite was aborting at maxfail=10 inside test_key_delete, so test_world_seed
(which would tell us whether the seed itself is reachable) never ran in CI.
Two diagnostic moves on this push, no behavior change:
* Rename ``test_world_seed.py`` → ``test_aaa_world_seed.py`` so it's
the first collected file. If it passes in CI we know the seed is
fine and the bug lives downstream; if it fails the same way the
bug is in the auth resolution path.
* Bump ``max-failures`` to 200 for this workflow so we see the full
failure surface instead of stopping at the first cascading setup
error. Will tighten back down once the suite is green.
Adds one new test ``test_proxy_admin_actor_can_create_keys_for_others``
that explicitly exercises the PROXY_ADMIN bypass via /key/generate with
an explicit user_id — the same shape the matrix setup helper uses but
without the matrix machinery muddying the diagnostic.
* ci(proxy-mgmt-behavior): await LiteLLM_VerificationTokenView creation in fixture
Fourth CI run still failed because the proxy's lifespan kicks off
``prisma_client.check_view_exists()`` as a fire-and-forget background
task — that task is what creates ``LiteLLM_VerificationTokenView``, the
SQL view ``user_api_key_auth`` queries to resolve a token to its
user_id / user_role / team.
On a fresh Postgres (CI), the first test races the background task. The
view doesn't exist when the first auth call runs, the resolver falls
through to a degraded path that returns ``user_id=None``, and every
matrix test that depends on the seeded actor's identity then fails
confusingly with "Got user_id=X, Your ID=None" 403s. Locally the view
persists across pytest runs so the race is invisible.
Fix: await ``prisma_client.check_view_exists()`` explicitly inside the
session ``proxy_app`` fixture, after the lifespan enters but before the
fixture yields. Deterministic regardless of whether the underlying DB is
fresh (CI) or warm (local).
* ci(proxy-mgmt-behavior): widen diagnostic to dump token / user / view shape
The fifth CI run isolated the failure to ``/key/generate`` with explicit
user_id while ``/key/info`` works for the same seeded PROXY_ADMIN actor.
The auth context's user_id is None even though the DB row has it set.
This commit widens the diagnostic test: on failure, dump the raw token
row's user_id, the user row's user_role, and what
``LiteLLM_VerificationTokenView`` actually returns for the seeded token.
If the view returns user_id=None we know the view shape is the problem;
if the view returns the right user_id we know it's a downstream code
path stripping it.
* ci(proxy-mgmt-behavior): unambiguous diagnostic view query
Previous diagnostic's raw SQL had an ambiguous user_id column from
joining the view with the user table, so the diagnostic itself crashed
before printing useful state. Simplified to query just the view's columns.
* ci(proxy-mgmt-behavior): add auth-resolver chain diagnostic
Six runs and the underlying data (token row, user row, view row) all
verified correct in CI, but auth still returns user_id=None. This
diagnostic calls the resolver primitives directly:
1. ``prisma.get_data(table_name="combined_view")`` → raw view object
2. ``get_key_object(...)`` → cached/DB UserAPIKeyAuth
3. ``get_user_object(...)`` → LiteLLM_UserTable row
4. ``_is_user_proxy_admin`` / ``_get_user_role``
and prints each intermediate via captured stdout (-s). Whichever step
returns None/False in CI is where the chain breaks. Imports come from
``litellm.proxy.auth`` (not management_endpoints), so G3 still passes.
* ci(proxy-mgmt-behavior): set LITELLM_MASTER_KEY env so lifespan doesn't wipe it
Real root cause of every CI run that returned ``Your ID=None`` for the
seeded actors:
* In ``initialize()``, ``master_key`` is set from the config YAML's
``general_settings.master_key`` (load_config code path at
proxy_server.py:4174).
* Then the FastAPI lifespan (``proxy_startup_event``) runs and at line
776 does ``master_key = get_secret_str("LITELLM_MASTER_KEY")``,
which UNCONDITIONALLY overwrites the global.
* In CI the env var is unset, so the post-lifespan ``master_key`` is
None.
Downstream every auth path degrades: master-key requests don't bypass
because ``secrets.compare_digest(api_key, None)`` raises and is caught
to ``is_master_key_valid=False``; seeded-actor requests cache a
``UserAPIKeyAuth`` whose ``user_role`` never resolves through the
PROXY_ADMIN bypass; ``_is_allowed_to_make_key_request`` then hits the
``user_id`` mismatch path with ``Your ID=None``.
Locally my shell happened to have ``LITELLM_MASTER_KEY`` set from a prior
session, which is why every local run was green and CI red — exactly the
"don't generalize from your environment to CI" memory.
Fix: ``os.environ.setdefault("LITELLM_MASTER_KEY", MASTER_KEY)`` and
``os.environ.setdefault("CONFIG_FILE_PATH", config_path)`` before
entering the lifespan, so its re-read produces the same value as
``initialize()``.
Whole-suite still green locally (130 tests, ~6.4s).
* ci(proxy-mgmt-behavior): force premium_user=True so /key/regenerate isn't gated
Ninth CI run cleared every ``Your ID=None`` failure (the master_key env
fix worked end-to-end) and exposed the next thin layer of failures:
``/key/regenerate`` returns 500 "Regenerating Virtual Keys is an
Enterprise feature" in CI because the proxy can't see a
``LITELLM_LICENSE``. Locally my license is set, so the matrix passes.
The behavior matrix is supposed to pin authz, not licensing — so flip
``proxy_server.premium_user = True`` directly, both before and after the
lifespan (the lifespan re-runs ``_license_check.is_premium()`` and would
otherwise reset it). With premium gating disabled, the regenerate matrix
exercises the same authz path /key/update does.
Whole-suite still green locally (130 tests, ~6.3s).
* test(proxy_behavior): trim debug diagnostics, restore default max-failures
Followup to the CI-bring-up sequence: now that the suite is green in CI
(130 → 129 tests after this trim; 156s wall-time on ubuntu-latest), drop
the diagnostic noise left over from debugging the master_key wipe:
* Rename ``test_aaa_world_seed.py`` back to ``test_world_seed.py`` —
no longer needs to run first.
* Remove ``test_auth_resolver_returns_correct_user_id_and_role`` —
that test reached into private auth helpers to localize the bug
between the DB and ``UserAPIKeyAuth``; it has served its purpose
and isn't HTTP-boundary.
* Keep ``test_proxy_admin_actor_can_create_keys_for_others`` (without
the failure-time dump) — it's a real authz contract that pins the
PROXY_ADMIN bypass on /key/generate, and would catch a regression
of the same conftest interaction this sequence revealed.
* Drop the workflow's ``max-failures: 200`` override — that was a
debug aid for seeing the full failure surface in CI. Default of 10
is right for a stable suite.
* chore(proxy_behavior): drop empty mutmut triage stub, fold protocol into README
The mutmut_triage/pr1.md file was a placeholder for numbers and
classifications that don't exist yet — the first mutmut run is a manual
follow-up. Empty stubs aren't evidence; deleting it.
The G5 protocol (run the workflow, triage survivors in the six Tier-1
handler functions, kill-or-accept-with-reason, zero unreviewed) moves
into the suite README's "Gate evidence" block. The real triage file
will land alongside the first mutmut follow-up.
pyproject.toml's [tool.mutmut].tests_dir entry stays — that's the
one-line wiring that makes the existing (manual-trigger) mutation-test
workflow include our suite next time someone runs it. Comment updated
to drop the dead file reference.
* chore(proxy_behavior): drop README + trim comments
Removes the suite README — its contents (local repro, layout, conventions)
were either restated by the file structure or already covered by the
workflow YAML and pyproject.toml. Trims docstrings and inline comments
across every test file to keep only non-obvious WHY (the masking
``_get_user_in_team`` reads, the LiteLLM_VerificationTokenView models-can't-
be-NULL gotcha, the org_admin/peer-visibility surprise, the rotation
contract).
Suite still 129 green locally.
* test(proxy_behavior): address Greptile review — env force, pagination, dedup
- conftest: force LITELLM_MASTER_KEY / CONFIG_FILE_PATH unconditionally
instead of setdefault. An ambient LITELLM_MASTER_KEY with a different
value would make the proxy authenticate on that key while the tests
still send MASTER_KEY → silent 401s.
- test_key_list: paginate /key/list instead of a single size=100 request.
size is capped at 100 by the endpoint, so on a non-fresh DB a single
page could truncate PROXY_ADMIN's view and a seeded key could fall off
the page. Walk total_pages.
- conftest: hoist the duplicated _create_scratch_key helper (copy-pasted
and already diverged across test_key_{update,regenerate,delete}.py)
into a single shared create_scratch_key.
- Delete regression_replay/README.md — G4 regression-replay evidence
belongs in the PR description, not a committed doc file (repo docs
policy + the effort's own plan both say so). Content moved to the PR.
xAI's Grok Voice Agent API now sends session.created as its first
realtime event (matching OpenAI), followed by conversation.created.
The E2E canary pinned the old conversation.created value and failed.
LiteLLM's xAI realtime path is a verbatim passthrough (provider_config
is None, raw forwarding), so the event ordering is xAI's own — no
transformation on our side. Update the pinned expected value and the
now-stale comments to match the current API behavior.
* chore(proxy): strict media-type match for form bodies (#27939)
* chore(proxy): strict media-type match for form bodies
``_read_request_body`` and ``get_request_body`` routed on
``"form" in content_type`` / ``"multipart/form-data" in content_type``,
which match any header containing the literal — ``application/form-json``,
``multiform/anything``, ``application/json; xform=1``. Starlette's
``request.form()`` returns an empty ``FormData`` for any non-canonical
type without consuming the body, so the auth-time pre-read saw ``{}``
and skipped the banned-param check while the handler's later
``request.body()`` saw the original JSON payload.
Parse the media type per RFC 7231 (substring before ``;``, trimmed,
lowercased) and accept only ``application/x-www-form-urlencoded`` and
``multipart/form-data``. Replace both substring sites with the shared
``_is_form_content_type`` helper.
Tests pin: case/whitespace/charset variants of the two real types
match; ``application/form-json`` and similar substring-match traps
fall through to the JSON parse path; real form POSTs continue to
route through ``request.form()``.
* chore(proxy): extract _is_json_content_type symmetric helper
Mirror ``_is_form_content_type`` for the JSON branch of
``get_request_body`` so both classifications share the same media-type
normalisation (strip params, trim, lowercase) and any future change
to the parsing rules has one place to update.
Adds tests for ``_is_json_content_type`` and for ``get_request_body``
covering the canonical JSON / form / unsupported / non-POST paths.
* chore(proxy): surface form-parse failures instead of caching empty body
Starlette's ``request.form()`` raises ``MultiPartException`` /
``ValueError`` / ``AssertionError`` on malformed multipart input
(missing boundary, malformed chunk encoding, etc.). The outer
``except Exception: return {}`` swallowed every form-parse failure
and cached an empty parsed body — auth-time pre-reads saw ``{}`` and
skipped every banned-param check while a later raw-body re-read in
the handler still saw the original payload. Same TOCTOU shape as the
substring-match bypass: the auth gate and the handler don't agree on
what the body is.
Wrap ``request.form()`` in a narrow ``try`` that converts any parse
failure to a 400 ``ProxyException``. The outer broad ``except`` is
retained for unrelated unexpected errors but no longer covers
form-parse-side bypass shapes.
Adds a regression test parametrised over the exception classes
Starlette can raise from ``request.form()``.
* chore(proxy): drop redundant _is_json_content_type test class
``_is_json_content_type`` is a 3-line wrapper around the shared
``_normalize_media_type`` helper. Positive coverage lives in
``TestGetRequestBody.test_json_with_charset_param_parses_as_json``;
negative coverage is covered transitively by
``TestIsFormContentType``'s non-form parametrize matrix (anything that
isn't a form type falls through to the JSON branch).
* chore(proxy): carry ASGI path into WebSocket auth synthetic Request (#27940)
``user_api_key_auth_websocket`` built a synthetic ``Request`` with a
two-key scope (``type`` + ``headers``) and set ``request._url =
websocket.url``. ``get_request_route`` reads ``scope.get("path", ...)``
and falls back to ``request.url.path`` only when ``path`` is absent.
For the WebSocket flow that fallback fires and resolves to the
Host-header-derived value (Starlette reconstructs ``websocket.url``
from the Host header), so a malformed Host collapses the resolved
route and lets the auth gate compare against the wrong value.
Carry the ASGI scope's ``path``, ``root_path``, and ``app_root_path``
into the synthetic scope so the lookup never reaches the fallback on
the legitimate path.
Regression test pins that the request handed to ``user_api_key_auth``
has ``scope["path"]`` equal to the ASGI scope's path.
---------
Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com>
* fix: serialize guardrail_response to JSON in OTEL traces
Guardrail spans previously set the `guardrail_response` attribute via
`safe_set_attribute`, which let dict payloads reach the OTEL exporter as
Python repr strings. Downstream log pipelines could not parse those as
JSON, breaking metric creation from guardrail traces.
Serialize `guardrail_response` with `safe_dumps` before setting the
attribute, matching how `masked_entity_count` is already handled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: cover dict-serialization and None-skip for guardrail_response
Address Greptile feedback on #28362 — add explicit coverage for the
two behavioral guarantees of this fix:
- Dict payloads (the OpenAI moderation case in the report) reach the
span as a JSON string, not a Python repr.
- ``None`` guardrail_response skips the attribute entirely, so no
``"null"`` leaks into traces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: add Xiaomi MiMo-V2.5-Pro and MiMo-V2.5 OpenRouter model entries (#27700)
Squash-merged by litellm-agent from TorvaldUtne's PR.
* fix(ui): trim whitespace from MCP inspector tool call inputs (#28203)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* gemini-3.1-flash-lite pricing (#27933)
* feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers
* fix pricing
* add service tier
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
* fix: incorrect /v1/agents request example (#28131)
* fix(anthropic): accept dict-shape reasoning_effort from Responses bridge (#28201)
* fix(anthropic): accept dict-shape reasoning_effort from Responses bridge
Issue #28196 — the Responses->Chat parser (transformation.py:184-200) keeps the full dict as reasoning_effort when summary is set; that branch was added in #25359. But the Anthropic transformation here still guarded on isinstance(value, str), silently dropping the param. Result: callers using the standard Reasoning(effort, summary) OpenAI-shaped object on Anthropic lose thinking entirely (0 reasoning_tokens, no thinking_blocks).
Coerce dict -> string before mapping. Same shape tolerance that gpt_5_transformation._normalize_reasoning_effort_for_chat_completion already implements. summary is irrelevant for Anthropic's thinking_blocks.
Adds two regression tests: one parametrized over string + dict shapes (with and without summary), one covering unparseable dict inputs (drops silently, no crash).
* test(anthropic): add non-adaptive model coverage for dict-shape reasoning_effort
Per Greptile feedback on PR #28198: the original regression test only exercised the adaptive (4.6+) path. Add a parametrized test for the non-adaptive branch (claude-sonnet-4-5) verifying that dict-shape reasoning_effort still maps to thinking.type='enabled' + budget_tokens, and that output_config is NOT set on pre-4.6 models.
* test(anthropic): convert unparseable-dict test to @pytest.mark.parametrize
Per @greptile-apps inline review on PR #28201 — matches the parametrize style of the two adjacent dict-shape tests and produces clearer failure messages (test ID per case instead of one collapsing for-loop).
* feat: add pricing entry for openrouter/google/gemini-3.1-flash-lite (#28280)
Squash-merged by litellm-agent from ro31337's PR.
* fix(router): wrap aresponses streaming iterator for mid-stream fallbacks (#28215)
Squash-merged by litellm-agent from cwang-otto's PR.
* fix(router): unblock staging — mypy + coverage for aresponses streaming fallback (#28318)
Squash-merged by litellm-agent from cwang-otto's PR.
* fix(responses): forward timeout on completion transformation path (Anthropic, Bedrock, Vertex) (#28133)
Squash-merged by litellm-agent from cwang-otto's PR.
* feat(ui): add pause/resume Switch to the models table (#28151)
Squash-merged by litellm-agent from Cyberfilo's PR.
* fix(responses): merge sync completion kwargs to avoid duplicate keys
Double-splatting litellm_completion_request and kwargs raised TypeError
when metadata or service_tier were set. Match the async merge pattern.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Use proxy base URL for CLI SSO form action (#28271)
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which was missing from the
cost map. This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
litellm.completion_cost lookup.
- Add mistral/ministral-8b-2512 entry to both the in-tree
model_prices_and_context_window.json and the bundled
litellm/model_prices_and_context_window_backup.json (mirrors the
existing openrouter/mistralai/ministral-8b-2512 pricing).
- litellm.model_cost is loaded at import time from the URL pinned to
main, so the new backup entry isn't visible at test runtime until
it also lands on main. Backfill any entries missing from the
remote-fetched map into litellm.model_cost in the local_testing
conftest so cost-calculator lookups succeed on this branch.
* fix(tests): drop unnecessary del of conftest backfill loop vars
* fix(router): harden streaming fallback wrapper for bridge iterators
- FallbackResponsesStreamWrapper now uses getattr fallbacks when copying
attributes from the source iterator. The bridge path
(LiteLLMCompletionStreamingIterator used by Anthropic/Bedrock/Vertex)
does not call super().__init__ and is missing response, logging_obj
(it uses litellm_logging_obj), responses_api_provider_config,
start_time, request_data, call_type, and _hidden_params. Previously,
wrapper construction raised AttributeError for any streaming fallback
on the bridge path.
- _aresponses_with_streaming_fallbacks now deep-copies the
litellm_metadata (and metadata) dicts into fallback_kwargs. The
primary attempt mutates this dict in place via
_update_kwargs_with_deployment, so a shallow copy of kwargs was
leaking primary-deployment fields (deployment, model_info, api_base)
into the mid-stream fallback request.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(router): use safe_deep_copy for fallback metadata snapshot
The ban_copy_deepcopy_kwargs CI check rejects copy.deepcopy() on any
variable whose name contains 'kwargs' (incl. fallback_kwargs). Swap
the two copy.deepcopy(fallback_kwargs[...]) calls for safe_deep_copy,
which handles non-picklable values (OTEL spans, etc.) by per-key
deepcopy with fallback to the original reference.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(ci): skip chronically flaky build_and_test integration tests
Both tests have been failing on every recent run of build_and_test
against this PR's HEAD (1686967, 1688402, 1689993, 1690877), and the
same two tests also fail intermittently on unrelated commits and other
branches, independent of any code change in this PR (which only touches
router fallback wrappers, the Anthropic Responses bridge, and unrelated
UI/cost-map files).
- tests.test_spend_logs.test_spend_logs: /spend/logs?request_id=...
returns 500 even after a 20s wait for the spend log to be written.
Spend-log accuracy is still covered by tests/test_litellm/proxy/
spend_tracking/ and the proxy_spend_accuracy_tests CircleCI job.
- tests.test_team_members.test_add_multiple_members: /team/info?team_id=
... intermittently returns 404/400 mid-loop after add_team_member
calls in the same fixture-created team. Single-member coverage in
test_add_single_member already exercises the same endpoints, and
team-member CRUD has dedicated unit coverage under
tests/test_litellm/proxy/management_endpoints/.
Skipping unblocks the build_and_test job until the underlying race in
the dockerized integration setup is root-caused.
* fix: preserve explicit timeout=0 in responses API handler
Use 'timeout if timeout is not None else request_timeout' instead of
'timeout or request_timeout' so an explicit timeout=0/0.0 isn't silently
replaced by the default request_timeout.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui): guard model_info access in pause Switch with optional chaining
* fix(ui): guard model_info access in pause Switch onChange handler
Mirror the optional-chaining guard already applied to the isPausing
check so a config-model row with a missing model_info cannot throw
when the toggle's onChange fires.
---------
Co-authored-by: TorvaldUtne <78661304+TorvaldUtne@users.noreply.github.com>
Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com>
Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com>
Co-authored-by: Roman Pushkin <roman.pushkin@gmail.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: boarder7395 <37314943+boarder7395@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(proxy): expose Prisma idle/connect timeout + extra DB URL params
Operators have reported large numbers of idle Prisma connections that
never get closed. The proxy already forwards `connection_limit` and
`pool_timeout` to the DATABASE_URL, but had no knob for capping idle
or slow connections. Add three new `general_settings` keys that thread
through to the DATABASE_URL / DIRECT_URL query string:
- `database_connect_timeout` -> Prisma `connect_timeout`
- `database_socket_timeout` -> Prisma `socket_timeout` (the main
knob for closing idle connections from the LiteLLM side)
- `database_extra_connection_params` -> untyped passthrough dict for
any other Prisma URL param (`pgbouncer`, `statement_cache_size`,
`sslmode`, ...); keys here override LiteLLM defaults.
Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a
single `_build_db_connection_url_params` helper.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update litellm/proxy/proxy_cli.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(interactions): never drop streamed text deltas; always emit terminal completion
The interactions streaming bridge had two bugs flagged by Greptile on PR #28153:
1. The first OutputTextDeltaEvent (and the second, when no ResponseCreatedEvent
precedes the deltas) was consumed to emit a synthetic interaction.created /
step.start event, but the chunk's text payload was never forwarded as a
step.delta. The text only reappeared in the terminal step.stop, which
defeats the purpose of incremental streaming.
2. When the upstream Responses API stream ended via StopIteration without a
ResponseCompletedEvent, the iterator emitted step.stop but never the
terminal interaction.completed event carrying the full collected text.
This refactors the iterator to translate each upstream chunk into a list of
events (instead of a single event) and buffers them in a deque. A text delta
now expands into [interaction.created, step.start, step.delta] on the first
chunk so no token is dropped, and the StopIteration / StopAsyncIteration
fallback always flushes a terminal interaction.completed event when one
hasn't already been sent.
Both behaviors are covered by new unit tests:
- test_no_text_token_is_dropped_during_streaming
- test_response_created_then_text_delta_emits_step_start_and_delta
- test_stop_iteration_fallback_emits_completion_event
- test_response_completed_emits_stop_then_completion (no double-emit)
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(interactions): correlate EOF terminal events with stream's interaction id
The StopIteration fallback path previously built the terminal step.stop /
interaction.completed events with id=None (legacy content.stop) and a
memory-address fallback string (interaction.completed), neither of which
matched the item_id used by the earlier interaction.created / step.start /
step.delta events in the same stream. Downstream consumers correlating
events by id would see a mismatch.
Persist the interaction id derived from the first upstream chunk (item_id
on an OutputTextDeltaEvent, or response.id on a ResponseCreatedEvent) and
reuse it when flushing the terminal events on EOF.
Author: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci(windows): raise UV_HTTP_TIMEOUT to 300s for uv sync
The using_litellm_on_windows job has been hitting flaky PyPI download
timeouts during 'uv sync --frozen --group dev' — different packages on
each rerun (six, pydantic-core), all surfacing the same uv error:
Failed to download distribution due to network timeout.
Try increasing UV_HTTP_TIMEOUT (current value: 30s).
uv's default 30s per-request timeout is too tight for the Windows runner
on this project (50+ deps, several multi-MB wheels), so bump it to 300s
to let slow individual downloads complete instead of failing the build.
* fix(interactions): correlate ResponseCompletedEvent terminal events with stream's interaction id
When a stream starts directly with OutputTextDeltaEvent (no preceding
ResponseCreatedEvent), interaction.created carries item_id while
interaction.completed previously carried response.id from
ResponseCompletedEvent. The two ids can differ, leaving consumers that
correlate events by id unable to match the start and completion events.
Fall back to self._interaction_id (set on the first chunk that derives
an id) before response.id, mirroring the EOF terminal path.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(mcp): allow native MCP OAuth redirect URIs (cursor://)
Discoverable OAuth /authorize rejected cursor:// callbacks because
validate_trusted_redirect_uri only accepted http/https. Add an
allowlisted native path with a built-in Cursor default and optional
MCP_TRUSTED_NATIVE_REDIRECT_URIS env for other clients.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): address Greptile native redirect URI review
Lowercase paths in normalizer so env allowlist entries match case-
insensitively. Tighten wildcard prefix matching to reject sibling
paths (e.g. callback-2) unless the prefix ends with /.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): reject query params on native OAuth redirect URIs
Greptile: normalization stripped query strings before allowlist compare,
so cursor://.../callback?injected=... could pass validation. Reject any
native redirect_uri with a query component (same as fragments).
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(model_cost_map): add mistral/ministral-8b-2512 entry
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which is not in the cost map.
This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
completion_cost lookup. Add the entry mirroring the existing
openrouter/mistralai/ministral-8b-2512 pricing.
* fix(mcp): lowercase default native redirect URIs
Make _parse_trusted_native_redirect_uris apply the same lowercasing
to built-in defaults as it does to env-var entries.
* fix(tests): backfill local model_cost into remote-fetched map
litellm.model_cost is loaded at import time from the URL pinned to main,
so pricing entries that exist only in this branch (e.g.
mistral/ministral-8b-2512, freshly added because Mistral now returns this
id from mistral-tiny) are absent at test time and completion_cost lookups
raise. Backfill the in-tree backup so cassette-driven cost calculations
resolve against the entries that ship with the branch under test.
Fixes the local_testing_part1 failures on test_completion_mistral_api and
test_completion_mistral_api_modified_input.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
* fix(vertex_ai): omit function_call id on Vertex Gemini 3.5+ tool turns
Vertex AI rejects `id` on function_call/function_response parts; only Google AI Studio accepts it for Gemini 3.5+ strict tool matching.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(vertex_ai): forward custom_llm_provider in context caching
Pass custom_llm_provider through to _gemini_convert_messages_with_history
in the context caching path so Gemini 3.5+ tool-call `id` forwarding
behaves consistently between cached and non-cached completions on Google
AI Studio.
Co-authored-by: Claude <claude@anthropic.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <claude@anthropic.com>
* test(ui-e2e): add admin key creation with a specific proxy model
Adds Playwright coverage for creating a key (no team) scoped to a single
proxy model, complementing the existing All-Proxy-Models test. Uses a
DOM-dispatched click on the antd dropdown option since the popup
animation can render the option outside the viewport.
* test(ui-e2e): verify scoped key works against mock /chat/completions
Extend the "Create a key with a specific proxy model" test to extract
the new key from the success modal and POST to /chat/completions for
the scoped model, asserting 200 and the mock response body. Without
this the test could pass even if the model selection failed to register.
* feat(interactions): migrate to Google Interactions API steps schema (May 2026)
Default to Api-Revision: 2026-05-20 (new `steps` schema). Add
`litellm.use_legacy_interactions_schema` global flag that sends
Api-Revision: 2026-05-07 for operators who need the legacy `outputs`
schema until June 8, 2026.
- Inject Api-Revision header in GoogleAIStudioInteractionsConfig.validate_environment()
- Auto-coalesce response_mime_type → response_format and image_config migration on new schema
- Add steps field to InteractionsAPIResponse and InteractionsAPIStreamingResponse
- Add StepStart/StepDelta/StepStop/InteractionCreated/etc. SSE event types
- Update streaming completion detection to handle interaction.completed event
- Bridge transformer populates both outputs and steps fields
- Bridge streaming iterator emits new-schema events by default
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(interactions): address greptile review feedback
- Avoid mutating caller's generation_config dict by shallow-copying
before popping image_config, preventing silent failures on retries
- Skip schema key in response_format when response_format is None to
avoid sending schema: null to the Google Interactions API
- Remove delta field from step.stop events (new schema only); the
StepStop model has no delta field and sending it duplicates already-
streamed text and breaks spec-conformant clients
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): parse use_legacy_interactions_schema string values safely
bool("false") returns True in Python, so quoted YAML values like
"false" or "False" silently activated the legacy Interactions API
schema. Match the env-var parsing pattern in litellm/__init__.py by
treating string inputs as true only when they equal "true" (case
insensitive).
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(interactions): only set object/id/delta on step.stop for legacy schema
StepStop (new schema) has no object, id, or delta fields. Setting them
unconditionally caused spec-breaking extra fields on new-schema step.stop
events in all four construction sites (sync/async × main-loop/StopIteration).
Legacy content.stop still receives id, object, and delta unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(interactions): stabilize streaming bridge schema, dict aliasing, and lost first delta
- Capture use_legacy_interactions_schema once at iterator construction so
all events emitted by a single stream use a consistent schema, even if
the global flag is mutated mid-stream.
- Check for the buffered interaction.complete/completed event before the
finished check in __next__/__anext__ so the final completion event
(which carries the full collected text in steps) is not dropped after
self.finished is set.
- Copy text content entries before appending to both outputs and the
steps content list to avoid shared mutable dict aliasing between the
two response fields.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix tests
* fix greptile review
* fix(interactions): address Greptile P1 review on schema coalescing and legacy deltas
Skip response_mime_type merge when response_format is already a list, avoid
in-place list mutation on image_config append, and restore delta.type on
legacy content.delta events.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(interactions): black-format gemini transformation.py
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
* fix(mcp): JWT on tools/list, REST server_id resolution, tool_server_mismatch
Sign outbound MCP JWTs for list_mcp_tools and inject headers on the tools/list
path. Resolve server_id on /mcp-rest/tools/call and return 403 tool_server_mismatch
when the tool does not belong to the requested server. Default missing arguments to {}.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): restrict list JWTs to mcp:tools/list and default REST arguments to {}
- List-only JWTs (call_type=list_mcp_tools) no longer carry the broad
mcp:tools/call scope. _build_scope() now emits only mcp:tools/list
when no tool name is provided, mirroring the existing least-privilege
rule that tool-call JWTs omit mcp:tools/list.
- REST /tools/call now defaults a missing 'arguments' field to {} so
execute_mcp_tool() and downstream **arguments / .keys() calls don't
receive None and crash with TypeError/AttributeError.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): validate tool/server in call_tool; skip JWT signer when not configured or static auth present
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): align tests and mypy with user_api_key_auth on tools/list
Update mocks for the new _get_tools_from_server parameter, mock server
registry in REST access-denied test, and narrow static_headers for mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): accept user_api_key_auth in get_tools_from_mcp_servers mock
The side_effect for the all-servers case did not accept the new kwarg,
so tools/list returned an empty list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): fail fast for unknown tools when server mapping exists
Server-name fallback in call_tool must not open an upstream session when
the tool is absent from a populated mapping. Update the HTTP transport test
to register a known tool before asserting not-found behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix mypy
* Fix mypy
* fix(mcp): preserve tools/call scope on missing tool name; pass user_api_key_auth in list_tools
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): match alias/server_name in _resolve_mcp_server_for_tool_call
The registry lookup in _resolve_mcp_server_for_tool_call previously only
compared candidate.name against the provided server_name, but tool name
prefixes can be derived from a server's alias or server_name (see
get_server_prefix). When the tool→server mapping is empty/stale (cold
start, dynamic tools), the lookup would fail for alias-configured
servers even though get_mcp_server_by_name (used by the REST path)
matches alias, server_name, and name.
Match the same priority of identifiers in both the registry pass and
the unprefixed fallback so the MCP protocol call_tool path is
consistent with the REST path.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): reuse proxy_logging DualCache in inject_mcp_jwt_headers_for_upstream
Instead of allocating a fresh DualCache() on every tools/list invocation,
prefer the shared proxy_logging_obj.internal_usage_cache.dual_cache when
available. The cache argument is currently unused by MCPJWTSigner, but
sharing the proxy's cache avoids per-call allocation overhead and matches
the cache identity used elsewhere in the proxy hook plumbing — so any
future per-request state stored in cache will survive across list calls.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(mcp): return 403 ip_filtering for IP-restricted servers in tools/call name lookup
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(test): accept user_api_key_auth kwarg in list_tools mocks
The proxy-infra job was failing on four TestMCPServerManager tests because
the mock_get_tools_from_server stubs did not accept the new
user_api_key_auth keyword argument that list_tools now forwards to
_get_tools_from_server. Add the kwarg to each stub so list_tools can call
through cleanly.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): skip JWT injection when per-user mcp_auth_header is set
MCPClient._get_auth_headers() applies extra_headers AFTER writing
Authorization from auth_value, so an injected JWT silently overwrites
the user's per-server OAuth token. Guard the JWT signer with
'not mcp_auth_header' so per-user OAuth (and any dict-form per-user
auth) takes precedence, mirroring the existing static_headers guard.
Adds a regression test that the signer's inject helper is not called
when mcp_auth_header is supplied.
* fix(mcp): skip JWT injection when extra_headers already has Authorization
When a server uses per-user OAuth tokens, the resolved token is passed
into _get_tools_from_server via extra_headers. The JWT injection guard
only checked mcp_auth_header and the server's static headers, so the
signer would silently overwrite the user's OAuth Authorization header.
Add a check for an existing Authorization entry in extra_headers so
caller-supplied per-user OAuth tokens take precedence over JWT signing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(mcp): cover JWT signer + tool-call resolution branches
Adds unit tests for the new MCPServerManager helpers (_resolve_mcp_server_for_tool_call,
_resolve_oauth2_headers_for_tool_call) and the new MCPJWTSigner paths
(_build_scope call_type branches and inject_mcp_jwt_headers_for_upstream).
Brings patch coverage above the auto target without changing behavior.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): retry tool-server lookup with prefixed name in REST mismatch check
When the REST /mcp-rest/tools/call path sends a raw tool name plus
requested_server_id, _get_mcp_server_from_tool_name(name) can return
None if the mapping only stores the prefixed form. That bypassed the
tool_server_mismatch 403 guard and let the call fall through to
trusting requested_server.
Retry the lookup with every known prefix of the requested server so
the mismatch check fires whenever the tool is actually registered.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): always reject unknown tools in server-name fallback
Defense-in-depth: _resolve_mcp_server_for_tool_call previously skipped
the unknown-tool check whenever the per-server mapping had no entries
yet (cold start, OAuth2 lazy listing, or upstream listing failure),
allowing arbitrary tool names to reach upstream servers.
Tighten the check so the server-name fallback always rejects tool
names not present in the mapping. Callers must call list_tools first
(standard MCP flow) before tools/call can resolve. Removes the
now-unused _mapping_has_tools_for_server helper and adds an
explicit empty-mapping rejection test alongside the existing
populated-mapping rejection test.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude (greptile subagent) <claude-greptile-bot@anthropic.com>
* feat(ui): team allowed_passthrough_routes create parity + edit load fix
Add the Allowed Pass Through Routes selector to the create-team modal
(previously only on the edit form), and fix the edit form silently
dropping the field: it lives under team metadata, so initialValues must
read info.metadata.allowed_passthrough_routes — otherwise the selector
renders empty and saving wipes admin-set routes. Both selectors are
gated to premium proxy admins, mirroring the server-side gate.
Resolves LIT-3019
* fix(ui): persist team allowed_passthrough_routes edits on save
The edit form loaded the selector but the save path never wrote it back:
allowed_passthrough_routes stayed in the raw metadata JSON textarea and
parsedMetadata (from that textarea) always won, so selector edits were
silently discarded. Strip it from the textarea initialValues and overlay
values.allowed_passthrough_routes into updateData.metadata, mirroring how
guardrails is handled.
Resolves LIT-3019
* fix(ui): preserve team passthrough routes for non-proxy-admins on save
Only proxy admins may set allowed_passthrough_routes (server-side gate).
For non-proxy-admins, write the team's stored value back into metadata
instead of the form value, so saving an unrelated setting can't silently
wipe routes; omit the key entirely when the team never had any.
Resolves LIT-3019
* chore(e2e): migrate runner to uv, add All Proxy Models key test
Switches the local e2e runner (run_e2e.sh) from poetry to uv to match
the rest of the repo and CI. Adds a Playwright test for creating an
admin key with no team selected (all-proxy-models flow), a SLOWMO env
hook for headed debugging, and a MIGRATION_TRACKING.md doc that maps
the manual UI QA checklist to e2e tests so future migration work has
a single source of truth.
* chore(e2e): address greptile feedback
- Remove MIGRATION_TRACKING.md (docs belong in litellm-docs repo)
- playwright.config.ts: fall back to 0 when SLOWMO is non-numeric
(parseInt returns NaN, which Playwright accepts silently)
- run_e2e.sh: add --frozen to uv sync for CI determinism
When a new filter is applied to spend logs, React Query's keepPreviousData
left stale rows on screen for 10–15s with no indication that a fetch was
in progress. The previous custom isFilteringResults flag was removed in
the #25847 toolbar refactor and only partially restored on the Fetch
button. Use React Query's isPlaceholderData to discriminate a real
filter change (queryKey changed, data not yet arrived) from a same-key
live-tail refetch, and feed it into the existing isLoading prop on the
toolbar pagination text and the table body. Live-tail polls still keep
previous rows without flicker.
Co-authored-by: Ryan <ryan@Ryans-MBP.localdomain>
* fix(router): use forwarded model_id for native Azure container IDs in _init_containers_api_endpoints
Azure code-interpreter containers return provider-native IDs (cntr_ + hex)
that carry no LiteLLM routing payload, so _decode_container_id returns
model_id=None. The router was falling through to call the handler directly,
bypassing _ageneric_api_call_with_fallbacks and leaving api_base=None for
Azure deployments. Fall back to the model_id forwarded from the proxy
ownership check so deployment credentials are always applied.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): strip /openai/responses path from api_base in AzureContainerConfig.get_complete_url
When a deployment's api_base is the responses endpoint URL
(e.g. .../openai/responses?api-version=...), AzureContainerConfig was
appending /openai/containers on top of it, producing the broken path
.../openai/responses/openai/containers. Azure returns 404 for that URL
while the correct path is .../openai/containers.
Strip any /openai/responses suffix from api_base before constructing
the containers URL so the resource root is always used as the starting point.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): prefer api-version from api_base URL over deployment's api_version
The deployment's api_version (e.g. 2024-08-01-preview) targets the chat/responses
API and is too old for the containers API, which requires 2025-04-01-preview.
The responses endpoint api_base already carries the correct api-version in its
query string. Extract it and use it for the containers URL, overriding the
stale deployment-level version.
Fixes DELETE and file-upload operations returning 404 due to wrong api-version.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(containers): pass params=None instead of params={} to httpx to preserve api-version
httpx erases a URL's query-string when params={} (empty dict) is passed,
silently stripping ?api-version=2025-04-01-preview from every container
POST/DELETE request. Azure's GET endpoints tolerate a missing api-version;
POST (upload) and DELETE are strict, so those returned 404.
Fix: use `params or None` in container_handler._async_handle and
llm_http_handler.async_container_delete_handler (and all sibling container
handlers) so that an empty params dict falls back to None, leaving httpx to
preserve the URL's existing query string intact.
Adds a regression test that directly documents the httpx behaviour.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): remove elif model_id branch from _init_containers_api_endpoints
Two reviewer findings addressed:
1. Truncated comment on the model_id fallback line — now complete.
2. Security: the elif branch that fired when container_id was absent allowed
any authenticated caller to supply model_id in a POST /v1/containers body
and route the request through an arbitrary deployment UUID, bypassing the
model-level access checks that only validate `model`. Removed the elif
branch; operations without container_id (create, list) route by the
caller-supplied `model` field as before. model_id forwarding is kept only
inside the container_id block, where the proxy ownership check has already
validated the container before forwarding the deployment ID.
Adds a regression test pinning the security boundary: no-container-id path
calls original_function directly even when model_id is in kwargs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(containers): validate proxy-to-router model_id forwarding for managed IDs
Add test_regression_get_container_forwarding_params_sets_model_id_for_managed_id
to verify that get_container_forwarding_params (the proxy-side half of the Azure
routing fix) correctly extracts and forwards model_id from a LiteLLM-managed
encoded container ID.
This closes the gap identified by Greptile P1: the previous regression test
only injected model_id as a direct kwarg, validating the router in isolation.
The new test exercises the actual proxy-to-router data flow through
ownership.get_container_forwarding_params, confirming that kwargs["model_id"]
is populated before _init_containers_api_endpoints is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): tighten endpoint-path strip to endswith match
Use path.endswith() instead of path.find() for _AZURE_ENDPOINT_PATHS so
the suffix strip only fires when api_base actually ends with one of the
endpoint-specific path suffixes. This is the more precise check greptile
flagged on the original find()-based implementation.
* Fix sync container handler to preserve URL query string
Mirror the async path fix: pass None instead of an empty params dict so
httpx does not strip the URL's existing query string (e.g.
?api-version=...), which is required for Azure container routing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(azure-containers): strip trailing slash before endpoint suffix match
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(containers): recover model_id from stored encoded id for native Azure container IDs
get_container_forwarding_params previously only set model_id when the
user-supplied container_id was a LiteLLM-managed encoded id. For native
upstream IDs (e.g. Azure 'cntr_<hex>') the decode fails and model_id was
never forwarded — making the router-side fallback in
_init_containers_api_endpoints unreachable in production.
Fall back to the stored 'unified_object_id' on the ownership row, which
is the encoded form captured at create time when the router selected a
specific deployment. Decoding that yields the deployment model_id and
restores router-based credential application (api_base, api_key) for
retrieve/delete and container-file operations on native IDs.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(proxy): normalize batch file IDs before ManagedObjectTable write
Run post_call_success_hook before update_batch_in_database on retrieve/cancel,
and ensure_batch_response_managed_file_ids so file_object never stores raw
provider output_file_id or error_file_id.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): address Greptile review on batch file ID normalization
Remove redundant resolve_* calls after update_batch_in_database and rename
loop variable to avoid shadowing hidden_params unified_file_id.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest
Mistral rotated the 'mistral/mistral-tiny' alias to return
'ministral-8b-2512' as the response model, which was missing from the
cost map. This caused test_completion_mistral_api and
test_completion_mistral_api_modified_input to fail in
litellm.completion_cost lookup.
- Add mistral/ministral-8b-2512 entry to both the in-tree
model_prices_and_context_window.json and the bundled
litellm/model_prices_and_context_window_backup.json (mirrors the
existing openrouter/mistralai/ministral-8b-2512 pricing).
- litellm.model_cost is loaded at import time from the URL pinned to
main, so the new backup entry isn't visible at test runtime until
it also lands on main. Backfill any entries missing from the
remote-fetched map into litellm.model_cost in the local_testing
conftest so cost-calculator lookups succeed on this branch.
* fix(tests): drop unnecessary del of conftest backfill loop vars
* fix: resolve batch response file IDs even when status unchanged
The status-unchanged early return in update_batch_in_database was
skipping ensure_batch_response_managed_file_ids, leaving raw provider
input_file_id (and other raw IDs) in the user-facing response when
polling an in-progress batch. Move the in-place file ID normalization
above the early return so the response always carries unified managed
IDs while still skipping the DB write when nothing changed.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(batches): cover ensure_batch_response_managed_file_ids branches
Add tests for the previously-uncovered paths in
ensure_batch_response_managed_file_ids: error_file_id normalization,
swallowed conversion errors, UserAPIKeyAuth fallback from
db_batch_object, model_name resolution from unified_file_id, and early
returns when managed_files_obj, model_id, or auth context are missing.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
The probe filter previously matched any server with Authorization in
extra_headers, including gateway-managed OAuth2 servers. Those would
then receive the resource_metadata= WWW-Authenticate challenge meant
for pass-through servers, instead of the authorization_uri= challenge
pointing at the gateway AS metadata. Use srv.is_oauth_passthrough so
only genuine pass-through servers get the resource-metadata challenge.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
- discoverable_endpoints: For pass-through MCP servers, when upstream
oauth-protected-resource returns a non-200/non-dict response, raise
HTTP 502 instead of falling through to default gateway metadata.
Falling through would direct MCP clients at the gateway, which is
not the authorization server for pass-through configs.
- mcp_server_manager: Wrap _get_tools_from_server in startup tool name
mapping with try/except. Since _get_tools_from_server now re-raises
MCPUpstreamAuthError, an upstream 401 from a pass-through server at
startup (when no user token is present) would otherwise abort the
loop and leave subsequent servers unmapped.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(spend_counter): seed Redis counter via SET NX to prevent cross-pod double-seed
Symptom
-------
Customers on multi-pod deployments see team `spend` jump to ~2x (or N x
the pod count) shortly after a Redis cache miss / TTL expiry, triggering
spurious "Budget Crossed" alerts and blocked requests until the value is
manually reset.
Root cause
----------
`SpendCounterReseed.coalesced` warmed the primary spend counter by
calling `redis.async_increment(key, value=db_spend, refresh_ttl=True)`,
which lowers to Redis `INCRBYFLOAT`. That is additive, not idempotent.
The per-counter `asyncio.Lock` only coalesces seeders inside one
process. With N pods sharing one Redis, on a cold key (cold start, TTL
expiry, manual delete) every pod independently passes its lock + Redis
re-check, reads the same `db_spend`, and issues `INCRBYFLOAT db_spend`.
Final value: N x db_spend.
Fix
---
Use `redis.async_set_cache(key, value=db_spend, nx=True)` for the seed.
SET NX is atomic across pods: exactly one writer initializes the key;
losers read the winner's value via `async_get_cache`. This is the same
idiom already used by `coalesced_window` in the same file, so the two
seed paths are now consistent.
Per-request deltas continue to use `INCRBYFLOAT` (correct - additive
behaviour is what we want for increments, not for initial seed).
Verification
------------
Live two-process repro against the same Postgres + Redis (DB
spend = 506):
Unpatched: 4/4 runs -> Redis counter = ~1012 (~2 x db_spend)
Patched: 12/12 runs -> Redis counter = ~506
Unit tests (`test_proxy_server.py`):
- New `test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed`
patches `_get_lock` to return a fresh lock per caller (otherwise the
per-process lock masks the race), races two `coalesced` calls, and
asserts final = 506 with exactly one of two SET NX attempts winning.
- 4 existing tests updated for the new seed contract (SET NX for the
seed, INCRBYFLOAT only for the per-request delta).
- Full `spend_counter or reseed or budget` slice: 22 passed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(spend_counter): make SET NX mock atomic so loser branch is exercised
Greptile flagged that `redis_set_cache` in
test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed
placed `await asyncio.sleep(0)` AFTER the NX membership check. Both
concurrent tasks observed an empty `redis_store`, passed the guard, and
both returned True - so the loser branch (else: read back winner's value)
was never exercised.
Fix the mock to model real atomic Redis SET NX:
- Yield BEFORE the membership check so two concurrent callers interleave
the way real SET NX does (first to resume runs check + write atomically
and wins; second resumes after the key exists and loses).
- Track set_cache return values; assert sorted([loser, winner]) so we
know exactly one task wins and one loses.
- Track async_get_cache calls that happen AFTER at least one SET NX has
completed; assert at least one such read - that is the loser-path
fallback (`current_value = float(cached)` when seeded is False).
Verified by temporarily reverting the mock to the old order: the test
now fails with `expected exactly one SET NX winner and one loser, got
[True, True]`, exactly the failure mode Greptile described.
No production code change.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(spend_counter): mock async_set_cache to populate redis_store in concurrent read+write test
`test_concurrent_read_and_write_paths_share_one_db_query` mocks
`async_increment` to populate the in-memory `redis_store`, but did not
mock `async_set_cache`. After the SET-NX seed change in `coalesced()`,
the seed step writes via `async_set_cache(nx=True)` (default AsyncMock,
no `redis_store` write), so the simulated Redis stays empty after the
first reseed. The second `get_current_spend` then sees a clean Redis
miss, re-enters the DB read path, and the test fails with
`expected 1 DB query, got 2`.
Fix: add a `redis_set_cache` side_effect that updates `redis_store` on
`nx=True` (and rejects when the key already exists), matching the
pattern used by the four sibling tests fixed in this branch's first
commit. Pre-existing assertions are unchanged.
Full `tests/test_litellm/proxy/test_proxy_server.py`: 158 passed.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
- _check_passthrough_upstream_auth now emits
'Bearer resource_metadata="..."' pointing at the gateway's
oauth-protected-resource well-known URL, mirroring the
pre-emptive 401 path. Pass-through servers don't use the gateway
as an authorization server, so the previous 'authorization_uri='
challenge sent clients to the wrong metadata endpoint.
- _parse_mcp_server_names_from_path now accepts server names that
contain a single slash (e.g. custom_solutions/user_123), mirroring
MCPRequestHandler._extract_target_server_names_from_path. Without
this, the cold-start bypass missed slashed-name servers and the
generic admission error propagated instead of the spec-compliant
401 challenge.
- _is_mcp_passthrough_cold_start drops the unused scope parameter
from its signature.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The merge of base-branch _target_servers_delegate_auth_to_upstream
into process_mcp_request inserts an additional
get_mcp_server_by_name(name) lookup ahead of the cold-start path,
which breaks two test patterns:
1. lookup_by_name(name) side-effect stubs in
TestMCPDelegateAuthToUpstream are called positionally by the
delegate check, then again by the cold-start path with
client_ip=... — raising TypeError: unexpected keyword argument
'client_ip'. Accept **_kwargs to match the real signature.
2. TestMCPPassthroughColdStartAdmission assertions count the lookup
exactly once with client_ip=..., but the delegate check now adds
a positional-only call ahead of it. Switch assert_called_once_with
to assert_any_call for the cold-start invocation, and assert
client_ip was *not* passed for the aggregate /mcp test where
cold-start must not fire.
Both updates align with CLAUDE.md guidance to keep monkeypatch stubs in
sync with the real signature when an optional parameter is added.
Co-authored-by: Claude <claude@anthropic.com>
- Return None from _parse_mcp_server_names_from_path for CSV multi-server
paths (/mcp/a,b). The regex previously truncated at the first comma and
silently passed a single server name to the cold-start gate.
- Switch _is_mcp_passthrough_cold_start to all-targets semantics, matching
_target_servers_use_oauth2: one non-passthrough target in a co-targeted
set must not flip the anonymous-admission bypass open for the others.
- Drop the redundant HTTPStatusError block in _extract_upstream_auth_failure
- any HTTPStatusError carries a .response, so the preceding generic block
already handles 401/403 detection.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
