The writer health probe only ran SELECT 1, which a read-only Postgres
session answers fine, so a pooled connection left pointing at a demoted
primary kept failing every write with SQLSTATE 25006 until the pod was
restarted. Probe transaction_read_only instead, treat a 25006 on the
request path as a signal to recreate the client, and back off
exponentially while the database as a whole stays read-only so a replica
or an in-progress failover does not get its engine killed every cycle.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): resolve root_path per request from SERVER_ROOT_PATHS
One deployment can encode exactly one client-visible URL path prefix
today: SERVER_ROOT_PATH is a scalar stamped onto the app at startup, so
a pod fronting several ingress prefixes 404s every prefix but one before
any handler runs, and MCP OAuth discovery can emit only one prefix's
URLs (RFC 9728 section 3 exact-match fails for the rest).
Add an opt-in outermost ASGI middleware that matches the request path
against a configured prefix list (SERVER_ROOT_PATHS, comma-separated) on
a segment boundary and sets scope["root_path"] for that request only.
Everything downstream is stock Starlette: route matching strips
root_path so routes stay registered root-relative, and request.base_url
re-includes it, so the discovery documents' resource and the 401
challenges' resource_metadata land under the prefix the client actually
called — with no discovery-builder changes.
LazyFeatureMiddleware now strips the scope root_path (falling back to
the cached SERVER_ROOT_PATH scalar) before feature prefix matching, so
lazily-registered routers — the MCP OAuth discovery router among them —
load under per-request prefixes.
Follow-up to the routing discussion on #35226; composes with, but does
not depend on, #35576.
* fix(proxy): import Sequence from collections.abc (ruff UP035 strict-budget gate)
* review(greptile): trim implementation commentary; fixture-own MCP registry state in tests
Addresses both P2s from the first Greptile pass:
- per_request_root_path_middleware.py (and the related _lazy_features /
proxy_server comments) cut down to the constraints the code cannot
express, per repo comment guidance
- the new discovery tests no longer clear/repopulate the shared MCP
registry inline; a fixture snapshots it, hands the test an empty
registry, and restores it afterwards so no state leaks between cases
* fix(lint): mutable-ok marker on the prefix accumulator (LIT002 type-discipline gate)
* fix(proxy): tie 401 challenges and get_custom_url to the per-request root_path
The per-request root_path middleware sets scope["root_path"] to the
prefix the client actually called, but the OAuth 401 challenges
(raise_user_oauth_challenge / raise_token_exchange_challenge) still
built their resource_metadata from SERVER_ROOT_PATH. On a pod fronting
several prefixes, the challenge advertised a discovery URL under a
different prefix than the discovery document served — the two
disagreed on where the resource metadata lives, and a strict RFC 9728
client refused the challenge. Route the challenges through a small
ContextVar the middleware populates so they read the same effective
root_path Starlette resolves the request under.
The same accessor fixes get_custom_url: when a request lives under a
SERVER_ROOT_PATHS-matched prefix, request.base_url already carries it,
so appending the SERVER_ROOT_PATH scalar on top produced e.g.
/tenant-a/legacy/sso/callback — a path that does not exist. Reading
the per-request prefix instead (and relying on join_paths's tail-dedup)
keeps SSO login/callback URLs under one prefix — the one the request
actually arrived on.
Fallback: outside a request (module-load-time UI URL builders,
background tasks) the ContextVar is unset and the accessor reads
SERVER_ROOT_PATH, matching get_server_root_path() so scalar-only
deployments are byte-identical.
* fix(mcp): challenge URL under per-request prefix must route, and mock parity
Two follow-ups to the review fix that made the 401 challenge use the
per-request root_path:
1. oauth_protected_resource_path must pick the URL structure that
actually routes for the mechanism in use:
- The scalar SERVER_ROOT_PATH deployment registers the well-known
routes with the prefix INSERTED (via well_known_root_suffix at
import time), matching RFC 8414 §3. The challenge URL must use the
same insertion or a client fetching it 404s.
- The per-request SERVER_ROOT_PATHS deployment can't register routes
per prefix; PerRequestRootPathMiddleware strips the prefix from
scope["path"] and the router matches the un-inserted route. The
URL must place the prefix BEFORE .well-known so the strip leaves a
matching path.
The previous fix used the insertion form for both, which 404'd the
discovery fetch on the per-request path — the discovery doc and the
challenge would then disagree on where the resource metadata lives,
the very failure the review flagged. End-to-end verified: the URL
the challenge advertises routes and the doc's `resource` field
equals the URL the client originally called (RFC 9728 §3).
2. get_request_root_path now delegates its fallback through
get_server_root_path() instead of reading the env directly, so every
existing `monkeypatch.setattr("litellm.proxy.utils.get_server_root_path"`
test override keeps working. This unstubbed the mock on the /v2/login
test that failed on the last CI run.
Plus the lint budget: annotate the local accumulator Final, tag the
scope["root_path"] rewrite as an intentional ASGI-contract mutation,
tag the reused `path`/`root_path` rebinds in LazyFeatureMiddleware, and
add reason strings to the two new PLC0415 lazy-import noqas.
* test(mcp): pin the reviewer's expected end-state — challenge URL routes, resource matches called URL
End-to-end regression test that mounts the discoverable router + the
per-request root_path middleware, hits an MCP endpoint that raises
raise_user_oauth_challenge, fetches the resource_metadata URL the
challenge advertises, and checks the returned document's `resource`
equals the URL the client originally called (RFC 9728 §3 exact match).
Covers /tenant-a, /tenant-b, and the unprefixed path on the same app so
a regression on any prefix — challenge URL 404s, or doc emits a
different prefix than the client called — fails at this test rather
than in a strict MCP client's discovery.
---------
Co-authored-by: gym-cmd <186399764+gym-cmd@users.noreply.github.com>
The proxy's exception tails defaulted `type` and `param` to the four-character
string "None", which is neither a known OpenAI error type nor the JSON null the
nullable `param` field is typed as, so a client's error handler matched nothing
and fell into its generic branch.
Lifts the helpers PR #39521 added for the unified LLM endpoints into
litellm/proxy/common_utils/openai_error_payload.py and calls them from the file,
rerank, image, realtime, anthropic, and pass-through route families, plus the
shared handle_exception_on_proxy handler that the management, batches,
fine-tuning, credential, SCIM, guardrail, and customer routes funnel through.
The remaining families (proxy_server, auth, health, spend tracking, and
management endpoints) follow in separate PRs so each slice stays QA'able on a
live proxy.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* fix(proxy): resolve entity access groups in the model listing endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse the fetched team object when listing models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover key-level access group resolution in model listing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): merge model-level guardrails before pre_call_hook
DB/UI-assigned guardrails (litellm_params.guardrails) only fire on
post_call paths today: _check_and_merge_model_level_guardrails is called
in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but
never before pre_call_hook in common_request_processing.py:963. PR #23774
fixed the non-streaming post_call case; pre_call was left broken.
At the pre_call site, add_litellm_data_to_request strips client-supplied
metadata.model_info (pricing spoofing guard) and route_request hasn't run
yet, so model_info.id is unavailable. Extend the helper to fall back to
llm_router.get_deployment_by_model_group_name(model_alias) when model_id
is missing — that uses the O(1) model-name index already maintained by
the router.
Closes#29652
* fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload
Addresses veria-ai Medium finding + proxy-infra CI failure on this PR.
ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool
hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format
was omitting the field, so a key with mcp_rpm_limit could exceed it via the
MCP path.
Reads from kwargs.get("mcp_rate_limit_server_name") to match how
pre_call_tool_check resolves the alias-then-server-name fallback before
invoking hooks.
* fix(proxy): union guardrails across group deployments on alias fallback
Addresses second veria-ai Medium on #29654: the alias fallback called
get_deployment_by_model_group_name(), which returns ONE deployment.
A guardrail set on a non-first deployment would silently not run on
pre_call when the model_id is missing.
Switch to get_model_list(model_name=...) and take the UNION of
litellm_params.guardrails across all matching deployments (with dedup).
Trade-off documented in the comment: pre_call cannot know which
deployment route_request will select, so the conservative choice is to
apply any guardrail set on any eligible deployment.
Updated test stubs to use get_model_list. Added 3 new tests covering
union, dedup, and the all-empty case.
* test(model_level_guardrails): align integration test with get_model_list union API
* fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint
Addresses 3rd veria-ai Medium on #29654:
add_litellm_data_to_request preserves client-supplied metadata.model_info
when the caller's key/team has allow_client_pricing_override. The pre_call
merge previously trusted that id, so a caller could spoof an unguarded
model_info.id while requesting a guarded alias and bypass guardrails.
New `trust_client_model_info: bool` param on the helper. The pre_call
call site passes False; post_call paths (existing) keep True.
Also fixes the ruff failure on the union loop: pulled the .get() into a
local + isinstance(list) check before iterating, so mypy stops complaining
about `object` not being iterable.
2 new regression tests covering spoof-and-bypass + default-trust behavior.
* fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance
Addresses two more reviewer findings on #29654:
veria-ai Medium: route_request resolves team-scoped public model names
with metadata.user_api_key_team_id. The pre_call alias fallback called
get_model_list(model_name=...) without the team_id, so team-scoped
deployments were invisible and their pre_call guardrails silently
skipped. Now reads team_id from metadata or litellm_metadata and passes
it to get_model_list.
greptile P1: the isinstance(deployment_guardrails, list) guard added for
mypy narrowing silently dropped bare-string guardrail values that the
existing post_call path used to truthy-accept. Restored by wrapping a
scalar string into a one-element list on both paths.
4 new tests: team_id passthrough (metadata + litellm_metadata), scalar
on post_call, scalar on alias-union. 36/36 tests pass.
* style: black formatting on _check_and_merge_model_level_guardrails team_id assignment
* chore: ruff format
* fix(lint): remove unused noqa PLR0915 directive
RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic
because PLR0915 is not in this repo's enabled ruff rule set
(lint.extend-select in ruff.toml), so the directive suppresses nothing
and fails the lint job.
* refactor(proxy): hoist guardrail-merge import to module top
The pre_call guardrail-merge helper was imported inside
common_processing_pre_call_logic with a # noqa: PLC0415, which the
type-discipline gate counts as an unexplained suppression (LIT003). The
inline import's cyclic-import justification does not hold: this module
already imports from litellm.proxy.utils at top level, and utils.py does
not import common_request_processing at module load. Fold the helper
into the existing top-level import and drop the inline import, clearing
the suppression instead of budgeting for it.
---------
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
* fix(proxy): source /v1/models token limits from cost map instead of Router.get_model_group_info
Resolves the per-model get_model_group_info fan-out on GET /v1/models
(and /models) that pegged the event loop on wildcard listings (#33636).
create_model_info_response now reads max_input_tokens/max_output_tokens
from litellm.get_model_info (the static cost map) rather than the router,
which aggregated and deepcopied every deployment in a group per listed
model.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject model-info lookup into create_model_info_response for deterministic coverage
Inject the cost-map lookup (defaulting to litellm.get_model_info) so the
except and max_output_tokens branches are exercised deterministically and
the token-limit tests no longer hardcode mutable cost-map values.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): surface custom deployment token limits on /v1/models via cheap index lookup
Add Router.get_configured_token_limits, an O(1) model-name index lookup that
reads a concrete deployment's configured max_input_tokens/max_output_tokens
without triggering pattern matching or deep copies. create_model_info_response
layers this over the cost map so custom deployments absent from the cost map
still surface their limits, and admin-configured limits override cost-map
defaults, while wildcard-expanded names stay on the fast path.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy/utils): pin bottom-of-file helper behavior
Pin current behavior of the bottom-of-file pure-function helpers in
litellm/proxy/utils.py (projection, team config, time helpers,
guardrail merge, error helpers, URL/path helpers, premium gate, model
access, and misc DB/API-key helpers).
Adds tests/test_litellm/proxy/utils/helpers/ with one happy + one error
test per pinned symbol; folds the prior single-test
tests/test_litellm/proxy/test_utils.py into test_url_helpers.py and
deletes the old file. _pin_check.py and _coverage_check.py serve as
local stopping gates.
Adds tests/test_litellm/proxy/utils to the existing test-path block in
.github/workflows/test-unit-proxy-endpoints.yml.
Plan: https://www.notion.so/37343b8acdab81f68f39f66915f62bcf
Pin list: https://www.notion.so/37343b8acdab8150acdbf40e5756869f
* test(proxy/utils): apply greptile fixes to behavior-pinning gates
Address findings from the sibling PR1/PR2 greptile reviews that also
apply to this PR:
- Commit pin_list.txt alongside the gate script (was previously a
gitignored .pin_list.txt fetched from Notion). The gate is now
reproducible without out-of-band setup.
- Resolve the coverage region by locating the first pinned symbol's
def line in litellm/proxy/utils.py at runtime, instead of hardcoded
line numbers that drift when lines above shift.
- Word-boundary the pin reference check so pins like update_spend do
not falsely match update_spend_logs_job.
- Drop the dead _harness_smoke_test.py exclusion; the test_*.py glob
already filters underscore-prefixed files.
* test(proxy/utils): drop local-only stopping-signal scripts
Remove _pin_check.py, _coverage_check.py, and pin_list.txt. These were
dev-time tooling for knowing when test authoring was done; they are
not wired into CI and the test files themselves are the merge artifact.
---------
Co-authored-by: Claude <noreply@anthropic.com>
