mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
6 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Roman D
|
55fe4a7894
|
feat(proxy): resolve root_path per request from a configured prefix list (SERVER_ROOT_PATHS) (#35935)
* feat(proxy): resolve root_path per request from SERVER_ROOT_PATHS One deployment can encode exactly one client-visible URL path prefix today: SERVER_ROOT_PATH is a scalar stamped onto the app at startup, so a pod fronting several ingress prefixes 404s every prefix but one before any handler runs, and MCP OAuth discovery can emit only one prefix's URLs (RFC 9728 section 3 exact-match fails for the rest). Add an opt-in outermost ASGI middleware that matches the request path against a configured prefix list (SERVER_ROOT_PATHS, comma-separated) on a segment boundary and sets scope["root_path"] for that request only. Everything downstream is stock Starlette: route matching strips root_path so routes stay registered root-relative, and request.base_url re-includes it, so the discovery documents' resource and the 401 challenges' resource_metadata land under the prefix the client actually called — with no discovery-builder changes. LazyFeatureMiddleware now strips the scope root_path (falling back to the cached SERVER_ROOT_PATH scalar) before feature prefix matching, so lazily-registered routers — the MCP OAuth discovery router among them — load under per-request prefixes. Follow-up to the routing discussion on #35226; composes with, but does not depend on, #35576. * fix(proxy): import Sequence from collections.abc (ruff UP035 strict-budget gate) * review(greptile): trim implementation commentary; fixture-own MCP registry state in tests Addresses both P2s from the first Greptile pass: - per_request_root_path_middleware.py (and the related _lazy_features / proxy_server comments) cut down to the constraints the code cannot express, per repo comment guidance - the new discovery tests no longer clear/repopulate the shared MCP registry inline; a fixture snapshots it, hands the test an empty registry, and restores it afterwards so no state leaks between cases * fix(lint): mutable-ok marker on the prefix accumulator (LIT002 type-discipline gate) * fix(proxy): tie 401 challenges and get_custom_url to the per-request root_path The per-request root_path middleware sets scope["root_path"] to the prefix the client actually called, but the OAuth 401 challenges (raise_user_oauth_challenge / raise_token_exchange_challenge) still built their resource_metadata from SERVER_ROOT_PATH. On a pod fronting several prefixes, the challenge advertised a discovery URL under a different prefix than the discovery document served — the two disagreed on where the resource metadata lives, and a strict RFC 9728 client refused the challenge. Route the challenges through a small ContextVar the middleware populates so they read the same effective root_path Starlette resolves the request under. The same accessor fixes get_custom_url: when a request lives under a SERVER_ROOT_PATHS-matched prefix, request.base_url already carries it, so appending the SERVER_ROOT_PATH scalar on top produced e.g. /tenant-a/legacy/sso/callback — a path that does not exist. Reading the per-request prefix instead (and relying on join_paths's tail-dedup) keeps SSO login/callback URLs under one prefix — the one the request actually arrived on. Fallback: outside a request (module-load-time UI URL builders, background tasks) the ContextVar is unset and the accessor reads SERVER_ROOT_PATH, matching get_server_root_path() so scalar-only deployments are byte-identical. * fix(mcp): challenge URL under per-request prefix must route, and mock parity Two follow-ups to the review fix that made the 401 challenge use the per-request root_path: 1. oauth_protected_resource_path must pick the URL structure that actually routes for the mechanism in use: - The scalar SERVER_ROOT_PATH deployment registers the well-known routes with the prefix INSERTED (via well_known_root_suffix at import time), matching RFC 8414 §3. The challenge URL must use the same insertion or a client fetching it 404s. - The per-request SERVER_ROOT_PATHS deployment can't register routes per prefix; PerRequestRootPathMiddleware strips the prefix from scope["path"] and the router matches the un-inserted route. The URL must place the prefix BEFORE .well-known so the strip leaves a matching path. The previous fix used the insertion form for both, which 404'd the discovery fetch on the per-request path — the discovery doc and the challenge would then disagree on where the resource metadata lives, the very failure the review flagged. End-to-end verified: the URL the challenge advertises routes and the doc's `resource` field equals the URL the client originally called (RFC 9728 §3). 2. get_request_root_path now delegates its fallback through get_server_root_path() instead of reading the env directly, so every existing `monkeypatch.setattr("litellm.proxy.utils.get_server_root_path"` test override keeps working. This unstubbed the mock on the /v2/login test that failed on the last CI run. Plus the lint budget: annotate the local accumulator Final, tag the scope["root_path"] rewrite as an intentional ASGI-contract mutation, tag the reused `path`/`root_path` rebinds in LazyFeatureMiddleware, and add reason strings to the two new PLC0415 lazy-import noqas. * test(mcp): pin the reviewer's expected end-state — challenge URL routes, resource matches called URL End-to-end regression test that mounts the discoverable router + the per-request root_path middleware, hits an MCP endpoint that raises raise_user_oauth_challenge, fetches the resource_metadata URL the challenge advertises, and checks the returned document's `resource` equals the URL the client originally called (RFC 9728 §3 exact match). Covers /tenant-a, /tenant-b, and the unprefixed path on the same app so a regression on any prefix — challenge URL 404s, or doc emits a different prefix than the client called — fails at this test rather than in a strict MCP client's discovery. --------- Co-authored-by: gym-cmd <186399764+gym-cmd@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
b76def0e5d
|
test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A TypeError from a refactor, a botched fixture, an import that moved: all of them read as the rejection the test claims to police, so the test goes green for the wrong reason and stays green after the behaviour it guards is gone. PT011 closes that gap for the 317 sites B017 could not reach, because B017 only fires on a single-statement body with no `as e` binding. Each pattern here is the message the code actually raised, recorded by running the sites under a plugin that logged the concrete type and text per call site, so the assertions describe observed behaviour rather than a guess. Where a site raises more than one message across its parametrize cases, the pattern is an alternation of what was seen; where the exception carries an empty `str()` and puts the text on `.message`, the site keeps a narrow `noqa` with the reason. PT014 removes four parametrize cases that were listed twice. The duplicate re-runs an assertion that already passed, and it usually marks a case someone meant to vary and forgot to edit. |
||
|
devin-ai-integration[bot]
|
1a45bf9afe
|
fix(proxy): resolve entity access groups in the model listing endpoints (#36230)
* fix(proxy): resolve entity access groups in the model listing endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): reuse the fetched team object when listing models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover key-level access group resolution in model listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
hcl
|
2a55d23731
|
fix(proxy): merge model-level guardrails before pre_call_hook (#29654)
* fix(proxy): merge model-level guardrails before pre_call_hook DB/UI-assigned guardrails (litellm_params.guardrails) only fire on post_call paths today: _check_and_merge_model_level_guardrails is called in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but never before pre_call_hook in common_request_processing.py:963. PR #23774 fixed the non-streaming post_call case; pre_call was left broken. At the pre_call site, add_litellm_data_to_request strips client-supplied metadata.model_info (pricing spoofing guard) and route_request hasn't run yet, so model_info.id is unavailable. Extend the helper to fall back to llm_router.get_deployment_by_model_group_name(model_alias) when model_id is missing — that uses the O(1) model-name index already maintained by the router. Closes #29652 * fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload Addresses veria-ai Medium finding + proxy-infra CI failure on this PR. ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format was omitting the field, so a key with mcp_rpm_limit could exceed it via the MCP path. Reads from kwargs.get("mcp_rate_limit_server_name") to match how pre_call_tool_check resolves the alias-then-server-name fallback before invoking hooks. * fix(proxy): union guardrails across group deployments on alias fallback Addresses second veria-ai Medium on #29654: the alias fallback called get_deployment_by_model_group_name(), which returns ONE deployment. A guardrail set on a non-first deployment would silently not run on pre_call when the model_id is missing. Switch to get_model_list(model_name=...) and take the UNION of litellm_params.guardrails across all matching deployments (with dedup). Trade-off documented in the comment: pre_call cannot know which deployment route_request will select, so the conservative choice is to apply any guardrail set on any eligible deployment. Updated test stubs to use get_model_list. Added 3 new tests covering union, dedup, and the all-empty case. * test(model_level_guardrails): align integration test with get_model_list union API * fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint Addresses 3rd veria-ai Medium on #29654: add_litellm_data_to_request preserves client-supplied metadata.model_info when the caller's key/team has allow_client_pricing_override. The pre_call merge previously trusted that id, so a caller could spoof an unguarded model_info.id while requesting a guarded alias and bypass guardrails. New `trust_client_model_info: bool` param on the helper. The pre_call call site passes False; post_call paths (existing) keep True. Also fixes the ruff failure on the union loop: pulled the .get() into a local + isinstance(list) check before iterating, so mypy stops complaining about `object` not being iterable. 2 new regression tests covering spoof-and-bypass + default-trust behavior. * fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance Addresses two more reviewer findings on #29654: veria-ai Medium: route_request resolves team-scoped public model names with metadata.user_api_key_team_id. The pre_call alias fallback called get_model_list(model_name=...) without the team_id, so team-scoped deployments were invisible and their pre_call guardrails silently skipped. Now reads team_id from metadata or litellm_metadata and passes it to get_model_list. greptile P1: the isinstance(deployment_guardrails, list) guard added for mypy narrowing silently dropped bare-string guardrail values that the existing post_call path used to truthy-accept. Restored by wrapping a scalar string into a one-element list on both paths. 4 new tests: team_id passthrough (metadata + litellm_metadata), scalar on post_call, scalar on alias-union. 36/36 tests pass. * style: black formatting on _check_and_merge_model_level_guardrails team_id assignment * chore: ruff format * fix(lint): remove unused noqa PLR0915 directive RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic because PLR0915 is not in this repo's enabled ruff rule set (lint.extend-select in ruff.toml), so the directive suppresses nothing and fails the lint job. * refactor(proxy): hoist guardrail-merge import to module top The pre_call guardrail-merge helper was imported inside common_processing_pre_call_logic with a # noqa: PLC0415, which the type-discipline gate counts as an unexplained suppression (LIT003). The inline import's cyclic-import justification does not hold: this module already imports from litellm.proxy.utils at top level, and utils.py does not import common_request_processing at module load. Fold the helper into the existing top-level import and drop the inline import, clearing the suppression instead of budgeting for it. --------- Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com> |
||
|
devin-ai-integration[bot]
|
8536e3b80e
|
fix(proxy): source /v1/models token limits from the cost map instead of Router.get_model_group_info (#33721)
* fix(proxy): source /v1/models token limits from cost map instead of Router.get_model_group_info Resolves the per-model get_model_group_info fan-out on GET /v1/models (and /models) that pegged the event loop on wildcard listings (#33636). create_model_info_response now reads max_input_tokens/max_output_tokens from litellm.get_model_info (the static cost map) rather than the router, which aggregated and deepcopied every deployment in a group per listed model. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): inject model-info lookup into create_model_info_response for deterministic coverage Inject the cost-map lookup (defaulting to litellm.get_model_info) so the except and max_output_tokens branches are exercised deterministically and the token-limit tests no longer hardcode mutable cost-map values. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): surface custom deployment token limits on /v1/models via cheap index lookup Add Router.get_configured_token_limits, an O(1) model-name index lookup that reads a concrete deployment's configured max_input_tokens/max_output_tokens without triggering pattern matching or deep copies. create_model_info_response layers this over the cost map so custom deployments absent from the cost map still surface their limits, and admin-configured limits override cost-map defaults, while wildcard-expanded names stay on the fast path. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
1aed5e1bbd
|
test(proxy/utils): pin bottom-of-file helper behavior (#29509)
* test(proxy/utils): pin bottom-of-file helper behavior Pin current behavior of the bottom-of-file pure-function helpers in litellm/proxy/utils.py (projection, team config, time helpers, guardrail merge, error helpers, URL/path helpers, premium gate, model access, and misc DB/API-key helpers). Adds tests/test_litellm/proxy/utils/helpers/ with one happy + one error test per pinned symbol; folds the prior single-test tests/test_litellm/proxy/test_utils.py into test_url_helpers.py and deletes the old file. _pin_check.py and _coverage_check.py serve as local stopping gates. Adds tests/test_litellm/proxy/utils to the existing test-path block in .github/workflows/test-unit-proxy-endpoints.yml. Plan: https://www.notion.so/37343b8acdab81f68f39f66915f62bcf Pin list: https://www.notion.so/37343b8acdab8150acdbf40e5756869f * test(proxy/utils): apply greptile fixes to behavior-pinning gates Address findings from the sibling PR1/PR2 greptile reviews that also apply to this PR: - Commit pin_list.txt alongside the gate script (was previously a gitignored .pin_list.txt fetched from Notion). The gate is now reproducible without out-of-band setup. - Resolve the coverage region by locating the first pinned symbol's def line in litellm/proxy/utils.py at runtime, instead of hardcoded line numbers that drift when lines above shift. - Word-boundary the pin reference check so pins like update_spend do not falsely match update_spend_logs_job. - Drop the dead _harness_smoke_test.py exclusion; the test_*.py glob already filters underscore-prefixed files. * test(proxy/utils): drop local-only stopping-signal scripts Remove _pin_check.py, _coverage_check.py, and pin_list.txt. These were dev-time tooling for knowing when test authoring was done; they are not wired into CI and the test files themselves are the merge artifact. --------- Co-authored-by: Claude <noreply@anthropic.com> |