mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-03 02:22:24 +00:00
2610 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
adc4a838bd
|
Merge origin/litellm_internal_staging into litellm_feat/v1.84.0-mcp-gateway-jwt-auth | ||
|
ryan-crabbe-berri
|
123a9ce487
|
fix(mcp): extend key access-group union to MCP servers (#28890)
A key's unified access_group_ids now extend the team's MCP scope instead of being capped by it — mirrors the model-side union from LIT-2404. The group's assigned_team_ids / assigned_key_ids still gate the override, so team members can't pull in MCPs via a foreign team's group. Resolves LIT-3189 |
||
|
Cursor Agent
|
1d73f1bac5
|
fix(mcp): reconcile cold-start bypass with x-mcp-servers header and skip non-absolute WWW-Authenticate fabrication
- _parse_mcp_server_names_from_path now fails closed when the x-mcp-servers header introduces any target not present in the path-derived target set, closing a header/path mismatch where the cold-start passthrough bypass could otherwise admit anonymously while the header advertises a non-passthrough server. - MCPUpstreamAuthError.to_http_exception no longer emits a relative resource_metadata URI when base_url is missing; per RFC 9728 3.2 the URI must be absolute, so we skip fabrication entirely rather than send a challenge strict MCP clients will reject. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
172c11b352
|
feat(mcp): add dedicated oauth_passthrough flag for non-oauth2 pass-through
Previously is_oauth_passthrough reused delegate_auth_to_upstream — a flag scoped to oauth2 servers (PKCE bypass) — to gate OAuth pass-through for auth_type=none servers. Overloading it risked regressing existing deployments that set delegate_auth_to_upstream, since the same flag would silently start driving pass-through (discovery proxying, 401 challenges, upstream 401/403 propagation) on non-oauth2 servers. Introduce a separate oauth_passthrough opt-in so the two behaviors never imply each other: - MCPServer.is_oauth_passthrough now requires oauth_passthrough (not delegate_auth_to_upstream). - Persist oauth_passthrough on LiteLLM_MCPServerTable (new column + migration) and wire it through config/DB load and API responses. - UI splits the single toggle into two: "Delegate auth to upstream (PKCE passthrough)" for oauth2 and "OAuth pass-through" for auth_type=none servers forwarding Authorization. Adds backend tests (property, round-trip, and a regression guard that delegate_auth_to_upstream alone never enables pass-through) and UI tests for the toggle split. |
||
|
Sameer Kankute
|
d52fbfb458
|
Litellm oss staging 250526 (#28770)
* fix(mcp): handle OAuth IdP error responses in /callback (LIT-2750) Per RFC 6749 section 4.1.2.1, when the IdP rejects an OAuth authorization request it redirects back to the client with ?error=...&error_description=... and no code. The MCP /callback handler declared code and state as required query params, so FastAPI rejected such error responses with a 422 before the handler ran -- stranding the MCP client waiting on the loopback. This change: - Makes code and state optional and accepts the RFC-defined error, error_description, and error_uri params. - When state decodes to a trusted client redirect_uri, propagates the error params back to that URI with the client's original (un-wrapped) state preserved, so the client's OAuth library can surface the failure. - When state is missing/undecryptable or the encoded redirect_uri is no longer trusted, renders a 400 HTML page with the (HTML-escaped) error details instead of leaking to an attacker-controlled redirect. - Preserves the existing success path (code + state -> 302 to validated client redirect_uri with original state). Fixes LIT-2750. * test(mcp): regression tests for /callback handling IdP error responses (LIT-2750) Adds a new test module covering the LIT-2750 fix: the MCP OAuth /callback endpoint must accept IdP error responses (e.g. ?error=access_denied) per RFC 6749 section 4.1.2.1 instead of returning a 422 because ``code`` is missing. Coverage: - IdP error with no state -> 400 HTML page surfacing the error. - HTML escaping of user-controlled error / error_description fields. - IdP error with a trusted (loopback) state -> 302 propagating error / error_description / original client state to the client. - IdP error with an untrusted redirect_uri encoded in state -> 400 inline (no open-redirect to attacker-controlled origin). - IdP error with an undecryptable state -> 400 HTML fallback. - Bare GET /callback with no params -> 400 HTML (not Pydantic 422). - Success path (code + state) still 302 to validated client redirect_uri with the original (un-wrapped) state preserved. * refactor(mcp): drop unused _OAUTH_ERROR_PARAMS constant (Greptile P2) The tuple was leftover scaffolding from an earlier draft of the LIT-2750 fix; nothing references it. The explanatory RFC 6749 §4.1.2.1 comment block above the callback handler covers the same intent. * fix(mcp/oauth): preserve empty original_state and clarify missing-param error in /callback Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(thinking): handle None thinking param in is_thinking_enabled (#28598) Squash-merged by litellm-agent from Terrajlz's PR. * feat(helm): support tpl rendering in podAnnotations (#28609) Squash-merged by litellm-agent from devauxbr's PR. * fix: apply black formatting to base_llm chat transformation Fix CI black --check failure on is_thinking_enabled return formatting. Co-authored-by: Cursor <cursoragent@cursor.com> * merge main (#28836) * fix(proxy): Bedrock Knowledge Base pass-through: preserve SigV4 headers and signed request body (#27526) * Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <cursoragent@cursor.com> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> * chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728) * chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214 The original account (888602223428) was put under a security restriction by AWS after a root access key leaked in a PR comment. While that account works its way through the AWS Support unlock process, Bedrock-touching CI tests have been migrated to a fresh account (941277531214). Changes: - Replace 26 hardcoded references to 888602223428 with 941277531214 across 8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime ARNs, batch execution role ARN, and example proxy config). - The provisioned-model and imported-model ARNs are referenced only from mocked unit tests — no AWS resources to recreate. - The batch execution IAM role has been recreated in the new account with the same name and equivalent permissions. - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC, hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account under the same names — see tools/agentcore-deploy/ in a follow-up. CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME were updated separately via the CircleCI API to point at the new account. Smoke-tested locally against the new account: aws bedrock-runtime converse --region us-west-2 \ --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \ --messages '[{"role":"user","content":[{"text":"ping"}]}]' → 200, model returned 'pong' Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes The first migration commit replaced just the account ID, but AgentCore auto-assigns a random 10-char suffix to every runtime on creation — we can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the new account. Updated the AgentCore-runtime ARNs in the three files that reference real runtime IDs (not the mock-based unit-test ARNs). Deployed runtimes: arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy Both runtimes are status=READY and pass a smoke invoke: $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}' → 200, {"result": "echo: ping"} The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the deploy artifacts). Tests that only verify the SDK wiring will pass; if any test asserts on agent output content, swap the echo for the real agent. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(tests): point Bedrock batch tests at new-account S3 bucket The account migration (888602223428 -> 941277531214) was a flat account-ID swap, which only rewrites ARNs that embed the account number. S3 bucket names carry no account ID, so the live Bedrock batch tests still uploaded to `litellm-proxy` — a bucket that lives in the old account. S3 names are globally unique, and the old account still holds that name, so it can't be recreated in the new account. Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees global uniqueness). The bucket must be created in 941277531214 and the batch execution role granted s3:GetObject/PutObject/ListBucket on it before this job is run in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): point live S3 logging test at new-account bucket Same account-ID-free blind spot as the batch bucket: `load-testing-oct` lives in the old account and its name can't be reused globally. The `logging_testing` CI job is wired into the workflow and runs test_basic_s3_logging, which uploads to this bucket with the CI env creds, then lists and deletes objects — a live dependency. Rename to `load-testing-oct-941277531214`. The bucket must exist in the new account with the CI IAM principal granted s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore(tests): repoint Bedrock guardrail IDs to new-account guardrails The migration left guardrail IDs untouched (no account ID in them), so all live guardrail tests failed with "guardrail identifier or version does not exist" against 941277531214. Recreated both guardrails in the new account and updated the hardcoded IDs: - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD, with explicit inputAction=ANONYMIZE so masking applies to INPUT, which is the source litellm's moderation hook sends) - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set to the exact string the tests assert on) Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the guardrailConfig in test_bedrock_completion.py. Verified locally: the 5 previously-failing guardrail tests now pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): migrate legacy models to current inference profiles The new CI account (941277531214) cannot invoke legacy Bedrock models (AWS gates them: "marked by provider as Legacy... not actively using in the last 30 days"). Migrated the live-call tests: - anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0 - anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0 Current Claude models on Bedrock require the us. inference-profile prefix (bare on-demand ids are rejected). cohere.command-r-plus has no working replacement (all Cohere is legacy- gated in the new account): swapped to claude-haiku-4-5 in provider- agnostic param lists. amazon.titan-image-generator skipped (no working replacement). Mocked/transformation/cost tests that reference the legacy strings are intentionally left unchanged. Verified live against the new account. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): repoint SageMaker + Knowledge Base to new-account resources These referenced account-scoped resources by hardcoded id that only existed in the old account, so the migration's account-ID swap missed them. Recreated in 941277531214 and repointed: - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614 -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge) - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless vector store + titan-embed-text-v2, seeded with a LiteLLM doc) Verified live: test_sagemaker.py (12 passed) and test_bedrock_knowledgebase_hook.py (12 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214) claude-opus-4-7 is listed in the new Bedrock CI account's foundation models but invoke is denied (AccessDeniedException: "not available for this account"). Bedrock access to the flagship Opus requires an AWS Sales request, not the self-serve model-access toggle, so it can't be enabled inline with the rest of the account migration. Add an optional `skip_reason` to ModelEntry and set it on the bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip. Cell count (231) and route coverage are unchanged, so the structural asserts still pass. Restore coverage by deleting the one skip_reason line once access is granted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(bedrock): swap/skip legacy-gated models unavailable on new CI account The migrated AWS account (941277531214) cannot access several models that the old account could, so the remaining red CI jobs were hitting real Bedrock "Access denied / Legacy" and "account not authorized" errors: - image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is legacy-gated), matching the existing titan skip. - batches: skip test_async_file_and_batch (Bedrock batch inference is not authorized on the new account; requires an AWS support case). - litellm_overhead: swap legacy claude-3-5-haiku for the active us.anthropic.claude-haiku-4-5 inference profile. - test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account - e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference is not authorized on account 941277531214) and migrate the missed s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214. - build_and_test: swap legacy bedrock claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured output e2e test. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (#28791) Replace the silent skips added for the new CI account with noisier behavior: - reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present) instead of skipping, so the missing entitlement stays visible in CI; they still skip when AWS creds are absent (local dev) - Bedrock batch inference tests: drop the skip so they run and fail until batch access is granted - Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the transform + cost-tracking path stays under test without live model access https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT Co-authored-by: Claude <noreply@anthropic.com> * test(bedrock): use pytest.xfail for known-failing opus-4-7 cells Replace pytest.fail with pytest.xfail when a model has a fail_reason, so known-broken cells stay visible as XFAIL without keeping CI red. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(otel): export SERVER span on management-endpoint success without http_request (#28794) Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> * chore(ci): merge dev branch (#28801) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> * chore(ci): merge dev branch (#28657) * feat(dashboard): navbar hierarchy + Agent Platform notifications (#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in #27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash #27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(ui): show 2-decimal precision for max_budget on key overview (#28809) The Key Info Overview tab's Spend card truncated sub-dollar budgets to "$0" because formatNumberWithCommas defaults to 0 decimals. The Settings tab passes 2; align the overview so a $0.10 budget renders as "$0.10". Resolves LIT-2845 * feat(proxy): allow `llm_api_routes` virtual keys to list MCP servers (#28442) * feat(proxy): allow llm_api_routes virtual keys to list MCP servers Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET /v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that virtual keys configured with `allowed_routes=["llm_api_routes"]` can discover the MCP servers they have access to. Previously these calls failed with 'Virtual key is not allowed to call this route. Only allowed to call routes: [llm_api_routes]'. The GET handlers already sanitize the response for restricted virtual keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping credential-bearing fields (url, headers, env). Write methods (POST/PUT/DELETE) on the same paths remain gated by the existing handler-level admin role checks. The new discovery list is intentionally kept OUT of `mcp_inference_routes`, so `is_llm_api_route()` still returns False for these paths — this preserves the existing contract that DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP servers. Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * refactor(proxy): make MCP discovery carve-out method-aware Replace the `mcp_discovery_routes` group in `llm_api_routes` with a method-aware special case inside `is_virtual_key_allowed_to_call_route`. Virtual keys with allowed_routes=["llm_api_routes"] are now permitted to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} — non-GET methods and multi-segment admin sub-paths fall through to the existing 403. This keeps the general llm_api_routes list free of management paths and avoids accidentally exposing POST/PUT/DELETE writes through the route-check layer. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> * chore(ci): merge dev branch (#28807) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> * fix(team): keep team_alias cache in sync on _cache_team_object writes (#28737) * fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on #28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o. * fix(proxy): hydrate wildcard discovery credentials (#28284) (#28822) * fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> * ci: add daily oss-agent-shin branch creation workflow (#28829) Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC. Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> * test(proxy): add harness for proxy_server.py behavior-pinning (#28827) * test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice. * feat(openai): apply regional-processing cost uplift for EU/US data residency (#28626) * feat(openai): apply regional-processing cost uplift for EU/US data residency OpenAI charges a 10% uplift on the latest GPT models when requests are served from a regionalized hostname (eu./us.api.openai.com). Infer the region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`, and multiply the computed cost by a per-model `regional_processing_uplift_multiplier_<region>` field. https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW * test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema * fix(cost): tighten data_residency inference and restore model_cost in tests - Only infer OpenAI data_residency when custom_llm_provider == "openai"; drop the implicit None fallback so non-OpenAI callers can't accidentally pick up a regional tag from a stray OpenAI hostname. - _local_model_cost_map fixture now snapshots and restores litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak state across the session. * refactor(openai): move data_residency helper under llms/openai * fix: thread data_residency through realtime stream cost calculation Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(cost): thread data_residency through batch_cost_calculator Apply the OpenAI regional-processing uplift multiplier to retrieve_batch cost paths so Batch API requests served via eu./us.api.openai.com are priced at the same uplifted token rates as completions/transcriptions. * refactor(openai): encapsulate provider check inside infer_openai_data_residency Move the custom_llm_provider == "openai" guard from get_litellm_params into the helper itself so the core utility no longer carries provider-specific dispatch logic. Callers pass through the provider unconditionally; the helper returns None for any non-OpenAI provider. * fix(responses): thread data_residency through Responses logging params The Responses API paths build their logging litellm_params dict after provider resolution but did not include data_residency, so cost calc saw None even when the effective api_base was a regional OpenAI host. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> * fix: preserve OTEL response payload and remove duplicate constant - _emit_management_endpoint_otel_span now passes result as response on success - remove duplicate _CREDENTIAL_LITELLM_PARAM_FIELDS assignment in model_checks Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix: address bug detection findings - pass_through_endpoints: use request.method instead of hardcoded POST in streaming SigV4-signed request path for consistency with the non-streaming branch - llm_cost_calc/utils: hoist DataResidency value set to a module-level frozenset to avoid rebuilding it on every cost calculation - example_config_yaml/oai_misc_config: replace real-looking AWS account ID with placeholder 123456789012 in example bucket and role ARN Co-authored-by: Yassin Kortam <yassin@berri.ai> * chore(github_copilot): refresh model catalog from upstream /models API (#28055) Aligns the github_copilot catalog with values returned by Copilot's public /models endpoint (capabilities.limits + capabilities.supports + model.supported_endpoints). - Adds 10 new model entries: claude-opus-4.7, claude-sonnet-4.6, gemini-3-flash-preview, gemini-3.1-pro-preview, gpt-4-0125-preview, gpt-5.2-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5, oswe-vscode-prime. - Updates max_input_tokens for existing entries to reflect each model's true context window (e.g. gpt-4o-mini 64000 -> 128000, gpt-5-mini 128000 -> 264000, gpt-5.3-codex 128000 -> 400000, claude-haiku-4.5 128000 -> 200000). - Adds supports_reasoning, supports_response_schema, supports_function_calling, supports_parallel_function_calling, supports_vision based on capabilities.supports. - Declares supported_endpoints for entries missing it (e.g. gpt-3.5-turbo, gpt-4o, embeddings). - For responses-only models (gpt-5.2-codex, gpt-5.4, gpt-5.4-mini, gpt-5.5), sets mode to 'responses'. - gpt-41-copilot.mode changes from 'completion' to 'chat' because Copilot reports capabilities.type = 'chat'. Revertible on request. Pricing fields and other manually-curated values are preserved. * feat(datadog): emit litellm.overhead.latency as a standalone Datadog metric (#28831) Adds a new `litellm.overhead.latency` gauge metric to `DatadogMetricsLogger` (the `/api/v2/series` path). The value is sourced from `hidden_params["litellm_overhead_time_ms"]` already computed in `ResponseMetadata` and exposed in `StandardLoggingPayload`. Matches the Prometheus integration which exposes the same value via `litellm_overhead_latency_metric`. Emitted in seconds (ms ÷ 1000) for consistency with the other latency series. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Shin <shin@litellm.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> * feat(arize): route Phoenix traces via per-project TracerProviders (#28876) Use LRU-cached TracerProviders with project-scoped OTEL Resources so team/key metadata routes traces correctly. On the proxy, project selection is limited to server-controlled user_api_key_auth_metadata; client metadata fields stay banned. * fix(arize_phoenix): skip _emit_semantic_logs on failure path Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(arize_phoenix): skip raw request logging and metrics on failure path Restores pre-refactor behavior: _handle_failure no longer emits raw-request sub-spans or records OTEL metrics, matching the original _handle_failure that did not call these helpers. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(security): close two medium telemetry trust-boundary issues Issue 1 (arize_phoenix.py — caller-controlled telemetry routing): - _is_proxy_request no longer detects proxy mode by checking user_api_key_auth_metadata in request metadata. That field is user-supplied, so an authenticated caller could fake proxy-mode detection and have _project_from_metadata_dict read their own dict for project selection, routing telemetry to arbitrary Arize/Phoenix projects. Proxy mode is now determined solely by the server-set proxy_server_request field in litellm_params. - auth_utils.py adds user_api_key_auth_metadata to the banned request body params list so the proxy rejects any attempt to supply the field at the HTTP layer. The field is server-reserved: it is written exclusively by add_user_api_key_auth_to_request_metadata from the authenticated key's database record after the ban check runs. Issue 2 (management_helpers/utils.py — API key in OTEL span): - _emit_management_endpoint_otel_span stripped plaintext credential fields (key, token, api_key, secret, …) from the response dict before passing it to the OTEL success hook. dict(result) on a Pydantic GenerateKeyResponse includes the freshly-generated key field, which would previously be written as a span attribute to every configured OTEL collector/backend. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Terrajlz <info@jouleselectrictech.com> Co-authored-by: Bruno Devaux <devaux.br@gmail.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo <mateo@Mateos-MacBook-Pro.local> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Yassin Kortam <yassinkortam@Yassins-MacBook-Pro.local> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> Co-authored-by: ishaan-berri <155045088+ishaan-berri@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> Co-authored-by: rinto <54238243+ririnto@users.noreply.github.com> Co-authored-by: Shin <shin@litellm.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> |
||
|
mateo-berri
|
ccea19d66f
|
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_feat/v1.84.0-mcp-gateway-jwt-auth
# Conflicts: # litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py |
||
|
Shivam Rawat
|
fbff60e9d9
|
fix(proxy): strip LiteLLM policy tracking from OpenAI batch metadata (#28425)
* fix(proxy): strip LiteLLM policy tracking from OpenAI batch metadata Batch create was failing with `Invalid type for 'metadata.applied_policies': expected a string, but got an array instead` whenever a policy attachment matched the request. The policy engine helpers wrote `applied_policies`, `applied_guardrails`, and `policy_sources` into `data["metadata"]` unconditionally, and `/v1/batches` forwarded that dict straight to OpenAI, which only accepts string values. - Route proxy-internal tracking into `litellm_metadata` for batch/file routes via a shared `_get_or_create_proxy_metadata_bucket` helper. - Sanitize `data["metadata"]` in `create_batch` to drop known internal keys and non-string values before building the OpenAI request. - Cover both behaviors with unit + endpoint tests. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): merge metadata buckets for batch policy response headers Ensure get_logging_caching_headers reads both metadata and litellm_metadata so policy/guardrail headers are emitted on batch routes with user metadata, and log dropped non-string OpenAI metadata at debug level. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
ryan-crabbe-berri
|
f75a7c6b22
|
fix(model-edit): allow clearing custom pricing on wildcard models (#28719)
* fix(model-edit): allow clearing custom input/output cost on wildcard deployments A user-set pricing override on a `/model/*` wildcard deployment could not be removed: clearing the Input/Output Cost fields in the UI succeeded visually, but the next read still showed the old values because both `litellm_params` and `model_info` (mirrored via `SPECIAL_MODEL_INFO_PARAMS`) retained the original rates. UI: when the pricing field is touched but left empty, send `null` instead of dropping it from the payload so the backend sees the clear intent. The cache-read-cost fallback now guards against `null` as well as `undefined` so a cleared input cost cannot silently wipe the cache-read override. Backend: `update_db_model` honors explicit-null clears, but ONLY for `SPECIAL_MODEL_INFO_PARAMS` (the 4 pricing fields). Restricting the null-clear path prevents a team-scoped caller from using this codepath to null out privileged fields like `team_id` or access groups. Tests cover both clear paths (`litellm_params` and `model_info`), the SPECIAL_MODEL_INFO_PARAMS mirror, PATCH semantics for omitted fields, and the security guard that non-pricing nulls don't reach the merged dict. Resolves LIT-3250 * fix(model-edit): run null-clears after both merges, not interleaved The previous version cleared `model_info` from inside the litellm_params merge block, but the subsequent `model_info.update(...)` re-injected the old pricing because the UI's PATCH carries the full model_info blob with the stale values still in it. Move the explicit-null clear pass to after both merges so a model_info passthrough cannot resurrect cleared fields. Adds a regression test for the realistic UI submit shape (both blobs in the patch, model_info still holding the old pricing). * test(e2e): clear-custom-pricing flow with create/delete cleanup Covers the dashboard model edit form's pricing-clear flow end-to-end: seeds a deployment with custom input/output pricing, drives the UI to clear both fields, asserts the outgoing PATCH sends explicit nulls, and confirms via /v2/model/info that the override is gone from both litellm_params and model_info. The dashboard DB persists across this suite, so beforeEach creates a uniquely-named deployment and afterEach POSTs /model/delete to leave the DB clean regardless of test outcome. * fix(model-edit): extend pricing clear to cache_read and cache_write costs Pre-existing parallel of the wildcard input/output cost bug: cleared cache_read_input_token_cost and cache_creation_input_token_cost overrides silently persisted because the UI omitted the key (delete or fallback) and the backend null-clear allowlist did not cover them. - types/router.py: add cache_read_input_token_cost and cache_creation_input_token_cost to SPECIAL_MODEL_INFO_PARAMS, so they are mirrored between litellm_params and model_info by Deployment.__init__ and honoured by the null-clear loop in update_db_model. - model_info_view.tsx: emit explicit null for touched-but-empty cache_read and cache_write fields. Preserve the input_cost->cache_read mirror only when cache_read itself was not touched. - model_management_endpoints.py: update the allowlist comment. - Tests: three new unit tests for cache clear paths and a preserve check; the e2e spec now seeds, clears, and asserts null PATCH + key-absence for all four pricing fields. |
||
|
Mateo Wang
|
48d7e15b83
|
chore(admin-ui): regenerate static export with trailingSlash: true (#28112)
* chore(admin-ui): regenerate static export with trailingSlash: true Rebuilds litellm/proxy/_experimental/out/ from ui/litellm-dashboard with `trailingSlash: true` enabled in next.config.mjs. Next.js now emits every route as <dir>/index.html (e.g. mcp/oauth/callback/index.html) instead of <dir>.html with a sibling metadata-only directory, which fixes the 404 on extensionless URLs served through FastAPI's StaticFiles(html=True) mount. This is the build artifact half of the fix; the config change, Dockerfile cleanup, and regression test live in the follow-up source PR that stacks on top of this branch. * fix(admin-ui): emit nested routes as <dir>/index.html (#28106) Linear and other OAuth providers redirect the user back to /ui/mcp/oauth/callback?code=...&state=... after the consent step. The packaged Next.js static export only produced /ui/mcp/oauth/callback.html, so FastAPI's StaticFiles served a 404 on the extensionless URL and the OAuth handshake never completed. The Dockerfile.non_root build step tried to paper over this at image-build time with `for html_file in *.html; do ...`, but that shell glob does not recurse, so nested routes like mcp/oauth/callback.html were left stranded next to an empty mcp/oauth/callback/ directory containing only Next.js metadata. The runtime restructure step in proxy_server.py was then skipped because the .litellm_ui_ready marker had already been dropped. Set trailingSlash: true in the dashboard's Next.js config so the export emits every nested route as <dir>/index.html natively. The Dockerfile loop is now a no-op for the bundled UI and has been removed; the .litellm_ui_ready marker is still written so the proxy keeps skipping the redundant Python restructure step at startup. Stacks on top of the static export regeneration in the parent branch. * chore: restore origin/litellm_internal_staging out files |
||
|
yuneng-jiang
|
f38c16c71e
|
test(proxy): add harness for proxy_server.py behavior-pinning (#28827)
* test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice. |
||
|
yuneng-jiang
|
7cd98508e7
|
fix(team): keep team_alias cache in sync on _cache_team_object writes (#28737)
* fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on #28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o. |
||
|
ryan-crabbe-berri
|
66f10bceea
|
feat(proxy): allow llm_api_routes virtual keys to list MCP servers (#28442)
* feat(proxy): allow llm_api_routes virtual keys to list MCP servers
Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET
/v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that
virtual keys configured with `allowed_routes=["llm_api_routes"]` can
discover the MCP servers they have access to. Previously these calls
failed with 'Virtual key is not allowed to call this route. Only allowed
to call routes: [llm_api_routes]'.
The GET handlers already sanitize the response for restricted virtual
keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping
credential-bearing fields (url, headers, env). Write methods
(POST/PUT/DELETE) on the same paths remain gated by the existing
handler-level admin role checks.
The new discovery list is intentionally kept OUT of
`mcp_inference_routes`, so `is_llm_api_route()` still returns False
for these paths — this preserves the existing contract that
DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP
servers.
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
* refactor(proxy): make MCP discovery carve-out method-aware
Replace the `mcp_discovery_routes` group in `llm_api_routes` with a
method-aware special case inside `is_virtual_key_allowed_to_call_route`.
Virtual keys with allowed_routes=["llm_api_routes"] are now permitted
to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} —
non-GET methods and multi-segment admin sub-paths fall through to the
existing 403. This keeps the general llm_api_routes list free of
management paths and avoids accidentally exposing POST/PUT/DELETE
writes through the route-check layer.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com>
|
||
|
yuneng-jiang
|
d98ada8c3f
|
chore(ci): merge dev branch (#28657)
* feat(dashboard): navbar hierarchy + Agent Platform notifications (#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in #27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash #27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
milan-berri
|
f45909cb81
|
fix(proxy): Bedrock Knowledge Base pass-through: preserve SigV4 headers and signed request body (#27526)
* Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <cursoragent@cursor.com> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <cursoragent@cursor.com> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <cursoragent@cursor.com> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
ryan-crabbe-berri
|
4148667671
|
Fix spend logs v2 route permissions (#28705)
Some checks failed
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Has been cancelled
Unit Tests: Security / security (push) Has been cancelled
Unit Tests: Proxy DB Operations / auth-checks (push) Has been cancelled
Unit Tests: Proxy DB Operations / budgets (push) Has been cancelled
Unit Tests: Proxy DB Operations / custom-logging (push) Has been cancelled
Unit Tests: Proxy DB Operations / db-and-spend (push) Has been cancelled
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Has been cancelled
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Has been cancelled
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Has been cancelled
Unit Tests: Proxy DB Operations / key-generation (push) Has been cancelled
Unit Tests: Proxy DB Operations / logging-misc (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-runtime (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-server-core (push) Has been cancelled
Unit Tests: Proxy DB Operations / schema-migration (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-utils (push) Has been cancelled
Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: ryan-crabbe-berri <ryan-crabbe-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
5f73ad4fe7
|
fix(team): refresh team cache on team_model_add/delete (LIT-3244) (#28683)
* fix(team): refresh team cache on team_model_add/delete (LIT-3244)
team_model_add and team_model_delete wrote to the DB but did not
invalidate the in-memory LiteLLM_TeamTableCachedObj used by
common_checks. After the v1.83.14 common_checks centralization made
team.models authoritative on /v1/files and /v1/vector_stores/*,
adding a Team-BYOK model silently failed to grant the new public
model name to team members until the cache TTL expired (and a
removed model kept working until then on the symmetric path).
Extract the cache-refresh snippet from update_team into a small
helper and apply it consistently at all three team-write sites.
* test: also assert updated models in team-cache-refresh pin
Strengthens the LIT-3244 regression test to also assert
`call_kwargs["team_table"].models` matches the updated row,
not just `team_id`. Both `existing_team` and `updated_team`
share `team_id` in the test setup, so the previous assertion
would have passed even if the implementation accidentally cached
the pre-mutation row.
Greptile review feedback.
* fix(team): hydrate object_permission on cache-refreshing team updates
The Prisma update calls in update_team, team_model_add, and
team_model_delete returned a team row with object_permission_id set
but object_permission=None (the relation was not requested via
include=). _refresh_cached_team then wrote that to the in-memory
LiteLLM_TeamTableCachedObj, and the cache-hit path in get_team_object
returns the cached object without re-hydrating. Downstream consumers
(validate_key_search_tools_against_team, the MCP/agent authz paths)
treat a missing object_permission as no team-level restriction, so
a team-write op silently dropped object-permission enforcement until
the cache TTL expired or a DB-fetch path re-hydrated it.
Add include={"object_permission": True} to all three updates so the
refresh writes a complete cached team. Extend the LIT-3244 regression
test to pin both the cached object_permission and the include shape
on the Prisma call.
Surfaced in PR review of LIT-3244.
|
||
|
Yassin Kortam
|
2eab9ee2c0
|
perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths (#28289)
* perf: reduce per-request and per-chunk overhead across Anthropic streaming hot paths
- Introduce pure-text fast-path in `_build_complete_streaming_response` that collapses O(N) `content_block_delta` events into a single equivalent SSE event before conversion, eliminating per-output-token Pydantic `ModelResponseStream` construction; non-text streams (tool_use, thinking, citations) fall back to the unchanged legacy path
- Skip agentic streaming wrapper entirely when no callback overrides `async_should_run_agentic_loop`; the wrapper buffered every chunk and rebuilt the SSE response only to call hooks that all return `(False, {})` — a pure no-op for the default config
- Serialize request body once (`json.dumps`) for both the pre-call log input and the wire, instead of twice; avoids a full O(payload) scan per request, significant for long-context Claude Code histories
- Add fast path in `async_streaming_data_generator` that bypasses the per-chunk `async_post_call_streaming_hook` coroutine await, response-string materialization, and cost-injection call when no callback/guardrail/cost-injection is active (the default config)
- Resolve `_DD_STREAMING_TRACE_ENABLED` once at import time; eliminate per-chunk `NullSpan` context manager allocation when Datadog tracing is disabled (the default)
- Memoize `get_type_hints(AnthropicMessagesRequestOptionalParams)` with `@lru_cache(maxsize=1)` — resolves once per process instead of once per `/v1/messages` request (~80µs each)
- Hoist `cost_injection_active` out of the per-chunk loop in `chunk_processor`; eliminates repeated `getattr` + endpoint-type checks on every streamed byte chunk
- Extract `_build_passthrough_logging_result` from `_route_streaming_logging_to_handler` as a standalone static method to facilitate future off-loop dispatch
- Convert `async_sse_data_generator` from an `async for: yield` trampoline to a direct return of the underlying generator, removing one async-generator layer per streamed chunk
- Skip redundant `strip_empty_text_blocks_from_anthropic_messages` scan in `anthropic_messages_handler` when the async wrapper already sanitized (signalled via `_litellm_messages_presanitized` sentinel, popped before reaching provider params)
- Gate debug log `f-string` evaluation behind `isEnabledFor(DEBUG)` in both the streaming generator and the transformation layer to avoid serializing entire message payloads on every request at non-debug log levels
- Add benchmark script (`scripts/benchmark_anthropic_messages_perf.py`) with a local mock Anthropic SSE provider for reproducible TTFT and TPM measurement across commits/branches
- Add parity tests asserting fast-path and legacy-path produce byte-identical logged/billed payloads, plus unit tests for agentic hook detection, pre-serialized body reuse, and memoized key resolution
* perf: address greptile review for anthropic streaming hot path
- Bail to legacy in `_collapse_pure_text_chunks` when content_block_delta
events from different block indexes are observed without an intervening
flush. Anthropic sends blocks strictly sequentially, but defensive bail
prevents silent text-merging if the protocol ever interleaves.
- Replace leaf-class `__dict__` check for `async_post_call_streaming_hook`
in `_callback_capabilities` with a function-identity comparison that
walks the MRO. A vendor base class can carry the override and the
registered class can add nothing else; before this PR the hook was
unconditionally invoked, so an inherited-override miss would silently
drop the hook on the streaming path.
- Add unit tests for both behaviors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(mypy): narrow model_name to str in cost-injection branch
The hoisted cost_injection_active flag in chunk_processor encodes the
`bool(model_name)` requirement but mypy can't track that invariant
through the local, so the per-chunk `_process_chunk_with_cost_injection(
chunk, model_name)` calls flagged Optional[str] vs str. Pin a typed
non-None local inside the cost-injection branch so mypy narrows
correctly without changing runtime behavior.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Michael-RZ-Berri
|
3b2ce201d8
|
encrypt callback_vars in key/team metadata at rest (#27141)
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local> Co-authored-by: Yuneng Jiang <yuneng@berri.ai> |
||
|
milan-berri
|
7270f723de
|
fix(mcp): forward upstream initialize instructions on cold gateway init (#28231)
Some checks are pending
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
Prefetch upstream InitializeResult.instructions before merging gateway initialize options when YAML/DB do not set instructions, so clients receive upstream server text on the first MCP initialize without list_tools. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
f35e7eb2f6
|
feat(guardrails): add Microsoft Purview DLP guardrail (#24966)
* feat(guardrails): add Microsoft Purview DLP guardrail
* fix(guardrails/purview): raise_for_status on HTTP errors, cap scope cache, reuse executor
* fix(guardrails/purview): propagate litellm_call_id as correlation_id to Purview
* chore: fixes
* refactor(guardrails): delegate get_user_prompt to get_last_user_message
PurviewGuardrailBase duplicated AzureGuardrailBase (and OpenAIGuardrailBase)
user-prompt extraction. The same logic already lived in
common_utils.get_last_user_message; wire guardrail bases to that helper,
fix the helper docstring, and drop its redundant self-import of
convert_content_list_to_str.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): make protection scope cache true LRU on hits
OrderedDict.get() does not update insertion order; call move_to_end on
TTL-valid cache hits so popitem(last=False) evicts least-recently-used
users instead of FIFO by first insert.
Add a regression test with a small max cache size.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* Fix mypy
* fix(guardrails/purview): harden user-id resolution and broaden DLP text
Prefer API key and proxy-injected metadata over client metadata for Entra
identity. Scan full message transcript pre-call and all completion choices
post-call. Align logging-only hook with the same user-id rules.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails/purview): scan /v1/completions prompt and TextChoices
Normalize text-completion prompts (string or list of strings); skip token-id-only
prompts. Run post-call DLP on TextCompletionResponse choices. Extend logging_only
hook for text_completion. Add tests and completion_prompt_to_str helper.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(purview-dlp): return data after DLP pass; per-call executor; dedupe text extraction
async_pre_call_hook now returns the request dict after a successful check so
callers match skip-path behavior. logging_hook uses a fresh ThreadPoolExecutor
per invocation like Presidio to avoid single-worker starvation. Response text
extraction is centralized in _completion_response_text_parts.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): fix LRU cache refresh position and add Responses API scanning
Two fixes to the Microsoft Purview DLP guardrail:
1. LRU cache bug (base.py): When a stale scope cache entry was re-fetched,
the assignment updated the value but
Python's OrderedDict.__setitem__ preserves the original insertion order for
existing keys. This left the refreshed entry near the front of the dict,
making it the first candidate for LRU eviction via popitem(last=False).
Fix: call move_to_end(user_id) after every write to an existing key.
2. Responses API coverage gap (purview_dlp.py): Requests to /v1/responses use
an 'input' field instead of 'messages' or 'prompt', so the pre-call hook
returned without scanning the content. Similarly, post-call hook did not
handle ResponsesAPIResponse.output. Fix: add _responses_api_input_to_str()
helper and handle 'responses'/'aresponses' call types in async_pre_call_hook,
async_post_call_success_hook (via _completion_response_text_parts), and
async_logging_hook.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): message separator, non-blocking logging_hook, TextChoices type error
Three bugs fixed in the Microsoft Purview DLP guardrail:
1. get_prompt_text_for_dlp message separator (base.py)
- Previously called get_str_from_messages() which concatenated all message
texts with NO separator, so 'end of msg1' + 'start of msg2' became
'end of msg1start of msg2'.
- Now joins per-message text with '\n\n' via convert_content_list_to_str(),
preserving DLP pattern detection accuracy across message boundaries.
2. logging_hook blocking the event loop thread (purview_dlp.py)
- Previously called future.result() which blocked the calling thread
(often the event loop thread) for the entire round-trip of two sequential
Microsoft Graph API calls (_compute_protection_scopes + _process_content).
- Now fires and forgets: when called inside a running loop, schedules the
coroutine with loop.create_task(); otherwise spawns a daemon thread.
Returns (kwargs, result) immediately in both cases.
- Removes unused concurrent.futures.ThreadPoolExecutor import; adds threading.
3. Incompatible assignment type error (purview_dlp.py:180)
- mypy inferred 'choice' as TextChoices from the first loop body, then
flagged the assignment in the second loop as incompatible with Choices.
- Fixed by using distinct loop variable names: text_choice (TextChoices) and
chat_choice (Choices).
Tests: 7 new tests added covering the separator fix (TestGetPromptTextForDlp)
and the non-blocking logging_hook (TestLoggingHookNonBlocking).
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): suppress API errors in logging-only mode and scan tool-call arguments
Three issues fixed:
1. _check_content except block re-raised unconditionally even when
block_on_violation=False. The docstring promised 'log only - do not
raise' but network/API errors always propagated. Fixed by checking
block_on_violation before re-raising; when False, log a warning and
continue.
2. async_logging_hook used a single try/except wrapping both the prompt
and response audit calls. When the first _check_content (uploadText)
raised due to an API error the second call (downloadText) was silently
skipped. Fixed by giving each audit call its own try/except so both
always run independently.
3. convert_content_list_to_str() only reads message.content, so
tool_calls[].function.arguments and function_call.arguments were
invisible to the Purview pre-call and post-call scans. An authenticated
caller could embed sensitive text in tool-call arguments and bypass DLP.
Fixed by:
- Adding PurviewGuardrailBase._extract_tool_call_args_from_message()
which handles both dict and object-style messages, covering both
tool_calls[] arrays and the legacy function_call field.
- Updating get_prompt_text_for_dlp() to include those arguments
alongside message content (request/prompt path).
- Changing _completion_response_text_parts() from @staticmethod to an
instance method and adding tool-call argument extraction for
ModelResponse choices (response path).
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* chore(ui): restructure pre-built Next.js output to directory-based routing
Flat page files (e.g. guardrails.html) replaced by directory-based
index.html equivalents (e.g. guardrails/index.html) matching the
Next.js App Router output format.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* fix(purview): comprehensive security hardening — identity spoofing, streaming bypass, token-id gap
Four security issues addressed:
1. end_user_id kwargs fallback missing in _resolve_user_id_from_logging_kwargs
user_id already fell back to kwargs.get("user_api_key_user_id") when absent
from metadata, but end_user_id only checked md.get("user_api_key_end_user_id")
with no kwargs-level fallback. Added or kwargs.get("user_api_key_end_user_id").
2. Streaming responses bypassed post_call blocking
async_post_call_success_hook only runs on assembled non-streaming responses.
For streaming requests the proxy already delivered all content before the
hook ran, so raising HTTPException there had no effect. Added
async_post_call_streaming_iterator_hook which buffers the entire stream,
assembles it via stream_chunk_builder, runs the Purview DLP check, and only
then re-yields chunks via MockResponseIterator. If a violation is detected the
exception is raised before any bytes reach the client. The proxy automatically
skips async_post_call_success_hook for guardrails that define this method,
preventing duplicate scans.
3. Caller-controlled Purview user identity in blocking modes
When a LiteLLM API key has no bound user_id the guardrail fell back to
metadata[user_id_field], which is supplied by the caller. A caller could set
this to any Entra object ID whose Purview policies are more permissive and
bypass DLP. Added _resolve_trusted_user_id() that only returns identities
from the proxy auth system (user_api_key_dict.user_id, end_user_id, or
proxy-injected metadata["user_api_key_user_id"]). Added
_resolve_user_id_for_blocking() used by all blocking-mode hooks: tries
trusted sources first; if only caller-supplied is available, logs a
SECURITY WARNING and still proceeds (backward compat); if nothing resolves,
skips with a warning.
4. Token-id prompt DLP bypass
When /v1/completions received a pure token-id array prompt,
completion_prompt_to_str() returned None and the pre_call hook silently
skipped the Purview scan. An authenticated caller could tokenize blocked
text and send it without DLP evaluation. The hook now detects this case
(raw_prompt present but prompt_text None) and logs a WARNING while letting
the request pass through — token-id payloads are opaque at the text layer
and cannot be scanned. This makes the gap explicit rather than silent.
Tests: 94 total, all passing.
Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com>
* Revert "chore(ui): restructure pre-built Next.js output to directory-based routing"
This reverts commit
|
||
|
Mateo Wang
|
b0b25ae4b9
|
Include team alias in CLI JWT token (#28621) | ||
|
Sameer Kankute
|
50a3f10a92
|
feat(proxy): persist allowlisted OIDC claims in CLI SSO poll (#28463)
* feat(proxy): persist allowlisted OIDC claims in CLI SSO poll
Map CLI_SSO_CLAIM_MAP sources into user metadata and return scalar
attribution_metadata from /sso/cli/poll. Build SSOUserDefinedValues in
cli_sso_callback so first-time CLI logins can upsert users. Add mock OIDC
scripts and tests for claim extraction and poll exposure.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(proxy): document CLI SSO attribution_metadata in client README
Co-authored-by: Cursor <cursoragent@cursor.com>
* Delete scripts/mock_oidc_server_for_cli_sso.py
* Delete scripts/test_cli_sso_claims_e2e.py
* fix(ui_sso): preserve claim types and avoid metadata. prefix stripping
- Replace _update_dictionary with a local recursive merge so string
OIDC claim values that happen to look numeric are not silently coerced
to int/float when persisting CLI SSO attribution metadata.
- Use a local dot-path resolver in _extract_sso_claim_value so that
source claim paths beginning with 'metadata.' are not silently stripped
by get_nested_value (which is designed for LiteLLM JWT metadata, not
arbitrary OIDC claims).
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Remove redundant metadata. prefix strip in _set_nested_metadata_value
The _parse_cli_sso_claim_map already strips the metadata. prefix from
dest keys before reaching the setter. The duplicate strip in
_set_nested_metadata_value was a no-op in normal flow but could
mis-place values for dest keys like metadata.metadata.foo.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Fix greptile
* Fix ruff
* Move CLI SSO user defined values build inside try/except for consistent error handling
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(proxy): enforce restricted SSO group on CLI SSO callback
Apply verify_user_in_restricted_sso_group before CLI session completion
and user upsert, matching the UI SSO path. Re-raise ProxyException so
restricted-group denials return 403 instead of 500.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): replace recursive CLI SSO metadata helpers with iterative merge
Use stack-based flatten/merge to satisfy recursive_detector CI. Fix mypy
types for UserApiKeyCache and user_id on CLI SSO session completion.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: resolve nested CustomOpenID extra_fields in CLI SSO claim extraction
When GENERIC_USER_EXTRA_ATTRIBUTES captures a parent object (e.g. org_info),
extra_fields stores it as {"org_info": {"department": "..."}}. A CLI claim
map entry using a dotted path like org_info.department would silently fail
because the lookup only checked the exact flat key. Fall back to dotted-path
resolution on extra_fields before model_dump().
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(sso): update CLI SSO test for new received_response kwarg and remove redundant 'token' secret fragment
Co-authored-by: Yassin Kortam <yassin@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
Sameer Kankute
|
ef36e89638
|
feat(mcp): Add tool call and tool list support via UI for Oauth mcps (#28454)
* feat(mcp): cache OAuth token client-side so Tools tab loads without re-auth
After a user creates an OAuth MCP server and completes the authorization
flow, the resulting access token is now stored in sessionStorage keyed by
server_id. The MCP Tools tab reads this cached token and includes it as
an MCP auth header when listing and invoking tools, so the user never sees
an empty tool list. When the session ends (tab close / new browser) an
Authorize button re-triggers the flow without leaving the Tools screen.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix(ui/mcp): surface listMCPTools 401 errors so auth gate reappears
listMCPTools previously swallowed all errors (including HTTP 401) by
returning a synthetic { tools: [], error: 'network_error', ... } payload.
That made the useQuery retry-on-401 guard and mcpToolsError dead code,
so expired OAuth tokens never re-triggered the auth gate.
- Throw an enhanced Error with .status attached on non-2xx responses
(still preserves the legacy shape for true network failures so the
caller can render a generic message without crashing).
- Clear the cached OAuth session token when the tools query fails with
401, mirroring callMCPTool's onError handler so the Authorize button
is shown again.
- Surface mcpToolsError in the existing error banner.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp-tools): stable onSuccess + reuse parsed flow state
- Pass stable setOauthToken setter directly as onSuccess to avoid
recreating useToolsOAuthFlow's resumeOAuthFlow on every render.
- Reuse the already-parsed FLOW_STATE_KEY value (peeked) instead of
re-reading and re-parsing sessionStorage in resumeOAuthFlow.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui/mcp): restore listMCPTools never-throws contract
The previous fix made listMCPTools throw on HTTP errors while still
returning a synthetic object on network errors. This inconsistent
contract broke existing callers (MCPToolPermissions, MCPAppsPanel,
MCPConnectPicker) which inspect result.error / result.message and
expect the function to never throw.
- Return a normalized { tools: [], error, message, status, ... }
object on HTTP errors (instead of throwing) so all callers see a
consistent shape and the user-visible error text from
result.message is preserved.
- Convert the returned error object into a thrown Error inside the
one caller that needs it — the useQuery in mcp_tools.tsx — so the
401 retry/onError handlers still trigger and clear the cached
OAuth token.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix greptile
* fix(mcp): align OAuth header alias lookup with dashboard sanitization
Backend auth header resolution now matches x-mcp-{alias} keys produced by
the dashboard sanitizer, and the Tools tab re-syncs OAuth tokens when
serverId changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): widen auth header lookup types for list_tools
Accept legacy str | dict server auth maps and annotate list_tools
server_auth_header as Union[str, dict] for mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(ui): extract shared buildCallbackUrl/clearStorage for MCP OAuth hooks
Hoist the duplicate buildCallbackUrl and clearStorage helpers out of
useToolsOAuthFlow and useUserMcpOAuthFlow into a new shared module
src/hooks/mcpOAuthUtils.ts so the two hooks cannot drift if the URL
construction or storage cleanup logic needs to change.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui): don't gate M2M OAuth MCP servers behind interactive authorize
M2M (client_credentials) OAuth servers share auth_type="oauth2" with
interactive PKCE servers, but the backend fetches their token internally
and they typically lack a user authorization endpoint. Gating tool
listing on them rendered an Authorize button that would fail or redirect
incorrectly. Detect M2M via the presence of token_url (matching the
existing heuristic in mcp_server_edit.tsx) and skip the auth gate.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(ui/mcp): return error shape when listMCPTools JSON parse fails
Restore the never-throws contract when response.json() fails on a 2xx
body so callers do not receive null and crash on result.tools.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
Sameer Kankute
|
7a93cceb9f
|
Add error_description and hint for oauth flows (#28471)
* Add error_description and hint for oauth flows * Fix tests * fix(mcp-oauth): improve redirect_uri errors without leaking internal config Use NoReturn on _oauth_invalid_request, structured errors for BYOK loopback validation, and refactor validate_trusted_redirect_uri to satisfy PLR0915. Keep PROXY_BASE_URL and raw proxy_base_url in server logs only, not in the HTTP 400 body returned to unauthenticated callers. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp-oauth): stop leaking internal proxy origin in redirect_uri 400 body The trusted-redirect-uri rejection helper included the proxy's resolved scheme/host/port (e.g. http://litellm-internal:4000) in both the error_description and as a top-level proxy_origin field. Since the OAuth /authorize endpoint is unauthenticated, any caller could probe with a crafted redirect_uri and enumerate the internal network topology behind a reverse proxy. Keep full diagnostic detail in the server-side warning log (including the computed proxy base) but omit proxy-side values from the HTTP 400 body. Also drop the duplicated origin computation in _raise_trusted_redirect_uri_rejected now that those values are no longer needed by the response. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(mcp-oauth): remove dead userinfo check in redirect_uri validation The first check combined missing netloc with userinfo presence, making the second userinfo-only check unreachable. Split into two distinct checks so each error message reflects the actual failure mode. Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
harish-berri
|
d04373f4ce
|
Add granian as a ASGI compliant web server. Provider better throughput stability, (#26027)
Some checks are pending
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
* Add granian as a ASGI compliant web server. Provides better stability, 10-20 RPS improvement under standard LT conditions. TODO: Verify poetry lock details and add locust numbers to PR * Update granian version in license_cache.json and pyproject.toml to 2.5.7 * Enhance proxy CLI tests by adding SSL initialization checks for Granian server. Remove Python version skip conditions and implement tests to ensure SSL certificate and key are required for server initialization. * update uv lock to fix granian import error |
||
|
Yassin Kortam
|
10bd7406e0
|
feat: add guardrail violation span attributes and fix missing spans on pre-call blocks (#28364)
- Fix missing guardrail child spans when a pre-call guardrail blocks the request before reaching the LLM provider; `async_post_call_failure_hook` now calls `_emit_guardrail_spans_from_request_data` to emit spans from `request_data["metadata"]` regardless of whether `_handle_failure` already fired - Add `guardrail_status`, `guardrail_action`, and `guardrail_violation_categories` as queryable top-level OTEL span attributes so trace backends can filter/group by violation type without parsing the redacted `guardrail_response` blob - Introduce `_emit_guardrail_spans_from_request_data` helper that constructs minimal kwargs from `request_data["metadata"]` and routes through `_create_guardrail_span`, sharing the same dedupe state to prevent double-emitting when both failure hooks fire - Extend `BedrockGuardrail` with `_build_tracing_detail` and `_extract_violation_category_names` which flatten BLOCKED assessments into human-readable category labels (topic names, content-filter types, PII entity types, named regex names) before redaction, and surface Bedrock's raw `action` field via `tracing_detail` - Security: violation category extraction deliberately omits `customWords.match` and unnamed regex `match` values because those fields carry the user-submitted content that triggered the rule; only operator-defined `name`/`type` labels are emitted - Add `violation_categories` and `guardrail_action` fields to `StandardLoggingGuardrailInformation` and `GuardrailTracingDetail` TypedDicts to carry the pre-redaction metadata through the logging pipeline - Add comprehensive test suite covering: guardrail span creation on failure, dedupe between `_handle_failure` and `async_post_call_failure_hook`, per-span status attributes for multi-guardrail sequences, Bedrock category extraction for all policy types, security leak prevention, and end-to-end `CustomGuardrail` violation path Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |
||
|
mateo-berri
|
c6eafec396
|
Merge origin/litellm_internal_staging into litellm_feat/v1.84.0-mcp-gateway-jwt-auth | ||
|
mateo-berri
|
ca8e04c05e
|
fix(mcp): emit absolute resource_metadata URI in fabricated 401 challenge
Per RFC 9728 §3.2 the resource_metadata Bearer challenge must be an absolute URI; strict MCP clients reject relative URIs and fail to initiate discovery. MCPUpstreamAuthError.to_http_exception now accepts the gateway base URL and prepends it when the upstream omitted WWW-Authenticate, and all four call sites (streamable HTTP, SSE, and the two REST tool-list paths) supply it. |
||
|
mateo-berri
|
064f31e826
|
fix(mcp): fabricate resource_metadata challenge when upstream 401 omits WWW-Authenticate
When an upstream pass-through MCP server returns 401 without a WWW-Authenticate header (non-compliant per RFC 7235 §3.1), to_http_exception() now produces a synthetic Bearer challenge pointing at the gateway's standard-pattern oauth-protected-resource well-known endpoint for that server. This keeps MCP clients on the RFC 9728 discovery flow instead of receiving a bare 401 with no recovery hint. |
||
|
mateo-berri
|
1b0b75ccd7
|
fix(jwt/mcp): warn on unscoped JWT fallback; route agent permission lookup through shared helper
- _build_decode_kwargs no longer suppresses the unscoped-fallback warning when LiteLLM_JWTAuth.issuers is set: tokens whose iss does not match any configured issuer still fall through to the global path, and that fallback is itself unscoped when JWT_AUDIENCE/JWT_ISSUER are absent. - _get_agent_object_permission now caches the agent_id -> object_permission_id mapping and delegates the permission lookup to the shared get_object_permission helper, so the agent path reuses the same cache entries as the org / team / key paths. |
||
|
mateo-berri
|
f5a193f29b
|
fix(mcp/jwt): dedupe cold-start path parser; reject conflicting audience flags
- _parse_mcp_server_names_from_path now delegates to MCPRequestHandler._extract_target_server_names_from_path so the names used by the cold-start passthrough bypass cannot drift from the names used by downstream routing. - JWTIssuerConfig now rejects the combination of audience and disable_audience_validation=True at validation time instead of silently ignoring the flag. |
||
|
mateo-berri
|
ec3c67084f
|
security(mcp): strip Authorization in call_tool when LiteLLM admission used legacy header
Mirror the OAuth pass-through admission check from _prepare_mcp_server_headers (list-tools path) in _call_regular_mcp_tool (tool-call path): when the server is OAuth pass-through and the caller did not supply x-litellm-api-key, Authorization on the inbound request may itself be the LiteLLM API key — so strip it before forwarding instead of leaking the gateway credential upstream. When x-litellm-api-key is present, admission is unambiguous and Authorization continues to carry the upstream OAuth bearer (transparent pass-through). |
||
|
Cursor Agent
|
e9ff79c96a
|
fix(mcp): admit and forward Authorization for passthrough OAuth return
For pass-through MCP servers (auth_type=none with delegate_auth_to_upstream) the RFC 9728 cold-start flow sends the client back with only "Authorization: Bearer <upstream-token>" after upstream OAuth discovery. Previously this path 1) was rejected in process_mcp_request because the oauth2_headers fallback only covered auth_type=oauth2 targets, and 2) had the Authorization header stripped by _prepare_mcp_server_headers when no x-litellm-api-key was present, treating the upstream token as a potential LiteLLM key leak. - Extend the elif oauth2_headers fallback to also admit anonymously when every target is a pass-through server. - Pass user_api_key_auth into _prepare_mcp_server_headers so it can forward Authorization for pass-through servers when admission did not consume the bearer as a LiteLLM key (api_key is unset). Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
0fada28153
|
fix(jwt): validate issuer audience at config load + dedicated key-miss exception
- Move JWTIssuerConfig audience-required guard into a Pydantic model_validator so misconfiguration fails at startup instead of on the first request. - Replace the string-match `No matching public key found` filter in get_public_key's multi-URL fallback with a dedicated NoMatchingJWTPublicKeyError; only that specific exception triggers continuation, every other error still surfaces. |
||
|
mateo-berri
|
0511b76ada
|
Gate MCP OAuth pass-through on delegate_auth_to_upstream flag
Sameer's review on #28356/#28008 flagged that the new pass-through behaviors (preemptive 401 challenges, /.well-known/oauth-protected- resource proxying, upstream 401/403 propagation as MCPUpstreamAuthError, and Authorization-stripping when no x-litellm-api-key is supplied) were implicitly enabled for every server with auth_type=none plus Authorization in extra_headers. Existing users doing static bearer pass-through for non-OAuth reasons would have silently regressed. Make the detection rule explicit: extend the existing delegate_auth_to_upstream flag (previously oauth2-only) to also gate is_oauth_passthrough. Now requires flag + auth_type=None + Authorization in extra_headers, per Sameer's suggested detection rule. The UI toggle now appears for both modes (oauth2 PKCE passthrough and auth_type=none OAuth pass-through) with mode-appropriate copy. Update test fixtures to set the flag where the test intent is to exercise OAuth pass-through behavior, and add negative tests covering the new default-false case. |
||
|
yuneng-jiang
|
697a90ea77
|
fix: end user logs (#27758) (#28290)
Some checks are pending
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
* fix: end user logs
* fix(auth): address PR review feedback on end-user id validation
- Gate DB validation behind litellm.validate_end_user_id_in_db (default
False) so arbitrary client-supplied identifiers still pass through.
- Reuse get_end_user_object / get_user_object / _get_fuzzy_user_object
instead of issuing raw Prisma queries in the auth hot path.
- Consolidate: builder does the resolution once and stores it on the
auth obj; centralized checks reuse it, the outer user_api_key_auth
copy is removed.
- Preserve end_user_id when litellm.max_end_user_budget_id is set so
the default end-user budget can still apply to new customers.
* fix(auth): gate JSON-blob user-id rejection behind validate_end_user_id_in_db
Addresses PR review feedback: the JSON-encoded dict/list rejection in
_coerce_user_id_to_str was unconditionally applied, which would silently
stop tracking spend for deployments passing JSON-encoded user identifiers
on upgrade. Per the backwards-compatibility rule, default-path behavior
changes must be opt-in.
Now only strings that decode to a JSON object/array are dropped when
litellm.validate_end_user_id_in_db is True. Non-string dict/list/tuple
values are still always dropped, since stringifying them produces
unusable "{'device_id': ...}"-shaped spend-log rows.
* fix(auth): route email end-user lookup through get_user_object cache
The email-shaped end-user id branch called _get_fuzzy_user_object directly,
bypassing get_user_object's _should_check_db throttle and user_api_key_cache.
Every unique email would hit an unbudgeted raw Prisma query on the critical
auth path. Collapsing the two calls into one get_user_object invocation
with user_email=end_user_id routes through the cached helper per PR review
feedback.
* fix(auth): keep end-user safety net at user_api_key_auth tail
Krrish flagged that removing the tail-of-user_api_key_auth assignment
was a regression risk: ``_user_api_key_auth_builder`` has multiple
early-return paths (master_key=None, /user/auth, JWT short-circuits)
that bypass the end-user resolution block, so dropping the safety net
silently strips end-user attribution from those paths.
Restore the assignment but route it through resolve_and_validate_end_user_id
so the same validation rules apply. Skip the second pass when the builder
already set an id.
Adds two tests pinning the behaviour: one for the early-return safety
net and one verifying we don't double-resolve when the builder set the id.
Co-authored-by: Dennis Henry <dennis.henry@okta.com>
|
||
|
Cursor Agent
|
e2e4f703b7
|
Merge latest litellm_feat/v1.84.0-mcp-gateway-jwt-auth into local merge branch
# Conflicts: # tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py # tests/test_litellm/proxy/auth/test_handle_jwt.py Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Cursor Agent
|
418e266968
|
fix(tests): align merged JWT and MCP cold-start assertions
Update the tests carried over from PR #28008 to match the assertions on the staging branch: - tests/test_litellm/proxy/auth/test_handle_jwt.py: unknown issuers now fall back to the legacy JWT_PUBLIC_KEY_URL path (per litellm_feat/v1.84.0-mcp-gateway-jwt-auth's '\''fall back to global JWKS on unknown issuer'\''), and mapped issuer claims that are absent no longer fail closed — they simply leave the normalised LiteLLM internal claim absent. - tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py: the aggregate '\''/mcp'\'' route still triggers the delegate-auth-to-upstream lookup once for the header-supplied server name; cold-start admission must NOT fire on top of that. Tighten the assertion to assert_called_once_with so a future regression that re-enters cold-start is caught. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Cursor Agent
|
86cf3efd16
|
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_feat/v1.84.0-mcp-gateway-jwt-auth
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
7905e996bd
|
fix(mcp,jwt): drop unneeded async helper + suppress misleading unscoped JWT warning
- _build_oauth_authorization_server_response: revert to sync (no awaits in body). The function only does dict construction and synchronous registry lookups; async added coroutine creation overhead per discovery call without need. - _build_decode_kwargs: accept has_issuer_config so the global path's 'JWT auth is unscoped' warning is suppressed when LiteLLM_JWTAuth.issuers provides per-issuer scoping. Previously the warning fired spuriously for admins who intentionally use only the new issuers config. |
||
|
Cursor Agent
|
6697fdb03d
|
Merge PR #28008 (gym-cmd/litellm:feat/v1.84.0-mcp-gateway-jwt-auth) into litellm_feat/v1.84.0-mcp-gateway-jwt-auth
# Conflicts: # litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py # litellm/proxy/_experimental/mcp_server/server.py # litellm/proxy/management_endpoints/mcp_management_endpoints.py # litellm/proxy/proxy_server.py # litellm/types/mcp_server/mcp_server_manager.py # tests/test_litellm/interactions/test_openapi_compliance.py # tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py # tests/test_litellm/proxy/management_endpoints/test_mcp_management_endpoints.py Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
31f8c56cb7
|
fix(mcp,tests): assert cold-start helper directly for aggregate /mcp
Threading client_ip into _target_servers_delegate_auth_to_upstream made get_mcp_server_by_name(name, client_ip=...) also fire from the delegate-auth check, so the call_args_list assertion on client_ip-in-kwargs no longer uniquely signals a cold-start lookup. Patch _is_mcp_passthrough_cold_start and assert it is not invoked, which is the actual contract the test is pinning. |
||
|
yuneng-jiang
|
37ef8d9059
|
fix(proxy): hydrate wildcard discovery credentials (#28284) (#28419)
* fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> |
||
|
yuneng-jiang
|
f99fb5f27f
|
chore(ci): merge dev branch (#28314)
* chore(proxy): strict media-type match for form bodies (#27939) * chore(proxy): strict media-type match for form bodies ``_read_request_body`` and ``get_request_body`` routed on ``"form" in content_type`` / ``"multipart/form-data" in content_type``, which match any header containing the literal — ``application/form-json``, ``multiform/anything``, ``application/json; xform=1``. Starlette's ``request.form()`` returns an empty ``FormData`` for any non-canonical type without consuming the body, so the auth-time pre-read saw ``{}`` and skipped the banned-param check while the handler's later ``request.body()`` saw the original JSON payload. Parse the media type per RFC 7231 (substring before ``;``, trimmed, lowercased) and accept only ``application/x-www-form-urlencoded`` and ``multipart/form-data``. Replace both substring sites with the shared ``_is_form_content_type`` helper. Tests pin: case/whitespace/charset variants of the two real types match; ``application/form-json`` and similar substring-match traps fall through to the JSON parse path; real form POSTs continue to route through ``request.form()``. * chore(proxy): extract _is_json_content_type symmetric helper Mirror ``_is_form_content_type`` for the JSON branch of ``get_request_body`` so both classifications share the same media-type normalisation (strip params, trim, lowercase) and any future change to the parsing rules has one place to update. Adds tests for ``_is_json_content_type`` and for ``get_request_body`` covering the canonical JSON / form / unsupported / non-POST paths. * chore(proxy): surface form-parse failures instead of caching empty body Starlette's ``request.form()`` raises ``MultiPartException`` / ``ValueError`` / ``AssertionError`` on malformed multipart input (missing boundary, malformed chunk encoding, etc.). The outer ``except Exception: return {}`` swallowed every form-parse failure and cached an empty parsed body — auth-time pre-reads saw ``{}`` and skipped every banned-param check while a later raw-body re-read in the handler still saw the original payload. Same TOCTOU shape as the substring-match bypass: the auth gate and the handler don't agree on what the body is. Wrap ``request.form()`` in a narrow ``try`` that converts any parse failure to a 400 ``ProxyException``. The outer broad ``except`` is retained for unrelated unexpected errors but no longer covers form-parse-side bypass shapes. Adds a regression test parametrised over the exception classes Starlette can raise from ``request.form()``. * chore(proxy): drop redundant _is_json_content_type test class ``_is_json_content_type`` is a 3-line wrapper around the shared ``_normalize_media_type`` helper. Positive coverage lives in ``TestGetRequestBody.test_json_with_charset_param_parses_as_json``; negative coverage is covered transitively by ``TestIsFormContentType``'s non-form parametrize matrix (anything that isn't a form type falls through to the JSON branch). * chore(proxy): carry ASGI path into WebSocket auth synthetic Request (#27940) ``user_api_key_auth_websocket`` built a synthetic ``Request`` with a two-key scope (``type`` + ``headers``) and set ``request._url = websocket.url``. ``get_request_route`` reads ``scope.get("path", ...)`` and falls back to ``request.url.path`` only when ``path`` is absent. For the WebSocket flow that fallback fires and resolves to the Host-header-derived value (Starlette reconstructs ``websocket.url`` from the Host header), so a malformed Host collapses the resolved route and lets the auth gate compare against the wrong value. Carry the ASGI scope's ``path``, ``root_path``, and ``app_root_path`` into the synthetic scope so the lookup never reaches the fallback on the legitimate path. Regression test pins that the request handed to ``user_api_key_auth`` has ``scope["path"]`` equal to the ASGI scope's path. --------- Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> |
||
|
Sameer Kankute
|
988196911a
|
Litellm oss staging 1 (#28337)
* feat: add Xiaomi MiMo-V2.5-Pro and MiMo-V2.5 OpenRouter model entries (#27700) Squash-merged by litellm-agent from TorvaldUtne's PR. * fix(ui): trim whitespace from MCP inspector tool call inputs (#28203) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * gemini-3.1-flash-lite pricing (#27933) * feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers * fix pricing * add service tier --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> * fix: incorrect /v1/agents request example (#28131) * fix(anthropic): accept dict-shape reasoning_effort from Responses bridge (#28201) * fix(anthropic): accept dict-shape reasoning_effort from Responses bridge Issue #28196 — the Responses->Chat parser (transformation.py:184-200) keeps the full dict as reasoning_effort when summary is set; that branch was added in #25359. But the Anthropic transformation here still guarded on isinstance(value, str), silently dropping the param. Result: callers using the standard Reasoning(effort, summary) OpenAI-shaped object on Anthropic lose thinking entirely (0 reasoning_tokens, no thinking_blocks). Coerce dict -> string before mapping. Same shape tolerance that gpt_5_transformation._normalize_reasoning_effort_for_chat_completion already implements. summary is irrelevant for Anthropic's thinking_blocks. Adds two regression tests: one parametrized over string + dict shapes (with and without summary), one covering unparseable dict inputs (drops silently, no crash). * test(anthropic): add non-adaptive model coverage for dict-shape reasoning_effort Per Greptile feedback on PR #28198: the original regression test only exercised the adaptive (4.6+) path. Add a parametrized test for the non-adaptive branch (claude-sonnet-4-5) verifying that dict-shape reasoning_effort still maps to thinking.type='enabled' + budget_tokens, and that output_config is NOT set on pre-4.6 models. * test(anthropic): convert unparseable-dict test to @pytest.mark.parametrize Per @greptile-apps inline review on PR #28201 — matches the parametrize style of the two adjacent dict-shape tests and produces clearer failure messages (test ID per case instead of one collapsing for-loop). * feat: add pricing entry for openrouter/google/gemini-3.1-flash-lite (#28280) Squash-merged by litellm-agent from ro31337's PR. * fix(router): wrap aresponses streaming iterator for mid-stream fallbacks (#28215) Squash-merged by litellm-agent from cwang-otto's PR. * fix(router): unblock staging — mypy + coverage for aresponses streaming fallback (#28318) Squash-merged by litellm-agent from cwang-otto's PR. * fix(responses): forward timeout on completion transformation path (Anthropic, Bedrock, Vertex) (#28133) Squash-merged by litellm-agent from cwang-otto's PR. * feat(ui): add pause/resume Switch to the models table (#28151) Squash-merged by litellm-agent from Cyberfilo's PR. * fix(responses): merge sync completion kwargs to avoid duplicate keys Double-splatting litellm_completion_request and kwargs raised TypeError when metadata or service_tier were set. Match the async merge pattern. Co-authored-by: Cursor <cursoragent@cursor.com> * Use proxy base URL for CLI SSO form action (#28271) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which was missing from the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in litellm.completion_cost lookup. - Add mistral/ministral-8b-2512 entry to both the in-tree model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json (mirrors the existing openrouter/mistralai/ministral-8b-2512 pricing). - litellm.model_cost is loaded at import time from the URL pinned to main, so the new backup entry isn't visible at test runtime until it also lands on main. Backfill any entries missing from the remote-fetched map into litellm.model_cost in the local_testing conftest so cost-calculator lookups succeed on this branch. * fix(tests): drop unnecessary del of conftest backfill loop vars * fix(router): harden streaming fallback wrapper for bridge iterators - FallbackResponsesStreamWrapper now uses getattr fallbacks when copying attributes from the source iterator. The bridge path (LiteLLMCompletionStreamingIterator used by Anthropic/Bedrock/Vertex) does not call super().__init__ and is missing response, logging_obj (it uses litellm_logging_obj), responses_api_provider_config, start_time, request_data, call_type, and _hidden_params. Previously, wrapper construction raised AttributeError for any streaming fallback on the bridge path. - _aresponses_with_streaming_fallbacks now deep-copies the litellm_metadata (and metadata) dicts into fallback_kwargs. The primary attempt mutates this dict in place via _update_kwargs_with_deployment, so a shallow copy of kwargs was leaking primary-deployment fields (deployment, model_info, api_base) into the mid-stream fallback request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(router): use safe_deep_copy for fallback metadata snapshot The ban_copy_deepcopy_kwargs CI check rejects copy.deepcopy() on any variable whose name contains 'kwargs' (incl. fallback_kwargs). Swap the two copy.deepcopy(fallback_kwargs[...]) calls for safe_deep_copy, which handles non-picklable values (OTEL spans, etc.) by per-key deepcopy with fallback to the original reference. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(ci): skip chronically flaky build_and_test integration tests Both tests have been failing on every recent run of build_and_test against this PR's HEAD (1686967, 1688402, 1689993, 1690877), and the same two tests also fail intermittently on unrelated commits and other branches, independent of any code change in this PR (which only touches router fallback wrappers, the Anthropic Responses bridge, and unrelated UI/cost-map files). - tests.test_spend_logs.test_spend_logs: /spend/logs?request_id=... returns 500 even after a 20s wait for the spend log to be written. Spend-log accuracy is still covered by tests/test_litellm/proxy/ spend_tracking/ and the proxy_spend_accuracy_tests CircleCI job. - tests.test_team_members.test_add_multiple_members: /team/info?team_id= ... intermittently returns 404/400 mid-loop after add_team_member calls in the same fixture-created team. Single-member coverage in test_add_single_member already exercises the same endpoints, and team-member CRUD has dedicated unit coverage under tests/test_litellm/proxy/management_endpoints/. Skipping unblocks the build_and_test job until the underlying race in the dockerized integration setup is root-caused. * fix: preserve explicit timeout=0 in responses API handler Use 'timeout if timeout is not None else request_timeout' instead of 'timeout or request_timeout' so an explicit timeout=0/0.0 isn't silently replaced by the default request_timeout. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(ui): guard model_info access in pause Switch with optional chaining * fix(ui): guard model_info access in pause Switch onChange handler Mirror the optional-chaining guard already applied to the isPausing check so a config-model row with a missing model_info cannot throw when the toggle's onChange fires. --------- Co-authored-by: TorvaldUtne <78661304+TorvaldUtne@users.noreply.github.com> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com> Co-authored-by: Roman Pushkin <roman.pushkin@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: boarder7395 <37314943+boarder7395@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Yassin Kortam
|
2f9ac77b24
|
fix(proxy): expose Prisma idle/connect timeout + extra DB URL params (#28395)
* fix(proxy): expose Prisma idle/connect timeout + extra DB URL params Operators have reported large numbers of idle Prisma connections that never get closed. The proxy already forwards `connection_limit` and `pool_timeout` to the DATABASE_URL, but had no knob for capping idle or slow connections. Add three new `general_settings` keys that thread through to the DATABASE_URL / DIRECT_URL query string: - `database_connect_timeout` -> Prisma `connect_timeout` - `database_socket_timeout` -> Prisma `socket_timeout` (the main knob for closing idle connections from the LiteLLM side) - `database_extra_connection_params` -> untyped passthrough dict for any other Prisma URL param (`pgbouncer`, `statement_cache_size`, `sslmode`, ...); keys here override LiteLLM defaults. Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a single `_build_db_connection_url_params` helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update litellm/proxy/proxy_cli.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
gym-cmd
|
2636bbcdc7 | test(proxy): cover issuer-scoped JWT auth | ||
|
Sameer Kankute
|
718c4637a8
|
feat(mcp): allow native MCP OAuth support for cursor (#28327)
* feat(mcp): allow native MCP OAuth redirect URIs (cursor://) Discoverable OAuth /authorize rejected cursor:// callbacks because validate_trusted_redirect_uri only accepted http/https. Add an allowlisted native path with a built-in Cursor default and optional MCP_TRUSTED_NATIVE_REDIRECT_URIS env for other clients. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): address Greptile native redirect URI review Lowercase paths in normalizer so env allowlist entries match case- insensitively. Tighten wildcard prefix matching to reject sibling paths (e.g. callback-2) unless the prefix ends with /. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): reject query params on native OAuth redirect URIs Greptile: normalization stripped query strings before allowlist compare, so cursor://.../callback?injected=... could pass validation. Reject any native redirect_uri with a query component (same as fragments). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(model_cost_map): add mistral/ministral-8b-2512 entry Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which is not in the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in completion_cost lookup. Add the entry mirroring the existing openrouter/mistralai/ministral-8b-2512 pricing. * fix(mcp): lowercase default native redirect URIs Make _parse_trusted_native_redirect_uris apply the same lowercasing to built-in defaults as it does to env-var entries. * fix(tests): backfill local model_cost into remote-fetched map litellm.model_cost is loaded at import time from the URL pinned to main, so pricing entries that exist only in this branch (e.g. mistral/ministral-8b-2512, freshly added because Mistral now returns this id from mistral-tiny) are absent at test time and completion_cost lookups raise. Backfill the in-tree backup so cassette-driven cost calculations resolve against the entries that ship with the branch under test. Fixes the local_testing_part1 failures on test_completion_mistral_api and test_completion_mistral_api_modified_input. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> |
||
|
Sameer Kankute
|
68efe6970c
|
fix(mcp): JWT on tools/list and REST tools/call server resolution (#28227)
* fix(mcp): JWT on tools/list, REST server_id resolution, tool_server_mismatch
Sign outbound MCP JWTs for list_mcp_tools and inject headers on the tools/list
path. Resolve server_id on /mcp-rest/tools/call and return 403 tool_server_mismatch
when the tool does not belong to the requested server. Default missing arguments to {}.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): restrict list JWTs to mcp:tools/list and default REST arguments to {}
- List-only JWTs (call_type=list_mcp_tools) no longer carry the broad
mcp:tools/call scope. _build_scope() now emits only mcp:tools/list
when no tool name is provided, mirroring the existing least-privilege
rule that tool-call JWTs omit mcp:tools/list.
- REST /tools/call now defaults a missing 'arguments' field to {} so
execute_mcp_tool() and downstream **arguments / .keys() calls don't
receive None and crash with TypeError/AttributeError.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): validate tool/server in call_tool; skip JWT signer when not configured or static auth present
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): align tests and mypy with user_api_key_auth on tools/list
Update mocks for the new _get_tools_from_server parameter, mock server
registry in REST access-denied test, and narrow static_headers for mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): accept user_api_key_auth in get_tools_from_mcp_servers mock
The side_effect for the all-servers case did not accept the new kwarg,
so tools/list returned an empty list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): fail fast for unknown tools when server mapping exists
Server-name fallback in call_tool must not open an upstream session when
the tool is absent from a populated mapping. Update the HTTP transport test
to register a known tool before asserting not-found behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix mypy
* Fix mypy
* fix(mcp): preserve tools/call scope on missing tool name; pass user_api_key_auth in list_tools
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): match alias/server_name in _resolve_mcp_server_for_tool_call
The registry lookup in _resolve_mcp_server_for_tool_call previously only
compared candidate.name against the provided server_name, but tool name
prefixes can be derived from a server's alias or server_name (see
get_server_prefix). When the tool→server mapping is empty/stale (cold
start, dynamic tools), the lookup would fail for alias-configured
servers even though get_mcp_server_by_name (used by the REST path)
matches alias, server_name, and name.
Match the same priority of identifiers in both the registry pass and
the unprefixed fallback so the MCP protocol call_tool path is
consistent with the REST path.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): reuse proxy_logging DualCache in inject_mcp_jwt_headers_for_upstream
Instead of allocating a fresh DualCache() on every tools/list invocation,
prefer the shared proxy_logging_obj.internal_usage_cache.dual_cache when
available. The cache argument is currently unused by MCPJWTSigner, but
sharing the proxy's cache avoids per-call allocation overhead and matches
the cache identity used elsewhere in the proxy hook plumbing — so any
future per-request state stored in cache will survive across list calls.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(mcp): return 403 ip_filtering for IP-restricted servers in tools/call name lookup
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(test): accept user_api_key_auth kwarg in list_tools mocks
The proxy-infra job was failing on four TestMCPServerManager tests because
the mock_get_tools_from_server stubs did not accept the new
user_api_key_auth keyword argument that list_tools now forwards to
_get_tools_from_server. Add the kwarg to each stub so list_tools can call
through cleanly.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): skip JWT injection when per-user mcp_auth_header is set
MCPClient._get_auth_headers() applies extra_headers AFTER writing
Authorization from auth_value, so an injected JWT silently overwrites
the user's per-server OAuth token. Guard the JWT signer with
'not mcp_auth_header' so per-user OAuth (and any dict-form per-user
auth) takes precedence, mirroring the existing static_headers guard.
Adds a regression test that the signer's inject helper is not called
when mcp_auth_header is supplied.
* fix(mcp): skip JWT injection when extra_headers already has Authorization
When a server uses per-user OAuth tokens, the resolved token is passed
into _get_tools_from_server via extra_headers. The JWT injection guard
only checked mcp_auth_header and the server's static headers, so the
signer would silently overwrite the user's OAuth Authorization header.
Add a check for an existing Authorization entry in extra_headers so
caller-supplied per-user OAuth tokens take precedence over JWT signing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(mcp): cover JWT signer + tool-call resolution branches
Adds unit tests for the new MCPServerManager helpers (_resolve_mcp_server_for_tool_call,
_resolve_oauth2_headers_for_tool_call) and the new MCPJWTSigner paths
(_build_scope call_type branches and inject_mcp_jwt_headers_for_upstream).
Brings patch coverage above the auto target without changing behavior.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): retry tool-server lookup with prefixed name in REST mismatch check
When the REST /mcp-rest/tools/call path sends a raw tool name plus
requested_server_id, _get_mcp_server_from_tool_name(name) can return
None if the mapping only stores the prefixed form. That bypassed the
tool_server_mismatch 403 guard and let the call fall through to
trusting requested_server.
Retry the lookup with every known prefix of the requested server so
the mismatch check fires whenever the tool is actually registered.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): always reject unknown tools in server-name fallback
Defense-in-depth: _resolve_mcp_server_for_tool_call previously skipped
the unknown-tool check whenever the per-server mapping had no entries
yet (cold start, OAuth2 lazy listing, or upstream listing failure),
allowing arbitrary tool names to reach upstream servers.
Tighten the check so the server-name fallback always rejects tool
names not present in the mapping. Callers must call list_tools first
(standard MCP flow) before tools/call can resolve. Removes the
now-unused _mapping_has_tools_for_server helper and adds an
explicit empty-mapping rejection test alongside the existing
populated-mapping rejection test.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude (greptile subagent) <claude-greptile-bot@anthropic.com>
|
||
|
milan-berri
|
0fb710400f
|
fix(spend_counter): seed Redis counter via SET NX to prevent cross-pod double-seed (#27854)
* fix(spend_counter): seed Redis counter via SET NX to prevent cross-pod double-seed Symptom ------- Customers on multi-pod deployments see team `spend` jump to ~2x (or N x the pod count) shortly after a Redis cache miss / TTL expiry, triggering spurious "Budget Crossed" alerts and blocked requests until the value is manually reset. Root cause ---------- `SpendCounterReseed.coalesced` warmed the primary spend counter by calling `redis.async_increment(key, value=db_spend, refresh_ttl=True)`, which lowers to Redis `INCRBYFLOAT`. That is additive, not idempotent. The per-counter `asyncio.Lock` only coalesces seeders inside one process. With N pods sharing one Redis, on a cold key (cold start, TTL expiry, manual delete) every pod independently passes its lock + Redis re-check, reads the same `db_spend`, and issues `INCRBYFLOAT db_spend`. Final value: N x db_spend. Fix --- Use `redis.async_set_cache(key, value=db_spend, nx=True)` for the seed. SET NX is atomic across pods: exactly one writer initializes the key; losers read the winner's value via `async_get_cache`. This is the same idiom already used by `coalesced_window` in the same file, so the two seed paths are now consistent. Per-request deltas continue to use `INCRBYFLOAT` (correct - additive behaviour is what we want for increments, not for initial seed). Verification ------------ Live two-process repro against the same Postgres + Redis (DB spend = 506): Unpatched: 4/4 runs -> Redis counter = ~1012 (~2 x db_spend) Patched: 12/12 runs -> Redis counter = ~506 Unit tests (`test_proxy_server.py`): - New `test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed` patches `_get_lock` to return a fresh lock per caller (otherwise the per-process lock masks the race), races two `coalesced` calls, and asserts final = 506 with exactly one of two SET NX attempts winning. - 4 existing tests updated for the new seed contract (SET NX for the seed, INCRBYFLOAT only for the per-request delta). - Full `spend_counter or reseed or budget` slice: 22 passed. Co-authored-by: Cursor <cursoragent@cursor.com> * test(spend_counter): make SET NX mock atomic so loser branch is exercised Greptile flagged that `redis_set_cache` in test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed placed `await asyncio.sleep(0)` AFTER the NX membership check. Both concurrent tasks observed an empty `redis_store`, passed the guard, and both returned True - so the loser branch (else: read back winner's value) was never exercised. Fix the mock to model real atomic Redis SET NX: - Yield BEFORE the membership check so two concurrent callers interleave the way real SET NX does (first to resume runs check + write atomically and wins; second resumes after the key exists and loses). - Track set_cache return values; assert sorted([loser, winner]) so we know exactly one task wins and one loses. - Track async_get_cache calls that happen AFTER at least one SET NX has completed; assert at least one such read - that is the loser-path fallback (`current_value = float(cached)` when seeded is False). Verified by temporarily reverting the mock to the old order: the test now fails with `expected exactly one SET NX winner and one loser, got [True, True]`, exactly the failure mode Greptile described. No production code change. Co-authored-by: Cursor <cursoragent@cursor.com> * test(spend_counter): mock async_set_cache to populate redis_store in concurrent read+write test `test_concurrent_read_and_write_paths_share_one_db_query` mocks `async_increment` to populate the in-memory `redis_store`, but did not mock `async_set_cache`. After the SET-NX seed change in `coalesced()`, the seed step writes via `async_set_cache(nx=True)` (default AsyncMock, no `redis_store` write), so the simulated Redis stays empty after the first reseed. The second `get_current_spend` then sees a clean Redis miss, re-enters the DB read path, and the test fails with `expected 1 DB query, got 2`. Fix: add a `redis_set_cache` side_effect that updates `redis_store` on `nx=True` (and rejects when the key already exists), matching the pattern used by the four sibling tests fixed in this branch's first commit. Pre-existing assertions are unchanged. Full `tests/test_litellm/proxy/test_proxy_server.py`: 158 passed. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |