mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-25 01:02:15 +00:00
9816 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
1b0b75ccd7
|
fix(jwt/mcp): warn on unscoped JWT fallback; route agent permission lookup through shared helper
- _build_decode_kwargs no longer suppresses the unscoped-fallback warning when LiteLLM_JWTAuth.issuers is set: tokens whose iss does not match any configured issuer still fall through to the global path, and that fallback is itself unscoped when JWT_AUDIENCE/JWT_ISSUER are absent. - _get_agent_object_permission now caches the agent_id -> object_permission_id mapping and delegates the permission lookup to the shared get_object_permission helper, so the agent path reuses the same cache entries as the org / team / key paths. |
||
|
mateo-berri
|
f5a193f29b
|
fix(mcp/jwt): dedupe cold-start path parser; reject conflicting audience flags
- _parse_mcp_server_names_from_path now delegates to MCPRequestHandler._extract_target_server_names_from_path so the names used by the cold-start passthrough bypass cannot drift from the names used by downstream routing. - JWTIssuerConfig now rejects the combination of audience and disable_audience_validation=True at validation time instead of silently ignoring the flag. |
||
|
mateo-berri
|
ec3c67084f
|
security(mcp): strip Authorization in call_tool when LiteLLM admission used legacy header
Mirror the OAuth pass-through admission check from _prepare_mcp_server_headers (list-tools path) in _call_regular_mcp_tool (tool-call path): when the server is OAuth pass-through and the caller did not supply x-litellm-api-key, Authorization on the inbound request may itself be the LiteLLM API key — so strip it before forwarding instead of leaking the gateway credential upstream. When x-litellm-api-key is present, admission is unambiguous and Authorization continues to carry the upstream OAuth bearer (transparent pass-through). |
||
|
Cursor Agent
|
e9ff79c96a
|
fix(mcp): admit and forward Authorization for passthrough OAuth return
For pass-through MCP servers (auth_type=none with delegate_auth_to_upstream) the RFC 9728 cold-start flow sends the client back with only "Authorization: Bearer <upstream-token>" after upstream OAuth discovery. Previously this path 1) was rejected in process_mcp_request because the oauth2_headers fallback only covered auth_type=oauth2 targets, and 2) had the Authorization header stripped by _prepare_mcp_server_headers when no x-litellm-api-key was present, treating the upstream token as a potential LiteLLM key leak. - Extend the elif oauth2_headers fallback to also admit anonymously when every target is a pass-through server. - Pass user_api_key_auth into _prepare_mcp_server_headers so it can forward Authorization for pass-through servers when admission did not consume the bearer as a LiteLLM key (api_key is unset). Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
mateo-berri
|
0fada28153
|
fix(jwt): validate issuer audience at config load + dedicated key-miss exception
- Move JWTIssuerConfig audience-required guard into a Pydantic model_validator so misconfiguration fails at startup instead of on the first request. - Replace the string-match `No matching public key found` filter in get_public_key's multi-URL fallback with a dedicated NoMatchingJWTPublicKeyError; only that specific exception triggers continuation, every other error still surfaces. |
||
|
mateo-berri
|
0511b76ada
|
Gate MCP OAuth pass-through on delegate_auth_to_upstream flag
Sameer's review on #28356/#28008 flagged that the new pass-through behaviors (preemptive 401 challenges, /.well-known/oauth-protected- resource proxying, upstream 401/403 propagation as MCPUpstreamAuthError, and Authorization-stripping when no x-litellm-api-key is supplied) were implicitly enabled for every server with auth_type=none plus Authorization in extra_headers. Existing users doing static bearer pass-through for non-OAuth reasons would have silently regressed. Make the detection rule explicit: extend the existing delegate_auth_to_upstream flag (previously oauth2-only) to also gate is_oauth_passthrough. Now requires flag + auth_type=None + Authorization in extra_headers, per Sameer's suggested detection rule. The UI toggle now appears for both modes (oauth2 PKCE passthrough and auth_type=none OAuth pass-through) with mode-appropriate copy. Update test fixtures to set the flag where the test intent is to exercise OAuth pass-through behavior, and add negative tests covering the new default-false case. |
||
|
Cursor Agent
|
e2e4f703b7
|
Merge latest litellm_feat/v1.84.0-mcp-gateway-jwt-auth into local merge branch
# Conflicts: # tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py # tests/test_litellm/proxy/auth/test_handle_jwt.py Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Cursor Agent
|
418e266968
|
fix(tests): align merged JWT and MCP cold-start assertions
Update the tests carried over from PR #28008 to match the assertions on the staging branch: - tests/test_litellm/proxy/auth/test_handle_jwt.py: unknown issuers now fall back to the legacy JWT_PUBLIC_KEY_URL path (per litellm_feat/v1.84.0-mcp-gateway-jwt-auth's '\''fall back to global JWKS on unknown issuer'\''), and mapped issuer claims that are absent no longer fail closed — they simply leave the normalised LiteLLM internal claim absent. - tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py: the aggregate '\''/mcp'\'' route still triggers the delegate-auth-to-upstream lookup once for the header-supplied server name; cold-start admission must NOT fire on top of that. Tighten the assertion to assert_called_once_with so a future regression that re-enters cold-start is caught. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Cursor Agent
|
86cf3efd16
|
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_feat/v1.84.0-mcp-gateway-jwt-auth
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
7905e996bd
|
fix(mcp,jwt): drop unneeded async helper + suppress misleading unscoped JWT warning
- _build_oauth_authorization_server_response: revert to sync (no awaits in body). The function only does dict construction and synchronous registry lookups; async added coroutine creation overhead per discovery call without need. - _build_decode_kwargs: accept has_issuer_config so the global path's 'JWT auth is unscoped' warning is suppressed when LiteLLM_JWTAuth.issuers provides per-issuer scoping. Previously the warning fired spuriously for admins who intentionally use only the new issuers config. |
||
|
Cursor Agent
|
6697fdb03d
|
Merge PR #28008 (gym-cmd/litellm:feat/v1.84.0-mcp-gateway-jwt-auth) into litellm_feat/v1.84.0-mcp-gateway-jwt-auth
# Conflicts: # litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py # litellm/proxy/_experimental/mcp_server/server.py # litellm/proxy/management_endpoints/mcp_management_endpoints.py # litellm/proxy/proxy_server.py # litellm/types/mcp_server/mcp_server_manager.py # tests/test_litellm/interactions/test_openapi_compliance.py # tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py # tests/test_litellm/proxy/management_endpoints/test_mcp_management_endpoints.py Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
mateo-berri
|
31f8c56cb7
|
fix(mcp,tests): assert cold-start helper directly for aggregate /mcp
Threading client_ip into _target_servers_delegate_auth_to_upstream made get_mcp_server_by_name(name, client_ip=...) also fire from the delegate-auth check, so the call_args_list assertion on client_ip-in-kwargs no longer uniquely signals a cold-start lookup. Patch _is_mcp_passthrough_cold_start and assert it is not invoked, which is the actual contract the test is pinning. |
||
|
yuneng-jiang
|
37ef8d9059
|
fix(proxy): hydrate wildcard discovery credentials (#28284) (#28419)
* fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <dibyo@adobe.com> |
||
|
yuneng-jiang
|
79a5a7abad
|
feat(tests): behavior-pinning harness + Key Tier-1 matrix (#28321)
* test(proxy_behavior): scaffold session-scoped async ASGI client + liveness smoke Slice 2 of the management-endpoints behavior-pinning effort. New top-level dir tests/proxy_behavior/management/ outside every existing pytest glob. conftest.py initialises the proxy app once per session against the DATABASE_URL the harness boots Postgres at, wraps it in httpx.AsyncClient via in-process ASGITransport. The one smoke test asserts /health/liveliness returns 200, which exercises the full FastAPI middleware stack against a real app — no mocks. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): connect prisma via real lifespan; key/generate de-risk Slice 3 of the management-endpoints behavior-pinning effort. The fixture now enters the real FastAPI lifespan (proxy_startup_event) instead of just calling initialize() — that is where prisma_client is connected, password migration is kicked off, and the rest of the startup wiring runs. Tests pin the loop to the session scope so the AsyncClient created in the session fixture and the prisma connection opened in the lifespan share the same loop as the test bodies. New de-risk smoke: POST /key/generate with the master key returns 200, the returned sk- token resolves to a hashed row in LiteLLM_VerificationToken, and the cleartext token is never stored. Proves auth + handler + helper + prisma all wire together end-to-end against a real Postgres. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): seed 8-actor read-world for the authz matrix Slice 4 of the management-endpoints behavior-pinning effort. New ``actors.py`` defines the actor enum + seeds an immutable world (2 orgs, 2 teams, 8 users, 8 verification tokens) under the ``behavior-pin-`` prefix so the rows are identifiable in psql and ``_wipe_world`` is targeted. Each actor key is created with its cleartext form generated locally and its hashed form (via ``litellm.proxy.utils.hash_token``) stored in ``LiteLLM_VerificationToken`` — so the real ``user_api_key_auth`` accepts the cleartext bearer token. Roles, ``team_id``, ``organization_id``, and the service-account metadata flag are all set on the seeded rows so the auth layer resolves the same scopes a real proxy would. The session-scoped ``world`` fixture re-seeds at session start (idempotent via wipe-then-create), and the smoke test confirms each of the 8 actor keys can call ``/key/info`` on itself and receive its own row back. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): per-test scratch namespace + targeted delete_many teardown Slice 5 of the management-endpoints behavior-pinning effort. Adds the ``scratch`` function-scoped fixture: each test gets a uuid4-derived namespace prefix, tags writes with it (``key_alias``, ``team_alias``, ``user_id``, ``budget_id``), and the fixture teardown ``delete_many``-s any row whose namespace column starts with that prefix. Cleanup uses Prisma model methods only (no raw SQL, per CLAUDE.md) and orders deletes children-before-parents to avoid FK conflicts. The Slice 3 de-risk smoke is migrated onto the same fixture so it stops accumulating untagged tokens across repeated local runs. Smoke proves both halves of the contract: one test writes a scratch-tagged key and asserts it lands; a second test runs after the first's teardown and asserts no rows in the scratch namespace survived. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): codify G3 (strict-import grep) as a pytest item Slice 6 of the management-endpoints behavior-pinning effort. Two new tests walk every .py file under tests/proxy_behavior/ and assert: * no ``from litellm.proxy.management_endpoints`` import — the suite is deliberately constrained to the HTTP boundary so it survives handler refactors; * no ``mock``/``patch`` on ``user_api_key_auth`` — mocking auth is the structural failure mode of the existing 11k-line mock suite, and the point of this harness is that the real auth layer runs. Codifying G3 as a CI test removes the "did someone forget to check the PR-description checklist" failure mode. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * style(proxy_behavior): apply black to G3 grep test Follow-up to |
||
|
yuneng-jiang
|
e23d06dda4
|
test(realtime): expect session.created as xAI realtime initial event (#28424)
xAI's Grok Voice Agent API now sends session.created as its first realtime event (matching OpenAI), followed by conversation.created. The E2E canary pinned the old conversation.created value and failed. LiteLLM's xAI realtime path is a verbatim passthrough (provider_config is None, raw forwarding), so the event ordering is xAI's own — no transformation on our side. Update the pinned expected value and the now-stale comments to match the current API behavior. |
||
|
yuneng-jiang
|
f99fb5f27f
|
chore(ci): merge dev branch (#28314)
* chore(proxy): strict media-type match for form bodies (#27939) * chore(proxy): strict media-type match for form bodies ``_read_request_body`` and ``get_request_body`` routed on ``"form" in content_type`` / ``"multipart/form-data" in content_type``, which match any header containing the literal — ``application/form-json``, ``multiform/anything``, ``application/json; xform=1``. Starlette's ``request.form()`` returns an empty ``FormData`` for any non-canonical type without consuming the body, so the auth-time pre-read saw ``{}`` and skipped the banned-param check while the handler's later ``request.body()`` saw the original JSON payload. Parse the media type per RFC 7231 (substring before ``;``, trimmed, lowercased) and accept only ``application/x-www-form-urlencoded`` and ``multipart/form-data``. Replace both substring sites with the shared ``_is_form_content_type`` helper. Tests pin: case/whitespace/charset variants of the two real types match; ``application/form-json`` and similar substring-match traps fall through to the JSON parse path; real form POSTs continue to route through ``request.form()``. * chore(proxy): extract _is_json_content_type symmetric helper Mirror ``_is_form_content_type`` for the JSON branch of ``get_request_body`` so both classifications share the same media-type normalisation (strip params, trim, lowercase) and any future change to the parsing rules has one place to update. Adds tests for ``_is_json_content_type`` and for ``get_request_body`` covering the canonical JSON / form / unsupported / non-POST paths. * chore(proxy): surface form-parse failures instead of caching empty body Starlette's ``request.form()`` raises ``MultiPartException`` / ``ValueError`` / ``AssertionError`` on malformed multipart input (missing boundary, malformed chunk encoding, etc.). The outer ``except Exception: return {}`` swallowed every form-parse failure and cached an empty parsed body — auth-time pre-reads saw ``{}`` and skipped every banned-param check while a later raw-body re-read in the handler still saw the original payload. Same TOCTOU shape as the substring-match bypass: the auth gate and the handler don't agree on what the body is. Wrap ``request.form()`` in a narrow ``try`` that converts any parse failure to a 400 ``ProxyException``. The outer broad ``except`` is retained for unrelated unexpected errors but no longer covers form-parse-side bypass shapes. Adds a regression test parametrised over the exception classes Starlette can raise from ``request.form()``. * chore(proxy): drop redundant _is_json_content_type test class ``_is_json_content_type`` is a 3-line wrapper around the shared ``_normalize_media_type`` helper. Positive coverage lives in ``TestGetRequestBody.test_json_with_charset_param_parses_as_json``; negative coverage is covered transitively by ``TestIsFormContentType``'s non-form parametrize matrix (anything that isn't a form type falls through to the JSON branch). * chore(proxy): carry ASGI path into WebSocket auth synthetic Request (#27940) ``user_api_key_auth_websocket`` built a synthetic ``Request`` with a two-key scope (``type`` + ``headers``) and set ``request._url = websocket.url``. ``get_request_route`` reads ``scope.get("path", ...)`` and falls back to ``request.url.path`` only when ``path`` is absent. For the WebSocket flow that fallback fires and resolves to the Host-header-derived value (Starlette reconstructs ``websocket.url`` from the Host header), so a malformed Host collapses the resolved route and lets the auth gate compare against the wrong value. Carry the ASGI scope's ``path``, ``root_path``, and ``app_root_path`` into the synthetic scope so the lookup never reaches the fallback on the legitimate path. Regression test pins that the request handed to ``user_api_key_auth`` has ``scope["path"]`` equal to the ASGI scope's path. --------- Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> |
||
|
Yassin Kortam
|
35520adb4f
|
fix: serialize guardrail_response to JSON in OTEL traces (#28362)
* fix: serialize guardrail_response to JSON in OTEL traces Guardrail spans previously set the `guardrail_response` attribute via `safe_set_attribute`, which let dict payloads reach the OTEL exporter as Python repr strings. Downstream log pipelines could not parse those as JSON, breaking metric creation from guardrail traces. Serialize `guardrail_response` with `safe_dumps` before setting the attribute, matching how `masked_entity_count` is already handled. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: cover dict-serialization and None-skip for guardrail_response Address Greptile feedback on #28362 — add explicit coverage for the two behavioral guarantees of this fix: - Dict payloads (the OpenAI moderation case in the report) reach the span as a JSON string, not a Python repr. - ``None`` guardrail_response skips the attribute entirely, so no ``"null"`` leaks into traces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Sameer Kankute
|
988196911a
|
Litellm oss staging 1 (#28337)
* feat: add Xiaomi MiMo-V2.5-Pro and MiMo-V2.5 OpenRouter model entries (#27700) Squash-merged by litellm-agent from TorvaldUtne's PR. * fix(ui): trim whitespace from MCP inspector tool call inputs (#28203) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * gemini-3.1-flash-lite pricing (#27933) * feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers * fix pricing * add service tier --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> * fix: incorrect /v1/agents request example (#28131) * fix(anthropic): accept dict-shape reasoning_effort from Responses bridge (#28201) * fix(anthropic): accept dict-shape reasoning_effort from Responses bridge Issue #28196 — the Responses->Chat parser (transformation.py:184-200) keeps the full dict as reasoning_effort when summary is set; that branch was added in #25359. But the Anthropic transformation here still guarded on isinstance(value, str), silently dropping the param. Result: callers using the standard Reasoning(effort, summary) OpenAI-shaped object on Anthropic lose thinking entirely (0 reasoning_tokens, no thinking_blocks). Coerce dict -> string before mapping. Same shape tolerance that gpt_5_transformation._normalize_reasoning_effort_for_chat_completion already implements. summary is irrelevant for Anthropic's thinking_blocks. Adds two regression tests: one parametrized over string + dict shapes (with and without summary), one covering unparseable dict inputs (drops silently, no crash). * test(anthropic): add non-adaptive model coverage for dict-shape reasoning_effort Per Greptile feedback on PR #28198: the original regression test only exercised the adaptive (4.6+) path. Add a parametrized test for the non-adaptive branch (claude-sonnet-4-5) verifying that dict-shape reasoning_effort still maps to thinking.type='enabled' + budget_tokens, and that output_config is NOT set on pre-4.6 models. * test(anthropic): convert unparseable-dict test to @pytest.mark.parametrize Per @greptile-apps inline review on PR #28201 — matches the parametrize style of the two adjacent dict-shape tests and produces clearer failure messages (test ID per case instead of one collapsing for-loop). * feat: add pricing entry for openrouter/google/gemini-3.1-flash-lite (#28280) Squash-merged by litellm-agent from ro31337's PR. * fix(router): wrap aresponses streaming iterator for mid-stream fallbacks (#28215) Squash-merged by litellm-agent from cwang-otto's PR. * fix(router): unblock staging — mypy + coverage for aresponses streaming fallback (#28318) Squash-merged by litellm-agent from cwang-otto's PR. * fix(responses): forward timeout on completion transformation path (Anthropic, Bedrock, Vertex) (#28133) Squash-merged by litellm-agent from cwang-otto's PR. * feat(ui): add pause/resume Switch to the models table (#28151) Squash-merged by litellm-agent from Cyberfilo's PR. * fix(responses): merge sync completion kwargs to avoid duplicate keys Double-splatting litellm_completion_request and kwargs raised TypeError when metadata or service_tier were set. Match the async merge pattern. Co-authored-by: Cursor <cursoragent@cursor.com> * Use proxy base URL for CLI SSO form action (#28271) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which was missing from the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in litellm.completion_cost lookup. - Add mistral/ministral-8b-2512 entry to both the in-tree model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json (mirrors the existing openrouter/mistralai/ministral-8b-2512 pricing). - litellm.model_cost is loaded at import time from the URL pinned to main, so the new backup entry isn't visible at test runtime until it also lands on main. Backfill any entries missing from the remote-fetched map into litellm.model_cost in the local_testing conftest so cost-calculator lookups succeed on this branch. * fix(tests): drop unnecessary del of conftest backfill loop vars * fix(router): harden streaming fallback wrapper for bridge iterators - FallbackResponsesStreamWrapper now uses getattr fallbacks when copying attributes from the source iterator. The bridge path (LiteLLMCompletionStreamingIterator used by Anthropic/Bedrock/Vertex) does not call super().__init__ and is missing response, logging_obj (it uses litellm_logging_obj), responses_api_provider_config, start_time, request_data, call_type, and _hidden_params. Previously, wrapper construction raised AttributeError for any streaming fallback on the bridge path. - _aresponses_with_streaming_fallbacks now deep-copies the litellm_metadata (and metadata) dicts into fallback_kwargs. The primary attempt mutates this dict in place via _update_kwargs_with_deployment, so a shallow copy of kwargs was leaking primary-deployment fields (deployment, model_info, api_base) into the mid-stream fallback request. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(router): use safe_deep_copy for fallback metadata snapshot The ban_copy_deepcopy_kwargs CI check rejects copy.deepcopy() on any variable whose name contains 'kwargs' (incl. fallback_kwargs). Swap the two copy.deepcopy(fallback_kwargs[...]) calls for safe_deep_copy, which handles non-picklable values (OTEL spans, etc.) by per-key deepcopy with fallback to the original reference. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(ci): skip chronically flaky build_and_test integration tests Both tests have been failing on every recent run of build_and_test against this PR's HEAD (1686967, 1688402, 1689993, 1690877), and the same two tests also fail intermittently on unrelated commits and other branches, independent of any code change in this PR (which only touches router fallback wrappers, the Anthropic Responses bridge, and unrelated UI/cost-map files). - tests.test_spend_logs.test_spend_logs: /spend/logs?request_id=... returns 500 even after a 20s wait for the spend log to be written. Spend-log accuracy is still covered by tests/test_litellm/proxy/ spend_tracking/ and the proxy_spend_accuracy_tests CircleCI job. - tests.test_team_members.test_add_multiple_members: /team/info?team_id= ... intermittently returns 404/400 mid-loop after add_team_member calls in the same fixture-created team. Single-member coverage in test_add_single_member already exercises the same endpoints, and team-member CRUD has dedicated unit coverage under tests/test_litellm/proxy/management_endpoints/. Skipping unblocks the build_and_test job until the underlying race in the dockerized integration setup is root-caused. * fix: preserve explicit timeout=0 in responses API handler Use 'timeout if timeout is not None else request_timeout' instead of 'timeout or request_timeout' so an explicit timeout=0/0.0 isn't silently replaced by the default request_timeout. Co-authored-by: Yassin Kortam <yassin@berri.ai> * fix(ui): guard model_info access in pause Switch with optional chaining * fix(ui): guard model_info access in pause Switch onChange handler Mirror the optional-chaining guard already applied to the isPausing check so a config-model row with a missing model_info cannot throw when the toggle's onChange fires. --------- Co-authored-by: TorvaldUtne <78661304+TorvaldUtne@users.noreply.github.com> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Isha <72744901+IshaMeera@users.noreply.github.com> Co-authored-by: cwang-otto <chengxuan.wang@ottotheagent.com> Co-authored-by: Roman Pushkin <roman.pushkin@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: boarder7395 <37314943+boarder7395@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Yassin Kortam
|
2f9ac77b24
|
fix(proxy): expose Prisma idle/connect timeout + extra DB URL params (#28395)
* fix(proxy): expose Prisma idle/connect timeout + extra DB URL params Operators have reported large numbers of idle Prisma connections that never get closed. The proxy already forwards `connection_limit` and `pool_timeout` to the DATABASE_URL, but had no knob for capping idle or slow connections. Add three new `general_settings` keys that thread through to the DATABASE_URL / DIRECT_URL query string: - `database_connect_timeout` -> Prisma `connect_timeout` - `database_socket_timeout` -> Prisma `socket_timeout` (the main knob for closing idle connections from the LiteLLM side) - `database_extra_connection_params` -> untyped passthrough dict for any other Prisma URL param (`pgbouncer`, `statement_cache_size`, `sslmode`, ...); keys here override LiteLLM defaults. Refactors the duplicated DATABASE_URL/DIRECT_URL param dicts into a single `_build_db_connection_url_params` helper. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Update litellm/proxy/proxy_cli.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
8acf64e16c
|
fix(interactions): never drop streamed text deltas; always emit terminal completion (#28394)
* fix(interactions): never drop streamed text deltas; always emit terminal completion The interactions streaming bridge had two bugs flagged by Greptile on PR #28153: 1. The first OutputTextDeltaEvent (and the second, when no ResponseCreatedEvent precedes the deltas) was consumed to emit a synthetic interaction.created / step.start event, but the chunk's text payload was never forwarded as a step.delta. The text only reappeared in the terminal step.stop, which defeats the purpose of incremental streaming. 2. When the upstream Responses API stream ended via StopIteration without a ResponseCompletedEvent, the iterator emitted step.stop but never the terminal interaction.completed event carrying the full collected text. This refactors the iterator to translate each upstream chunk into a list of events (instead of a single event) and buffers them in a deque. A text delta now expands into [interaction.created, step.start, step.delta] on the first chunk so no token is dropped, and the StopIteration / StopAsyncIteration fallback always flushes a terminal interaction.completed event when one hasn't already been sent. Both behaviors are covered by new unit tests: - test_no_text_token_is_dropped_during_streaming - test_response_created_then_text_delta_emits_step_start_and_delta - test_stop_iteration_fallback_emits_completion_event - test_response_completed_emits_stop_then_completion (no double-emit) Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(interactions): correlate EOF terminal events with stream's interaction id The StopIteration fallback path previously built the terminal step.stop / interaction.completed events with id=None (legacy content.stop) and a memory-address fallback string (interaction.completed), neither of which matched the item_id used by the earlier interaction.created / step.start / step.delta events in the same stream. Downstream consumers correlating events by id would see a mismatch. Persist the interaction id derived from the first upstream chunk (item_id on an OutputTextDeltaEvent, or response.id on a ResponseCreatedEvent) and reuse it when flushing the terminal events on EOF. Author: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * ci(windows): raise UV_HTTP_TIMEOUT to 300s for uv sync The using_litellm_on_windows job has been hitting flaky PyPI download timeouts during 'uv sync --frozen --group dev' — different packages on each rerun (six, pydantic-core), all surfacing the same uv error: Failed to download distribution due to network timeout. Try increasing UV_HTTP_TIMEOUT (current value: 30s). uv's default 30s per-request timeout is too tight for the Windows runner on this project (50+ deps, several multi-MB wheels), so bump it to 300s to let slow individual downloads complete instead of failing the build. * fix(interactions): correlate ResponseCompletedEvent terminal events with stream's interaction id When a stream starts directly with OutputTextDeltaEvent (no preceding ResponseCreatedEvent), interaction.created carries item_id while interaction.completed previously carried response.id from ResponseCompletedEvent. The two ids can differ, leaving consumers that correlate events by id unable to match the start and completion events. Fall back to self._interaction_id (set on the first chunk that derives an id) before response.id, mirroring the EOF terminal path. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
gym-cmd
|
2636bbcdc7 | test(proxy): cover issuer-scoped JWT auth | ||
|
Sameer Kankute
|
718c4637a8
|
feat(mcp): allow native MCP OAuth support for cursor (#28327)
* feat(mcp): allow native MCP OAuth redirect URIs (cursor://) Discoverable OAuth /authorize rejected cursor:// callbacks because validate_trusted_redirect_uri only accepted http/https. Add an allowlisted native path with a built-in Cursor default and optional MCP_TRUSTED_NATIVE_REDIRECT_URIS env for other clients. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): address Greptile native redirect URI review Lowercase paths in normalizer so env allowlist entries match case- insensitively. Tighten wildcard prefix matching to reject sibling paths (e.g. callback-2) unless the prefix ends with /. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): reject query params on native OAuth redirect URIs Greptile: normalization stripped query strings before allowlist compare, so cursor://.../callback?injected=... could pass validation. Reject any native redirect_uri with a query component (same as fragments). Co-authored-by: Cursor <cursoragent@cursor.com> * fix(model_cost_map): add mistral/ministral-8b-2512 entry Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which is not in the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in completion_cost lookup. Add the entry mirroring the existing openrouter/mistralai/ministral-8b-2512 pricing. * fix(mcp): lowercase default native redirect URIs Make _parse_trusted_native_redirect_uris apply the same lowercasing to built-in defaults as it does to env-var entries. * fix(tests): backfill local model_cost into remote-fetched map litellm.model_cost is loaded at import time from the URL pinned to main, so pricing entries that exist only in this branch (e.g. mistral/ministral-8b-2512, freshly added because Mistral now returns this id from mistral-tiny) are absent at test time and completion_cost lookups raise. Backfill the in-tree backup so cassette-driven cost calculations resolve against the entries that ship with the branch under test. Fixes the local_testing_part1 failures on test_completion_mistral_api and test_completion_mistral_api_modified_input. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> |
||
|
Sameer Kankute
|
fecf212d70
|
fix(vertex_ai): omit function_call id on Vertex Gemini 3.5+ tool turns (#28324)
* fix(vertex_ai): omit function_call id on Vertex Gemini 3.5+ tool turns Vertex AI rejects `id` on function_call/function_response parts; only Google AI Studio accepts it for Gemini 3.5+ strict tool matching. Co-authored-by: Cursor <cursoragent@cursor.com> * Update litellm/llms/vertex_ai/gemini/vertex_and_google_ai_studio_gemini.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(vertex_ai): forward custom_llm_provider in context caching Pass custom_llm_provider through to _gemini_convert_messages_with_history in the context caching path so Gemini 3.5+ tool-call `id` forwarding behaves consistently between cached and non-cached completions on Google AI Studio. Co-authored-by: Claude <claude@anthropic.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <claude@anthropic.com> |
||
|
Sameer Kankute
|
f3a669fc5d
|
feat(interactions): migrate to Google Interactions API steps schema (May 2026) (#28153)
* feat(interactions): migrate to Google Interactions API steps schema (May 2026)
Default to Api-Revision: 2026-05-20 (new `steps` schema). Add
`litellm.use_legacy_interactions_schema` global flag that sends
Api-Revision: 2026-05-07 for operators who need the legacy `outputs`
schema until June 8, 2026.
- Inject Api-Revision header in GoogleAIStudioInteractionsConfig.validate_environment()
- Auto-coalesce response_mime_type → response_format and image_config migration on new schema
- Add steps field to InteractionsAPIResponse and InteractionsAPIStreamingResponse
- Add StepStart/StepDelta/StepStop/InteractionCreated/etc. SSE event types
- Update streaming completion detection to handle interaction.completed event
- Bridge transformer populates both outputs and steps fields
- Bridge streaming iterator emits new-schema events by default
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(interactions): address greptile review feedback
- Avoid mutating caller's generation_config dict by shallow-copying
before popping image_config, preventing silent failures on retries
- Skip schema key in response_format when response_format is None to
avoid sending schema: null to the Google Interactions API
- Remove delta field from step.stop events (new schema only); the
StepStop model has no delta field and sending it duplicates already-
streamed text and breaks spec-conformant clients
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): parse use_legacy_interactions_schema string values safely
bool("false") returns True in Python, so quoted YAML values like
"false" or "False" silently activated the legacy Interactions API
schema. Match the env-var parsing pattern in litellm/__init__.py by
treating string inputs as true only when they equal "true" (case
insensitive).
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(interactions): only set object/id/delta on step.stop for legacy schema
StepStop (new schema) has no object, id, or delta fields. Setting them
unconditionally caused spec-breaking extra fields on new-schema step.stop
events in all four construction sites (sync/async × main-loop/StopIteration).
Legacy content.stop still receives id, object, and delta unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(interactions): stabilize streaming bridge schema, dict aliasing, and lost first delta
- Capture use_legacy_interactions_schema once at iterator construction so
all events emitted by a single stream use a consistent schema, even if
the global flag is mutated mid-stream.
- Check for the buffered interaction.complete/completed event before the
finished check in __next__/__anext__ so the final completion event
(which carries the full collected text in steps) is not dropped after
self.finished is set.
- Copy text content entries before appending to both outputs and the
steps content list to avoid shared mutable dict aliasing between the
two response fields.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix tests
* fix greptile review
* fix(interactions): address Greptile P1 review on schema coalescing and legacy deltas
Skip response_mime_type merge when response_format is already a list, avoid
in-place list mutation on image_config append, and restore delta.type on
legacy content.delta events.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(interactions): black-format gemini transformation.py
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Sameer Kankute
|
68efe6970c
|
fix(mcp): JWT on tools/list and REST tools/call server resolution (#28227)
* fix(mcp): JWT on tools/list, REST server_id resolution, tool_server_mismatch
Sign outbound MCP JWTs for list_mcp_tools and inject headers on the tools/list
path. Resolve server_id on /mcp-rest/tools/call and return 403 tool_server_mismatch
when the tool does not belong to the requested server. Default missing arguments to {}.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): restrict list JWTs to mcp:tools/list and default REST arguments to {}
- List-only JWTs (call_type=list_mcp_tools) no longer carry the broad
mcp:tools/call scope. _build_scope() now emits only mcp:tools/list
when no tool name is provided, mirroring the existing least-privilege
rule that tool-call JWTs omit mcp:tools/list.
- REST /tools/call now defaults a missing 'arguments' field to {} so
execute_mcp_tool() and downstream **arguments / .keys() calls don't
receive None and crash with TypeError/AttributeError.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): validate tool/server in call_tool; skip JWT signer when not configured or static auth present
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): align tests and mypy with user_api_key_auth on tools/list
Update mocks for the new _get_tools_from_server parameter, mock server
registry in REST access-denied test, and narrow static_headers for mypy.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): accept user_api_key_auth in get_tools_from_mcp_servers mock
The side_effect for the all-servers case did not accept the new kwarg,
so tools/list returned an empty list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): fail fast for unknown tools when server mapping exists
Server-name fallback in call_tool must not open an upstream session when
the tool is absent from a populated mapping. Update the HTTP transport test
to register a known tool before asserting not-found behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix mypy
* Fix mypy
* fix(mcp): preserve tools/call scope on missing tool name; pass user_api_key_auth in list_tools
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): match alias/server_name in _resolve_mcp_server_for_tool_call
The registry lookup in _resolve_mcp_server_for_tool_call previously only
compared candidate.name against the provided server_name, but tool name
prefixes can be derived from a server's alias or server_name (see
get_server_prefix). When the tool→server mapping is empty/stale (cold
start, dynamic tools), the lookup would fail for alias-configured
servers even though get_mcp_server_by_name (used by the REST path)
matches alias, server_name, and name.
Match the same priority of identifiers in both the registry pass and
the unprefixed fallback so the MCP protocol call_tool path is
consistent with the REST path.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): reuse proxy_logging DualCache in inject_mcp_jwt_headers_for_upstream
Instead of allocating a fresh DualCache() on every tools/list invocation,
prefer the shared proxy_logging_obj.internal_usage_cache.dual_cache when
available. The cache argument is currently unused by MCPJWTSigner, but
sharing the proxy's cache avoids per-call allocation overhead and matches
the cache identity used elsewhere in the proxy hook plumbing — so any
future per-request state stored in cache will survive across list calls.
Co-authored-by: Claude <noreply@anthropic.com>
* fix(mcp): return 403 ip_filtering for IP-restricted servers in tools/call name lookup
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(test): accept user_api_key_auth kwarg in list_tools mocks
The proxy-infra job was failing on four TestMCPServerManager tests because
the mock_get_tools_from_server stubs did not accept the new
user_api_key_auth keyword argument that list_tools now forwards to
_get_tools_from_server. Add the kwarg to each stub so list_tools can call
through cleanly.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): skip JWT injection when per-user mcp_auth_header is set
MCPClient._get_auth_headers() applies extra_headers AFTER writing
Authorization from auth_value, so an injected JWT silently overwrites
the user's per-server OAuth token. Guard the JWT signer with
'not mcp_auth_header' so per-user OAuth (and any dict-form per-user
auth) takes precedence, mirroring the existing static_headers guard.
Adds a regression test that the signer's inject helper is not called
when mcp_auth_header is supplied.
* fix(mcp): skip JWT injection when extra_headers already has Authorization
When a server uses per-user OAuth tokens, the resolved token is passed
into _get_tools_from_server via extra_headers. The JWT injection guard
only checked mcp_auth_header and the server's static headers, so the
signer would silently overwrite the user's OAuth Authorization header.
Add a check for an existing Authorization entry in extra_headers so
caller-supplied per-user OAuth tokens take precedence over JWT signing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(mcp): cover JWT signer + tool-call resolution branches
Adds unit tests for the new MCPServerManager helpers (_resolve_mcp_server_for_tool_call,
_resolve_oauth2_headers_for_tool_call) and the new MCPJWTSigner paths
(_build_scope call_type branches and inject_mcp_jwt_headers_for_upstream).
Brings patch coverage above the auto target without changing behavior.
Co-authored-by: Claude <claude@anthropic.com>
* fix(mcp): retry tool-server lookup with prefixed name in REST mismatch check
When the REST /mcp-rest/tools/call path sends a raw tool name plus
requested_server_id, _get_mcp_server_from_tool_name(name) can return
None if the mapping only stores the prefixed form. That bypassed the
tool_server_mismatch 403 guard and let the call fall through to
trusting requested_server.
Retry the lookup with every known prefix of the requested server so
the mismatch check fires whenever the tool is actually registered.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(mcp): always reject unknown tools in server-name fallback
Defense-in-depth: _resolve_mcp_server_for_tool_call previously skipped
the unknown-tool check whenever the per-server mapping had no entries
yet (cold start, OAuth2 lazy listing, or upstream listing failure),
allowing arbitrary tool names to reach upstream servers.
Tighten the check so the server-name fallback always rejects tool
names not present in the mapping. Callers must call list_tools first
(standard MCP flow) before tools/call can resolve. Removes the
now-unused _mapping_has_tools_for_server helper and adds an
explicit empty-mapping rejection test alongside the existing
populated-mapping rejection test.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude (greptile subagent) <claude-greptile-bot@anthropic.com>
|
||
|
Sameer Kankute
|
7f563b2593
|
fix(router): use forwarded model_id for native Azure container IDs (#27921)
* fix(router): use forwarded model_id for native Azure container IDs in _init_containers_api_endpoints
Azure code-interpreter containers return provider-native IDs (cntr_ + hex)
that carry no LiteLLM routing payload, so _decode_container_id returns
model_id=None. The router was falling through to call the handler directly,
bypassing _ageneric_api_call_with_fallbacks and leaving api_base=None for
Azure deployments. Fall back to the model_id forwarded from the proxy
ownership check so deployment credentials are always applied.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): strip /openai/responses path from api_base in AzureContainerConfig.get_complete_url
When a deployment's api_base is the responses endpoint URL
(e.g. .../openai/responses?api-version=...), AzureContainerConfig was
appending /openai/containers on top of it, producing the broken path
.../openai/responses/openai/containers. Azure returns 404 for that URL
while the correct path is .../openai/containers.
Strip any /openai/responses suffix from api_base before constructing
the containers URL so the resource root is always used as the starting point.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): prefer api-version from api_base URL over deployment's api_version
The deployment's api_version (e.g. 2024-08-01-preview) targets the chat/responses
API and is too old for the containers API, which requires 2025-04-01-preview.
The responses endpoint api_base already carries the correct api-version in its
query string. Extract it and use it for the containers URL, overriding the
stale deployment-level version.
Fixes DELETE and file-upload operations returning 404 due to wrong api-version.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(containers): pass params=None instead of params={} to httpx to preserve api-version
httpx erases a URL's query-string when params={} (empty dict) is passed,
silently stripping ?api-version=2025-04-01-preview from every container
POST/DELETE request. Azure's GET endpoints tolerate a missing api-version;
POST (upload) and DELETE are strict, so those returned 404.
Fix: use `params or None` in container_handler._async_handle and
llm_http_handler.async_container_delete_handler (and all sibling container
handlers) so that an empty params dict falls back to None, leaving httpx to
preserve the URL's existing query string intact.
Adds a regression test that directly documents the httpx behaviour.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): remove elif model_id branch from _init_containers_api_endpoints
Two reviewer findings addressed:
1. Truncated comment on the model_id fallback line — now complete.
2. Security: the elif branch that fired when container_id was absent allowed
any authenticated caller to supply model_id in a POST /v1/containers body
and route the request through an arbitrary deployment UUID, bypassing the
model-level access checks that only validate `model`. Removed the elif
branch; operations without container_id (create, list) route by the
caller-supplied `model` field as before. model_id forwarding is kept only
inside the container_id block, where the proxy ownership check has already
validated the container before forwarding the deployment ID.
Adds a regression test pinning the security boundary: no-container-id path
calls original_function directly even when model_id is in kwargs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(containers): validate proxy-to-router model_id forwarding for managed IDs
Add test_regression_get_container_forwarding_params_sets_model_id_for_managed_id
to verify that get_container_forwarding_params (the proxy-side half of the Azure
routing fix) correctly extracts and forwards model_id from a LiteLLM-managed
encoded container ID.
This closes the gap identified by Greptile P1: the previous regression test
only injected model_id as a direct kwarg, validating the router in isolation.
The new test exercises the actual proxy-to-router data flow through
ownership.get_container_forwarding_params, confirming that kwargs["model_id"]
is populated before _init_containers_api_endpoints is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(azure-containers): tighten endpoint-path strip to endswith match
Use path.endswith() instead of path.find() for _AZURE_ENDPOINT_PATHS so
the suffix strip only fires when api_base actually ends with one of the
endpoint-specific path suffixes. This is the more precise check greptile
flagged on the original find()-based implementation.
* Fix sync container handler to preserve URL query string
Mirror the async path fix: pass None instead of an empty params dict so
httpx does not strip the URL's existing query string (e.g.
?api-version=...), which is required for Azure container routing.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(azure-containers): strip trailing slash before endpoint suffix match
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(containers): recover model_id from stored encoded id for native Azure container IDs
get_container_forwarding_params previously only set model_id when the
user-supplied container_id was a LiteLLM-managed encoded id. For native
upstream IDs (e.g. Azure 'cntr_<hex>') the decode fails and model_id was
never forwarded — making the router-side fallback in
_init_containers_api_endpoints unreachable in production.
Fall back to the stored 'unified_object_id' on the ownership row, which
is the encoded form captured at create time when the router selected a
specific deployment. Decoding that yields the deployment model_id and
restores router-based credential application (api_base, api_key) for
retrieve/delete and container-file operations on native IDs.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
Sameer Kankute
|
183092d797
|
fix(proxy): normalize batch file IDs before ManagedObjectTable write (#28339)
* fix(proxy): normalize batch file IDs before ManagedObjectTable write Run post_call_success_hook before update_batch_in_database on retrieve/cancel, and ensure_batch_response_managed_file_ids so file_object never stores raw provider output_file_id or error_file_id. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): address Greptile review on batch file ID normalization Remove redundant resolve_* calls after update_batch_in_database and rename loop variable to avoid shadowing hidden_params unified_file_id. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): add mistral/ministral-8b-2512 to cost map and backfill in conftest Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which was missing from the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in litellm.completion_cost lookup. - Add mistral/ministral-8b-2512 entry to both the in-tree model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json (mirrors the existing openrouter/mistralai/ministral-8b-2512 pricing). - litellm.model_cost is loaded at import time from the URL pinned to main, so the new backup entry isn't visible at test runtime until it also lands on main. Backfill any entries missing from the remote-fetched map into litellm.model_cost in the local_testing conftest so cost-calculator lookups succeed on this branch. * fix(tests): drop unnecessary del of conftest backfill loop vars * fix: resolve batch response file IDs even when status unchanged The status-unchanged early return in update_batch_in_database was skipping ensure_batch_response_managed_file_ids, leaving raw provider input_file_id (and other raw IDs) in the user-facing response when polling an in-progress batch. Move the in-place file ID normalization above the early return so the response always carries unified managed IDs while still skipping the DB write when nothing changed. Co-authored-by: Yassin Kortam <yassin@berri.ai> * test(batches): cover ensure_batch_response_managed_file_ids branches Add tests for the previously-uncovered paths in ensure_batch_response_managed_file_ids: error_file_id normalization, swallowed conversion errors, UserAPIKeyAuth fallback from db_batch_object, model_name resolution from unified_file_id, and early returns when managed_files_obj, model_id, or auth context are missing. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <claude@anthropic.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
milan-berri
|
0fb710400f
|
fix(spend_counter): seed Redis counter via SET NX to prevent cross-pod double-seed (#27854)
* fix(spend_counter): seed Redis counter via SET NX to prevent cross-pod double-seed Symptom ------- Customers on multi-pod deployments see team `spend` jump to ~2x (or N x the pod count) shortly after a Redis cache miss / TTL expiry, triggering spurious "Budget Crossed" alerts and blocked requests until the value is manually reset. Root cause ---------- `SpendCounterReseed.coalesced` warmed the primary spend counter by calling `redis.async_increment(key, value=db_spend, refresh_ttl=True)`, which lowers to Redis `INCRBYFLOAT`. That is additive, not idempotent. The per-counter `asyncio.Lock` only coalesces seeders inside one process. With N pods sharing one Redis, on a cold key (cold start, TTL expiry, manual delete) every pod independently passes its lock + Redis re-check, reads the same `db_spend`, and issues `INCRBYFLOAT db_spend`. Final value: N x db_spend. Fix --- Use `redis.async_set_cache(key, value=db_spend, nx=True)` for the seed. SET NX is atomic across pods: exactly one writer initializes the key; losers read the winner's value via `async_get_cache`. This is the same idiom already used by `coalesced_window` in the same file, so the two seed paths are now consistent. Per-request deltas continue to use `INCRBYFLOAT` (correct - additive behaviour is what we want for increments, not for initial seed). Verification ------------ Live two-process repro against the same Postgres + Redis (DB spend = 506): Unpatched: 4/4 runs -> Redis counter = ~1012 (~2 x db_spend) Patched: 12/12 runs -> Redis counter = ~506 Unit tests (`test_proxy_server.py`): - New `test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed` patches `_get_lock` to return a fresh lock per caller (otherwise the per-process lock masks the race), races two `coalesced` calls, and asserts final = 506 with exactly one of two SET NX attempts winning. - 4 existing tests updated for the new seed contract (SET NX for the seed, INCRBYFLOAT only for the per-request delta). - Full `spend_counter or reseed or budget` slice: 22 passed. Co-authored-by: Cursor <cursoragent@cursor.com> * test(spend_counter): make SET NX mock atomic so loser branch is exercised Greptile flagged that `redis_set_cache` in test_primary_spend_counter_redis_concurrent_seed_does_not_double_seed placed `await asyncio.sleep(0)` AFTER the NX membership check. Both concurrent tasks observed an empty `redis_store`, passed the guard, and both returned True - so the loser branch (else: read back winner's value) was never exercised. Fix the mock to model real atomic Redis SET NX: - Yield BEFORE the membership check so two concurrent callers interleave the way real SET NX does (first to resume runs check + write atomically and wins; second resumes after the key exists and loses). - Track set_cache return values; assert sorted([loser, winner]) so we know exactly one task wins and one loses. - Track async_get_cache calls that happen AFTER at least one SET NX has completed; assert at least one such read - that is the loser-path fallback (`current_value = float(cached)` when seeded is False). Verified by temporarily reverting the mock to the old order: the test now fails with `expected exactly one SET NX winner and one loser, got [True, True]`, exactly the failure mode Greptile described. No production code change. Co-authored-by: Cursor <cursoragent@cursor.com> * test(spend_counter): mock async_set_cache to populate redis_store in concurrent read+write test `test_concurrent_read_and_write_paths_share_one_db_query` mocks `async_increment` to populate the in-memory `redis_store`, but did not mock `async_set_cache`. After the SET-NX seed change in `coalesced()`, the seed step writes via `async_set_cache(nx=True)` (default AsyncMock, no `redis_store` write), so the simulated Redis stays empty after the first reseed. The second `get_current_spend` then sees a clean Redis miss, re-enters the DB read path, and the test fails with `expected 1 DB query, got 2`. Fix: add a `redis_set_cache` side_effect that updates `redis_store` on `nx=True` (and rejects when the key already exists), matching the pattern used by the four sibling tests fixed in this branch's first commit. Pre-existing assertions are unchanged. Full `tests/test_litellm/proxy/test_proxy_server.py`: 158 passed. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Cursor Agent
|
f0eb54ea9f
|
fix(mcp): correct passthrough probe 401 + slashed-name cold start parser
- _check_passthrough_upstream_auth now emits 'Bearer resource_metadata="..."' pointing at the gateway's oauth-protected-resource well-known URL, mirroring the pre-emptive 401 path. Pass-through servers don't use the gateway as an authorization server, so the previous 'authorization_uri=' challenge sent clients to the wrong metadata endpoint. - _parse_mcp_server_names_from_path now accepts server names that contain a single slash (e.g. custom_solutions/user_123), mirroring MCPRequestHandler._extract_target_server_names_from_path. Without this, the cold-start bypass missed slashed-name servers and the generic admission error propagated instead of the spec-compliant 401 challenge. - _is_mcp_passthrough_cold_start drops the unused scope parameter from its signature. Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Claude
|
ee38ba16e3
|
fix(mcp,tests): sync stubs and cold-start assertions with delegate-check
The merge of base-branch _target_servers_delegate_auth_to_upstream into process_mcp_request inserts an additional get_mcp_server_by_name(name) lookup ahead of the cold-start path, which breaks two test patterns: 1. lookup_by_name(name) side-effect stubs in TestMCPDelegateAuthToUpstream are called positionally by the delegate check, then again by the cold-start path with client_ip=... — raising TypeError: unexpected keyword argument 'client_ip'. Accept **_kwargs to match the real signature. 2. TestMCPPassthroughColdStartAdmission assertions count the lookup exactly once with client_ip=..., but the delegate check now adds a positional-only call ahead of it. Switch assert_called_once_with to assert_any_call for the cold-start invocation, and assert client_ip was *not* passed for the aggregate /mcp test where cold-start must not fire. Both updates align with CLAUDE.md guidance to keep monkeypatch stubs in sync with the real signature when an optional parameter is added. Co-authored-by: Claude <claude@anthropic.com> |
||
|
Claude
|
326c6bb84f
|
fix(mcp): add upstream auth pre-flight in SSE handler
Mirror handle_streamable_http_mcp by calling _check_passthrough_upstream_auth after the cold-start 401 emitter so expired/invalid upstream tokens surface a proper 401 + WWW-Authenticate challenge before the SSE session commits 200 headers, instead of letting list_tools silently return [] when the upstream rejects the token. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Claude
|
6f82537c35
|
Merge branch 'litellm_internal_staging' into litellm_feat/v1.84.0-mcp-gateway-jwt-auth
Resolved conflict in tests/local_testing/conftest.py by keeping the branch-specific comment that describes the ministral-8b-2512 backfill case (this branch adds that entry). Co-authored-by: Claude <claude@anthropic.com> |
||
|
Sameer Kankute
|
99a63d5180
|
feat(gemini): add gemini-3.1-flash-lite model cost map (#28320)
* feat(gemini): add gemini-3.1-flash-lite model cost map entries Co-authored-by: Cursor <cursoragent@cursor.com> * Update model_prices_and_context_window.json * Update source URL for model pricing information * Sync source URL for gemini-3.1-flash-lite in backup JSON * fix(model_cost_map): add mistral/ministral-8b-2512 entry Mistral rotated the 'mistral/mistral-tiny' alias to return 'ministral-8b-2512' as the response model, which is not in the cost map. This caused test_completion_mistral_api and test_completion_mistral_api_modified_input to fail in completion_cost lookup. Add the entry mirroring the existing openrouter/mistralai/ministral-8b-2512 pricing. * test(cost_calculator): assert output_cost_per_reasoning_token for gemini-3.1-flash-lite * fix(tests): backfill local backup entries into runtime model_cost litellm.model_cost is loaded from LITELLM_MODEL_COST_MAP_URL (pinned to main) at import time, so any pricing entries added to the in-tree backup on this branch aren't visible at test runtime until they also land on main. The Mistral cassette currently returns model=ministral-8b-2512 and the cost-calculator lookup in test_completion_mistral_api / test_completion_mistral_api_modified_input fails despite the entry existing in the local backup. Backfill missing backup entries into litellm.model_cost in the local_testing conftest so these lookups succeed against the cassette state the branch is being tested with. * fix(tests): guard conftest backfill against empty local cost map --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
gym-cmd
|
eae390b16d | fix(mcp): respect client ip for delegated auth | ||
|
Claude
|
488fcc25e1
|
fix(jwt,mcp): fall back to global JWKS on unknown issuer; prune fetch locks
- handle_jwt._get_configured_issuer now returns None for tokens whose 'iss' is not in the configured issuers list, letting auth_jwt fall through to the legacy JWT_PUBLIC_KEY_URL path instead of hard-raising. This keeps existing tokens from non-configured IdPs working when an operator adds the new 'issuers' list to a live deployment. - discoverable_endpoints._prune_oauth_metadata_cache now also prunes entries in _OAUTH_METADATA_FETCH_LOCKS whose cache entry has been evicted and whose lock isn't currently held, bounding the locks dict to match the cache it guards. Co-authored-by: Claude <claude@anthropic.com> |
||
|
gym-cmd
|
b4df1a9bde |
Merge remote-tracking branch 'upstream/litellm_internal_staging' into feat/v1.84.0-mcp-gateway-jwt-auth-public
# Conflicts: # .github/workflows/test-unit-proxy-db.yml # litellm-proxy-extras/pyproject.toml # litellm/proxy/_experimental/mcp_server/discoverable_endpoints.py # litellm/proxy/_experimental/mcp_server/oauth_utils.py # litellm/proxy/_experimental/mcp_server/server.py # litellm/proxy/_experimental/out/404.html # litellm/proxy/_experimental/out/__next.__PAGE__.txt # litellm/proxy/_experimental/out/__next._full.txt # litellm/proxy/_experimental/out/__next._head.txt # litellm/proxy/_experimental/out/__next._index.txt # litellm/proxy/_experimental/out/__next._tree.txt # litellm/proxy/_experimental/out/_next/static/8TZ2JbOi7SZ6BCj9ScTHW/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/8TZ2JbOi7SZ6BCj9ScTHW/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/8TZ2JbOi7SZ6BCj9ScTHW/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/L07LLek3NGtynjTxlCdLS/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/L07LLek3NGtynjTxlCdLS/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/L07LLek3NGtynjTxlCdLS/_ssgManifest.js # litellm/proxy/_experimental/out/_next/static/LpD6ruZoEpvYpT5IvMEoa/_buildManifest.js # litellm/proxy/_experimental/out/_next/static/LpD6ruZoEpvYpT5IvMEoa/_clientMiddlewareManifest.json # litellm/proxy/_experimental/out/_next/static/LpD6ruZoEpvYpT5IvMEoa/_ssgManifest.js # litellm/proxy/_experimental/out/_not-found.html # litellm/proxy/_experimental/out/_not-found.txt # litellm/proxy/_experimental/out/_not-found/__next._full.txt # litellm/proxy/_experimental/out/_not-found/__next._head.txt # litellm/proxy/_experimental/out/_not-found/__next._index.txt # litellm/proxy/_experimental/out/_not-found/__next._not-found.__PAGE__.txt # litellm/proxy/_experimental/out/_not-found/__next._not-found.txt # litellm/proxy/_experimental/out/_not-found/__next._tree.txt # litellm/proxy/_experimental/out/api-reference.html # litellm/proxy/_experimental/out/api-reference.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.api-reference.__PAGE__.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.api-reference.txt # litellm/proxy/_experimental/out/api-reference/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/api-reference/__next._full.txt # litellm/proxy/_experimental/out/api-reference/__next._head.txt # litellm/proxy/_experimental/out/api-reference/__next._index.txt # litellm/proxy/_experimental/out/api-reference/__next._tree.txt # litellm/proxy/_experimental/out/chat.html # litellm/proxy/_experimental/out/chat.txt # litellm/proxy/_experimental/out/chat/__next._full.txt # litellm/proxy/_experimental/out/chat/__next._head.txt # litellm/proxy/_experimental/out/chat/__next._index.txt # litellm/proxy/_experimental/out/chat/__next._tree.txt # litellm/proxy/_experimental/out/chat/__next.chat.__PAGE__.txt # litellm/proxy/_experimental/out/chat/__next.chat.txt # litellm/proxy/_experimental/out/experimental/api-playground.html # litellm/proxy/_experimental/out/experimental/api-playground.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.api-playground.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.api-playground.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._full.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._head.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._index.txt # litellm/proxy/_experimental/out/experimental/api-playground/__next._tree.txt # litellm/proxy/_experimental/out/experimental/budgets.html # litellm/proxy/_experimental/out/experimental/budgets.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.budgets.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.budgets.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/budgets/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._full.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._head.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._index.txt # litellm/proxy/_experimental/out/experimental/budgets/__next._tree.txt # litellm/proxy/_experimental/out/experimental/caching.html # litellm/proxy/_experimental/out/experimental/caching.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.caching.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.caching.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/caching/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/caching/__next._full.txt # litellm/proxy/_experimental/out/experimental/caching/__next._head.txt # litellm/proxy/_experimental/out/experimental/caching/__next._index.txt # litellm/proxy/_experimental/out/experimental/caching/__next._tree.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins.html # litellm/proxy/_experimental/out/experimental/claude-code-plugins.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.claude-code-plugins.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.claude-code-plugins.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._full.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._head.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._index.txt # litellm/proxy/_experimental/out/experimental/claude-code-plugins/__next._tree.txt # litellm/proxy/_experimental/out/experimental/old-usage.html # litellm/proxy/_experimental/out/experimental/old-usage.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.old-usage.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.old-usage.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._full.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._head.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._index.txt # litellm/proxy/_experimental/out/experimental/old-usage/__next._tree.txt # litellm/proxy/_experimental/out/experimental/prompts.html # litellm/proxy/_experimental/out/experimental/prompts.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.prompts.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.prompts.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/prompts/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._full.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._head.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._index.txt # litellm/proxy/_experimental/out/experimental/prompts/__next._tree.txt # litellm/proxy/_experimental/out/experimental/tag-management.html # litellm/proxy/_experimental/out/experimental/tag-management.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.tag-management.__PAGE__.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.tag-management.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.experimental.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._full.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._head.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._index.txt # litellm/proxy/_experimental/out/experimental/tag-management/__next._tree.txt # litellm/proxy/_experimental/out/guardrails.html # litellm/proxy/_experimental/out/guardrails.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.guardrails.__PAGE__.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.guardrails.txt # litellm/proxy/_experimental/out/guardrails/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/guardrails/__next._full.txt # litellm/proxy/_experimental/out/guardrails/__next._head.txt # litellm/proxy/_experimental/out/guardrails/__next._index.txt # litellm/proxy/_experimental/out/guardrails/__next._tree.txt # litellm/proxy/_experimental/out/index.html # litellm/proxy/_experimental/out/index.txt # litellm/proxy/_experimental/out/login.html # litellm/proxy/_experimental/out/login.txt # litellm/proxy/_experimental/out/login/__next._full.txt # litellm/proxy/_experimental/out/login/__next._head.txt # litellm/proxy/_experimental/out/login/__next._index.txt # litellm/proxy/_experimental/out/login/__next._tree.txt # litellm/proxy/_experimental/out/login/__next.login.__PAGE__.txt # litellm/proxy/_experimental/out/login/__next.login.txt # litellm/proxy/_experimental/out/logs.html # litellm/proxy/_experimental/out/logs.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.logs.__PAGE__.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.logs.txt # litellm/proxy/_experimental/out/logs/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/logs/__next._full.txt # litellm/proxy/_experimental/out/logs/__next._head.txt # litellm/proxy/_experimental/out/logs/__next._index.txt # litellm/proxy/_experimental/out/logs/__next._tree.txt # litellm/proxy/_experimental/out/mcp/oauth/callback.html # litellm/proxy/_experimental/out/mcp/oauth/callback.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._full.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._head.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._index.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next._tree.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.callback.__PAGE__.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.callback.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.oauth.txt # litellm/proxy/_experimental/out/mcp/oauth/callback/__next.mcp.txt # litellm/proxy/_experimental/out/model-hub.html # litellm/proxy/_experimental/out/model-hub.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.model-hub.__PAGE__.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.model-hub.txt # litellm/proxy/_experimental/out/model-hub/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/model-hub/__next._full.txt # litellm/proxy/_experimental/out/model-hub/__next._head.txt # litellm/proxy/_experimental/out/model-hub/__next._index.txt # litellm/proxy/_experimental/out/model-hub/__next._tree.txt # litellm/proxy/_experimental/out/model_hub.html # litellm/proxy/_experimental/out/model_hub.txt # litellm/proxy/_experimental/out/model_hub/__next._full.txt # litellm/proxy/_experimental/out/model_hub/__next._head.txt # litellm/proxy/_experimental/out/model_hub/__next._index.txt # litellm/proxy/_experimental/out/model_hub/__next._tree.txt # litellm/proxy/_experimental/out/model_hub/__next.model_hub.__PAGE__.txt # litellm/proxy/_experimental/out/model_hub/__next.model_hub.txt # litellm/proxy/_experimental/out/model_hub_table.html # litellm/proxy/_experimental/out/model_hub_table.txt # litellm/proxy/_experimental/out/model_hub_table/__next._full.txt # litellm/proxy/_experimental/out/model_hub_table/__next._head.txt # litellm/proxy/_experimental/out/model_hub_table/__next._index.txt # litellm/proxy/_experimental/out/model_hub_table/__next._tree.txt # litellm/proxy/_experimental/out/model_hub_table/__next.model_hub_table.__PAGE__.txt # litellm/proxy/_experimental/out/model_hub_table/__next.model_hub_table.txt # litellm/proxy/_experimental/out/models-and-endpoints.html # litellm/proxy/_experimental/out/models-and-endpoints.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.models-and-endpoints.__PAGE__.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.models-and-endpoints.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._full.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._head.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._index.txt # litellm/proxy/_experimental/out/models-and-endpoints/__next._tree.txt # litellm/proxy/_experimental/out/onboarding.html # litellm/proxy/_experimental/out/onboarding.txt # litellm/proxy/_experimental/out/onboarding/__next._full.txt # litellm/proxy/_experimental/out/onboarding/__next._head.txt # litellm/proxy/_experimental/out/onboarding/__next._index.txt # litellm/proxy/_experimental/out/onboarding/__next._tree.txt # litellm/proxy/_experimental/out/onboarding/__next.onboarding.__PAGE__.txt # litellm/proxy/_experimental/out/onboarding/__next.onboarding.txt # litellm/proxy/_experimental/out/organizations.html # litellm/proxy/_experimental/out/organizations.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.organizations.__PAGE__.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.organizations.txt # litellm/proxy/_experimental/out/organizations/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/organizations/__next._full.txt # litellm/proxy/_experimental/out/organizations/__next._head.txt # litellm/proxy/_experimental/out/organizations/__next._index.txt # litellm/proxy/_experimental/out/organizations/__next._tree.txt # litellm/proxy/_experimental/out/playground.html # litellm/proxy/_experimental/out/playground.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.playground.__PAGE__.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.playground.txt # litellm/proxy/_experimental/out/playground/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/playground/__next._full.txt # litellm/proxy/_experimental/out/playground/__next._head.txt # litellm/proxy/_experimental/out/playground/__next._index.txt # litellm/proxy/_experimental/out/playground/__next._tree.txt # litellm/proxy/_experimental/out/policies.html # litellm/proxy/_experimental/out/policies.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.policies.__PAGE__.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.policies.txt # litellm/proxy/_experimental/out/policies/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/policies/__next._full.txt # litellm/proxy/_experimental/out/policies/__next._head.txt # litellm/proxy/_experimental/out/policies/__next._index.txt # litellm/proxy/_experimental/out/policies/__next._tree.txt # litellm/proxy/_experimental/out/settings/admin-settings.html # litellm/proxy/_experimental/out/settings/admin-settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.admin-settings.__PAGE__.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.admin-settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._full.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._head.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._index.txt # litellm/proxy/_experimental/out/settings/admin-settings/__next._tree.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts.html # litellm/proxy/_experimental/out/settings/logging-and-alerts.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.logging-and-alerts.__PAGE__.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.logging-and-alerts.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._full.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._head.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._index.txt # litellm/proxy/_experimental/out/settings/logging-and-alerts/__next._tree.txt # litellm/proxy/_experimental/out/settings/router-settings.html # litellm/proxy/_experimental/out/settings/router-settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.router-settings.__PAGE__.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.router-settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/router-settings/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._full.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._head.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._index.txt # litellm/proxy/_experimental/out/settings/router-settings/__next._tree.txt # litellm/proxy/_experimental/out/settings/ui-theme.html # litellm/proxy/_experimental/out/settings/ui-theme.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.ui-theme.__PAGE__.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.settings.ui-theme.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._full.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._head.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._index.txt # litellm/proxy/_experimental/out/settings/ui-theme/__next._tree.txt # litellm/proxy/_experimental/out/skills.html # litellm/proxy/_experimental/out/skills.txt # litellm/proxy/_experimental/out/skills/__next.!KGRhc2hib2FyZCk.skills.__PAGE__.txt # litellm/proxy/_experimental/out/skills/__next.!KGRhc2hib2FyZCk.skills.txt # litellm/proxy/_experimental/out/skills/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/skills/__next._full.txt # litellm/proxy/_experimental/out/skills/__next._head.txt # litellm/proxy/_experimental/out/skills/__next._index.txt # litellm/proxy/_experimental/out/skills/__next._tree.txt # litellm/proxy/_experimental/out/teams.html # litellm/proxy/_experimental/out/teams.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.teams.__PAGE__.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.teams.txt # litellm/proxy/_experimental/out/teams/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/teams/__next._full.txt # litellm/proxy/_experimental/out/teams/__next._head.txt # litellm/proxy/_experimental/out/teams/__next._index.txt # litellm/proxy/_experimental/out/teams/__next._tree.txt # litellm/proxy/_experimental/out/test-key.html # litellm/proxy/_experimental/out/test-key.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.test-key.__PAGE__.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.test-key.txt # litellm/proxy/_experimental/out/test-key/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/test-key/__next._full.txt # litellm/proxy/_experimental/out/test-key/__next._head.txt # litellm/proxy/_experimental/out/test-key/__next._index.txt # litellm/proxy/_experimental/out/test-key/__next._tree.txt # litellm/proxy/_experimental/out/tools/mcp-servers.html # litellm/proxy/_experimental/out/tools/mcp-servers.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.mcp-servers.__PAGE__.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.mcp-servers.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.tools.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._full.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._head.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._index.txt # litellm/proxy/_experimental/out/tools/mcp-servers/__next._tree.txt # litellm/proxy/_experimental/out/tools/vector-stores.html # litellm/proxy/_experimental/out/tools/vector-stores.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.vector-stores.__PAGE__.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.tools.vector-stores.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._full.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._head.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._index.txt # litellm/proxy/_experimental/out/tools/vector-stores/__next._tree.txt # litellm/proxy/_experimental/out/usage.html # litellm/proxy/_experimental/out/usage.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.usage.__PAGE__.txt # litellm/proxy/_experimental/out/usage/__next.!KGRhc2hib2FyZCk.usage.txt # litellm/proxy/_experimental/out/usage/__next._full.txt # litellm/proxy/_experimental/out/usage/__next._head.txt # litellm/proxy/_experimental/out/usage/__next._index.txt # litellm/proxy/_experimental/out/usage/__next._tree.txt # litellm/proxy/_experimental/out/users.html # litellm/proxy/_experimental/out/users.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.users.__PAGE__.txt # litellm/proxy/_experimental/out/users/__next.!KGRhc2hib2FyZCk.users.txt # litellm/proxy/_experimental/out/users/__next._full.txt # litellm/proxy/_experimental/out/users/__next._head.txt # litellm/proxy/_experimental/out/users/__next._index.txt # litellm/proxy/_experimental/out/users/__next._tree.txt # litellm/proxy/_experimental/out/virtual-keys.html # litellm/proxy/_experimental/out/virtual-keys.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.virtual-keys.__PAGE__.txt # litellm/proxy/_experimental/out/virtual-keys/__next.!KGRhc2hib2FyZCk.virtual-keys.txt # litellm/proxy/_experimental/out/virtual-keys/__next._full.txt # litellm/proxy/_experimental/out/virtual-keys/__next._head.txt # litellm/proxy/_experimental/out/virtual-keys/__next._index.txt # litellm/proxy/_experimental/out/virtual-keys/__next._tree.txt # litellm/proxy/auth/auth_utils.py # litellm/proxy/management_endpoints/internal_user_endpoints.py # litellm/proxy/management_endpoints/mcp_management_endpoints.py # litellm/proxy/proxy_server.py # litellm/types/mcp_server/mcp_server_manager.py # pyproject.toml # tests/test_litellm/interactions/test_openapi_compliance.py # tests/test_litellm/litellm_core_utils/prompt_templates/test_litellm_core_utils_prompt_templates_common_utils.py # tests/test_litellm/proxy/_experimental/mcp_server/test_discoverable_endpoints.py # tests/test_litellm/proxy/_experimental/mcp_server/test_mcp_server.py # tests/test_litellm/proxy/_experimental/mcp_server/test_openapi_to_mcp_generator.py # tests/test_litellm/proxy/auth/test_auth_utils.py # tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py # tests/test_litellm/proxy/management_endpoints/test_mcp_management_endpoints.py # tests/test_litellm/proxy/rag_endpoints/test_rag_endpoints.py # tests/test_litellm/proxy/test_proxy_server.py # uv.lock |
||
|
Claude
|
087a4da116
|
fix(tests): backfill local model_cost into remote-fetched map
litellm.model_cost is loaded at import time from LITELLM_MODEL_COST_MAP_URL (pinned to main), so pricing entries that exist only in this branch (e.g. mistral/ministral-8b-2512, freshly added because Mistral's API now returns this id from mistral-tiny) are absent at test time and completion_cost lookups raise 'This model isn't mapped yet'. Backfill the in-tree backup into litellm.model_cost in the local_testing conftest so cassette-driven cost calculations resolve against the entries that ship with the branch under test. Fixes local_testing_part1 failures on test_completion_mistral_api and test_completion_mistral_api_modified_input. |
||
|
Claude
|
d42a66adb6
|
Merge branch 'litellm_internal_staging' into litellm_feat/v1.84.0-mcp-gateway-jwt-auth
Resolved unrelated-history merge using
|
||
|
yuneng-jiang
|
37845acb57
|
fix(tests): migrate realtime + rerank tests off shut-down upstream models (#28191)
* fix(tests): use gpt-realtime in realtime guardrails test
OpenAI shut down gpt-4o-realtime-preview-2024-12-17 on 2026-05-07, so
the live OpenAI realtime guardrails integration test now fails with
model_not_found (session.created never arrives, _wait_for_event times
out). Point OPENAI_REALTIME_URL at the current GA model, gpt-realtime.
Scope limited to this test: the pricing-catalog JSON keeps the retired
entries intentionally (historical cost calc + separate Azure timeline),
and the Azure realtime cost-calc test is unaffected.
* fix(tests): mock nvidia_nim rerank instead of hitting EOL'd endpoint
NVIDIA reached end-of-life for the hosted nvidia/llama-3.2-nv-rerankqa-1b-v2
rerank API on 2026-05-18 with no published replacement, so the live
BaseLLMRerankTest.test_basic_rerank for nvidia_nim now returns HTTP 410
("Gone"). NVIDIA's hosted catalog rotates on a schedule, so swapping in
another live model would only defer the failure.
Override test_basic_rerank in TestNvidiaNim to mock the sync/async HTTP
transport (same pattern as test_nvidia_nim_rerank_ranking_endpoint in this
file) and inject a fake NVIDIA_NIM_API_KEY via monkeypatch. The
request/response transformation and cost calculation stay covered offline.
Scope limited to nvidia_nim; other BaseLLMRerankTest providers untouched.
* fix(tests): migrate remaining realtime tests off shut-down gpt-4o-realtime-preview
OpenAI's 2026-05-07 shutdown removed the entire gpt-4o-realtime-preview
family, including the undated 'gpt-4o-realtime-preview' alias (not just the
dated snapshot fixed earlier). Three live tests still connected with the
dead alias and failed with messages_received=1 (an error event instead of
session.created):
- test_openai_realtime_simple.py: get_model() -> gpt-realtime (drives
TestOpenAIRealtime.test_realtime_connection / test_realtime_with_query_params)
- test_openai_realtime.py: test_openai_realtime_direct_call_no_intent and
test_openai_realtime_direct_call_with_intent -> openai/gpt-realtime
(the with_intent test shares the same dead alias even though it was not
in the failing set this run)
Mocked unit tests (test_realtime_query_params_construction,
test_realtime_query_params_use_normalized_model_name) are left as-is: they
never hit the network and assert string plumbing only.
Also fixes test_text_message_blocked_by_guardrail_no_ai_response, which now
connects (the earlier URL swap worked) but tripped a model-wording-brittle
assertion. The guardrail flow asks the model to voice the block message
verbatim; gpt-4o-realtime-preview complied (output contained 'blocked'),
gpt-realtime refuses verbatim-repeat instructions ('I'm sorry, but I can't
repeat that message.'). Since the original user message is blocked before
it reaches OpenAI, the refusal is still a safe outcome. Assertion #3 now
accepts both voicing and refusal, and adds a hard check that the blocked
phrase never leaks into AI output.
(cherry picked from commit
|
||
|
yuneng-jiang
|
c65e598402
|
fix(tests): replace shut-down gpt-4o-audio-preview with gpt-audio-1.5 (#28281)
* fix(tests): replace shut-down gpt-4o-audio-preview with gpt-audio-1.5
OpenAI shut down gpt-4o-audio-preview on 2026-05-07, so the live audio
calls in test_stream_chunk_builder_openai_audio_output_usage and
test_standard_logging_payload_audio now hard-fail with a model-not-found
error on every PR. The error was not "openai-internal", so the except
block swallowed it and execution fell through to an unbound
completion/response (UnboundLocalError).
Switch both tests to gpt-audio-1.5, OpenAI's recommended successor
(GA, not deprecated, already present in the litellm cost map so the
response_cost assertion still resolves). Also broaden the except to
skip with the real error in the reason instead of crashing, so a
transient upstream blip can't reintroduce the UnboundLocalError.
* fix(tests): narrow audio-test skip to model-not-found, re-raise the rest
Address review feedback: an unconditional skip on any exception would
silently mask a litellm-internal regression in the audio path (broken
param transformation, serialization, bad header) instead of failing CI.
Skip only on the upstream-unavailable class (model_not_found / "does not
exist" / openai-internal) and re-raise everything else, so genuine
regressions still fail loudly. The UnboundLocalError is still fixed
because the handler either skips or raises - it never falls through.
* fix(tests): add budget_exceeded to expected Interaction status enum
Staging added budget_exceeded to the Interaction OpenAPI status enum; the staging merge into this branch picked up the spec change but not the matching test update, so test_status_enum_values failed in CI. Align the test's expected list (exact-match by design) with the live spec.
* fix(tests): mock HTTP fetch in test_img_url_token_counter
The test parameterized a live third-party image URL (blog.purpureus.net) which now 404s, causing get_image_dimensions to fall through to its base64 decode path and crash with 'not enough values to unpack' on every PR run. Mock safe_get with a tiny 1x1 PNG so the URL branch is still exercised without any network dependency.
* fix(tests): swap gpt-4o-audio-preview to gpt-audio-1.5 in test_gpt4o_audio
OpenAI shut down gpt-4o-audio-preview on 2026-05-07, so both live tests in test_gpt4o_audio.py (test_audio_output_from_model and test_audio_input_to_model) hard-fail model_not_found on every PR. Swap the hardcoded model to OpenAI's successor gpt-audio-1.5 (same chat-completions audio surface; already in the litellm cost map). Mirror the narrowed-skip pattern from the prior audio fixes: skip on model_not_found / does-not-exist / openai-internal, re-raise everything else so genuine litellm regressions still fail CI loudly.
(cherry picked from commit
|
||
|
Yuneng Jiang
|
76700e7bb3
|
test(fireworks): mock remaining live smoke tests
test_completion_fireworks_ai and test_completion_cost_fireworks_ai
made real Fireworks calls and broke whenever Fireworks rotated its
serverless catalog (no externally-verifiable model list exists).
They also asserted nothing — just printed.
Mock the HTTP post and assert real behavior instead: the request is
built with the right model/messages and the OpenAI-compatible
response parses back; the cost path yields a non-zero cost against
the local cost map. No network, no model dependency, stronger than
the old smoke checks.
(cherry picked from commit
|
||
|
Yuneng Jiang
|
a0b61f6dcb
|
test(fireworks): replace deprecated llama-v3p3-70b-instruct model
Fireworks removed llama-v3p3-70b-instruct from serverless, so every
live test using it now fails with NotFoundError ("Model not found,
inaccessible, and/or not deployed").
Swap the 6 references (3 files) to the currently-served
accounts/fireworks/models/deepseek-v3p1 — the canonical model in
Fireworks' current docs examples and present in LiteLLM's cost map.
test_get_model_params_fireworks_ai is a pure pricing-heuristic test
(no network) asserting the >16b branch, so it uses llama-v3p1-70b-
instruct instead to keep the "fireworks-ai-above-16b" assertion and
branch coverage intact.
(cherry picked from commit
|
||
|
Yuneng Jiang
|
93d3ed7ee5
|
fix(tests): drop dall-e-only test classes; route live image tests via gpt-image-1
Second wave of failures from the 2026-05-12 DALL-E shutdown:
- tests/image_gen_tests/test_image_edits.py::TestOpenAIImageEditDallE2
and tests/image_gen_tests/test_image_generation.py::TestOpenAIDalle3
are explicitly named for the deprecated models and can't pass; remove.
gpt-image-1 coverage already exists in sibling classes.
- tests/local_testing/test_router.py image gen tests use dall-e-3 only
as a routing example; swap to gpt-image-1.
- tests/local_testing/test_custom_callback_input.py image_generation
success/failure paths swapped to gpt-image-1.
(cherry picked from commit
|
||
|
yuneng-jiang
|
a119324127
|
test: replace dall-e-3 with gpt-image-1 in health check and router tests (#27813)
OpenAI returns 'The model dall-e-3 does not exist' for the test account,
breaking test_openai_img_gen_health_check and test_image_generation.
Switch to gpt-image-1, matching the existing TestOpenAIGPTImage1 pattern.
(cherry picked from commit
|
||
|
Claude
|
806b8be451
|
fix(mcp,jwt): address greptile review concerns
- Cache _get_agent_object_permission via user_api_key_cache (sentinel for no-permission rows) so MCP requests from agent keys don't hit the DB on every tool-list / tool-call. - Re-raise HTTPException in handle_sse_mcp so 401 + WWW-Authenticate challenges (and other HTTP errors) propagate to SSE clients instead of being swallowed as 500. - Normalise booleans in _validate_token_response so admin rules written as JSON-style "true" / "false" match upstream responses that return Python True / False. - Treat configured JWT issuer claim mappings as advisory: when a mapped field is absent or empty, leave the normalised claim unset instead of raising, matching the global litellm_jwtauth path. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
gym-cmd
|
216e055910 | fix(interactions): align status enum with openapi spec | ||
|
Artem Dudarev
|
f6359cbf04 |
fix(mcp): forward Authorization in pass-through when x-litellm-api-key is admission
Commit
|
||
|
Sameer Kankute
|
e59e34bed3
|
Gemini managed agents support (#28270)
Some checks are pending
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
* Add support for environment variable in interactions api * Add sdk support for gemini create agent * Add agents endpoint support via proxy * Add outputs of each api * Add routing for model and agents param * Remove redundant condition in get_provider_agents_api_config LlmProviders.GEMINI.value is literally the string "gemini", so the second clause of the or was checking the exact same thing as the first. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: forward query-param credentials to list/get/delete/versions Gemini agent endpoints The list_gemini_agents, get_gemini_agent, delete_gemini_agent, and list_gemini_agent_versions endpoints previously constructed a hardcoded data dict with no mechanism to pass provider credentials. Unlike create_gemini_agent (POST, reads litellm_params_template from body), these GET/DELETE endpoints gave no way for multi-tenant callers to supply a per-request api_key or other LiteLLM params. Fix: - Add _merge_query_params_into_data() helper that reads query parameters from the request and merges them into the data dict without overwriting already-set keys (e.g. path params like 'name'). - Support a JSON-encoded litellm_params_template query parameter (matching the POST body pattern) as well as flat key=value pairs (e.g. api_key=AIza...). - Apply the helper in all four affected endpoints. - Add 13 unit tests covering the helper and each endpoint. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: pass model=None for managed agent proxy endpoints to prevent agent name polluting data["model"] Endpoints acreate_agent, aget_agent, adelete_agent, and alist_agent_versions were passing model=<agent_name> to base_process_llm_request. This caused common_processing_pre_call_logic to write the agent name into self.data["model"], which then triggered spurious model-alias mapping, rate-limiting lookups, and logging tied to a non-existent model deployment. The agent name is already carried in data["name"] and is passed correctly to the SDK functions (litellm.interactions.agents.*). There is no reason to also set model=<agent_name>; the correct value is model=None for all five managed-agent management routes. Adds tests/test_litellm/proxy/google_endpoints/test_managed_agents_model_param.py to verify all five managed-agent endpoints pass model=None. Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> * fix: address greptile P1/P2 review comments P1 (router.py): Restore fallback/retry support for acreate_interaction and create_interaction. Both were silently moved to _init_interactions_api_endpoints (direct call, no fallbacks). Moved them back to _ageneric_api_call_with_fallbacks so users with configured fallback models keep retry behaviour. P1 security (agents_endpoints.py): Remove flat query-param credential path (e.g. ?api_key=AIza...) from _merge_query_params_into_data. Credentials in URL query strings appear verbatim in server access logs, CDN edge logs, and browser history. Only the JSON-encoded litellm_params_template query param (matching the POST body pattern) is retained. P2 (interactions/http_handler.py): Extract _BaseHTTPHandler with shared _handle_error, _sync_client, and _async_client helpers. InteractionsHTTPHandler now extends _BaseHTTPHandler. The _async_client reads the provider from litellm_params instead of hardcoding GEMINI. P2 (interactions/agents/http_handler.py): AgentsHTTPHandler now extends InteractionsHTTPHandler (which inherits _BaseHTTPHandler) so all shared HTTP infrastructure is reused rather than duplicated. Removes the hardcoded LlmProviders.GEMINI from the async client path. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: address CI failures from greptile review fixes - black: format interactions/agents/main.py and utils.py - tests: update test_gemini_agents_endpoints.py to match new _merge_query_params_into_data behaviour (flat credential params are rejected; only JSON-encoded litellm_params_template is accepted) - ci: add test_gemini_agents_endpoints.py to endpoints-and-responses shard in test-unit-proxy-db.yml so assert-shard-coverage passes - tests: add _initialize_managed_agents_endpoints and _init_managed_agents_api_endpoints test coverage so router_code_coverage passes; also fix TestRouterCreateInteractionRouting to reflect that acreate_interaction now correctly routes through _ageneric_api_call_with_fallbacks (restoring fallback support) Co-authored-by: Cursor <cursoragent@cursor.com> * fix: remove InteractionsHTTPHandler._handle_error override to fix type errors AgentsHTTPHandler extends InteractionsHTTPHandler and calls self._handle_error(provider_config=agents_api_config) where agents_api_config is BaseAgentsAPIConfig. Python MRO resolved _handle_error to InteractionsHTTPHandler._handle_error which expected BaseInteractionsAPIConfig, causing 10 mypy arg-type errors in interactions/agents/http_handler.py. Removing the redundant override lets both classes inherit _BaseHTTPHandler._handle_error (provider_config: Any) which is structurally correct for both config types. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: agent-only interactions and managed agents provider routing Resolve None custom_llm_provider in agents HTTP client lookup and set custom_llm_provider on GenericLiteLLMParams for all agent CRUD paths. Stop mapping agent names to proxy model routing; route interactions through _init_interactions_api_endpoints with fallbacks only when model is set. Consolidate duplicate router elif branches for interaction APIs. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix greptile review * test(agents): add unit tests for managed agents SDK and HTTP handler Adds coverage for the new `litellm.interactions.agents` surface area: - main.py: sync/async entry points (create/list/get/delete/list_versions), provider config lookup, logging-obj helper, async error wrapping - http_handler.py: every CRUD method (sync + async paths), `_is_async` dispatch branches, and provider error mapping through GeminiAgentsConfig - utils.py: get_provider_agents_api_config for supported / unsupported providers Brings patch coverage on these files from <25% to ~100% so codecov/patch is satisfied. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * docs(gemini-agents): fix misleading credential-passing examples in GET/DELETE docstrings (#28293) The four GET/DELETE endpoint docstrings (list_gemini_agents, get_gemini_agent, delete_gemini_agent, list_gemini_agent_versions) documented passing per-request credentials as flat query parameters (e.g. ?api_key=AIza...). However, _merge_query_params_into_data only reads the JSON-encoded litellm_params_template query parameter and intentionally ignores flat params (URL query strings appear verbatim in access logs, browser history, and Referer headers). Callers following the documented curl examples would have their credentials silently dropped and hit auth failures against Gemini. Update the examples to use the supported JSON-encoded litellm_params_template query parameter, matching _merge_query_params_into_data's own docstring. Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * refactor(agents): rename provider-agnostic agent response types Move GeminiAgent{ListResponse,DeleteResult,VersionsResponse} to provider-neutral names (AgentListResponse, AgentDeleteResult, AgentVersionsResponse) so the BaseAgentsAPIConfig interface no longer references Gemini-specific type names. * fix(gemini-agents): close veria-flagged credential-escalation gaps Two high-severity findings from the veria-ai PR review are addressed: 1. **api_base override could leak the shared Gemini key** GeminiAgentsConfig.validate_environment falls back to GOOGLE_API_KEY / GEMINI_API_KEY when no api_key is supplied. Combined with caller-controlled api_base on the proxy CRUD endpoints, an authenticated user could redirect the outbound request to an attacker-controlled host and capture the operator's shared Gemini key from the x-goog-api-key header. The config now refuses env-fallback whenever api_base is explicitly overridden. 2. **Managed-agent CRUD exposed to ordinary LLM keys** The new /v1beta/agents routes live in google_routes (i.e. llm_api_routes), so any non-admin LLM key can reach them. Unlike /v1beta/models/...: generateContent these endpoints are NOT model-routed and have no model_list-supplied credentials, so env-fallback would let any LLM key list / create / delete agents inside the operator's Gemini project. Each endpoint now calls _enforce_caller_supplied_provider_key, which requires non-admin callers to supply their own Gemini api_key via litellm_params_template. Proxy admins keep the env-fallback convenience. Tests cover non-admin rejection, admin allow-through, the api_base override guard, and SDK env-fallback when api_base is not overridden. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * test(router): restore strict assert_called_once_with on interactions default-provider test --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Sameer Kankute <Sameerlite@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
3c3d131f01
|
Day 0 support : Gemini 3.5 Flash (#28268)
* Add day 0 support for gemini 3.5 flash * Fix pricing * Fix greptile review * Fix failing test * Fix tests * Fix: revert tool removing logic * fix greptile and test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Yassin Kortam
|
cde4b1a94d
|
feat: propagate team_id and team_alias to all child OTEL spans (#28273)
- Add `_set_team_attributes_on_span` helper to stamp team_id/team_alias onto any span, ensuring these attributes are not limited to the root litellm_request span - Add `_set_team_attributes_from_kwargs` helper to extract team metadata from the standard_logging_object in kwargs and apply them to a span - Apply team attributes to raw request spans via `_maybe_log_raw_request` so downstream consumers can filter traces by team without needing the root span - Apply team attributes to guardrail spans so guardrail activity can be correlated to teams in tracing backends - Apply team attributes to exception logging spans to preserve team context during failure paths - Add comprehensive unit tests covering all new helpers, including edge cases where metadata or standard_logging_object is absent Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |