mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
2787 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mateo Wang
|
72bcb748b9
|
chore: remove _experimental/out (#31546)
* chore: remove _experimental/out * fix(ci): recreate _experimental/out before copying UI build output The build scripts cp the Next.js output into litellm/proxy/_experimental/out, which was removed from git. cp failed because the target directory no longer existed; mkdir -p recreates it before the copy. * fix(proxy): make UI serving resilient to a missing _experimental/out Removing the committed UI export means the source/test tree no longer ships litellm/proxy/_experimental/out. Three things assumed it was always present and broke once it was gone: - get_favicon hard-coded the built favicon path and 404'd without it; it now falls back to the bundled swagger/favicon.ico - the /_next and /ui static mounts raised at construction when the export was absent, so the whole UI-setup block was swallowed and no mounts registered; they now use check_dir=False - _restructure_ui_html_files was a nested function only exposed as a module attribute when that block happened to succeed; it is now a real module-level function test_admin_ui_export_serves_nested_extensionless_routes validated the committed artifact, whose premise this PR removes; it now drives the same MCP OAuth callback restructure guarantee through a synthetic export. * chore(greptile): ignore generated _experimental/out so review fits the file limit * Revert "chore(greptile): ignore generated _experimental/out so review fits the file limit" ignorePatterns is applied after Greptile counts the files changed, so it does not bring the diff under the file limit; the config had no effect. |
||
|
tin-berri
|
7baf25526f
|
fix(mcp): support client_secret_basic for upstream OAuth token endpoints (#31635)
The MCP gateway authenticated to upstream OAuth token endpoints only with
client_secret_post (client_secret placed in the POST body). Providers that
require HTTP Basic client authentication (client_secret_basic, the OIDC
default) reject that with invalid_client, which surfaced as a 500 on the
/<server>/token exchange and broke both the initial authorization_code
exchange and refresh.
Add a per-server token_endpoint_auth_method ("client_secret_basic" |
"client_secret_post") and a single helper that builds the right headers and
body for the configured method, then route every upstream token-endpoint POST
through it: the inbound exchange and refresh in discoverable_endpoints, the v1
per-user refresh in db, the v2 authorization_code refresher, the M2M
client_credentials fetch, and the RFC 8693 token exchange. The default stays
client_secret_post so existing servers are unaffected; basic sends
Authorization: Basic base64(form-urlencode(client_id):form-urlencode(secret))
per RFC 6749 section 2.3.1 and omits the secret from the body.
client_secret_basic is a confidential-client method, so a server configured for
it with a missing client_id/secret raises rather than silently downgrading to a
body request (no-silent-fallback); the inbound endpoint maps that to a 400 and
the refresh paths to a failed-refresh / needs-reauth. A secretless client_id
under the default method stays valid for public clients authenticating with PKCE.
Resolves LIT-4091
|
||
|
Mateo Wang
|
375659ef04
|
fix(proxy): emit x-litellm-response-cost header on /messages and /generateContent (LIT-4076) (#31675)
* fix(proxy): emit x-litellm-response-cost header on /messages and /generateContent (LIT-4076) The Anthropic /v1/messages and Google native :generateContent routes return TypedDict results (AnthropicMessagesResponse, GenerateContentResponseBody) that are plain dicts at runtime and cannot hold a _hidden_params attribute. The cost is computed by update_response_metadata, but ResponseMetadata.apply() only persists _hidden_params back when the result object has that attribute, so for those two routes the computed response_cost was dropped. The non-streaming header build in base_process_llm_request then saw an empty response_cost and get_custom_headers filtered the x-litellm-response-cost header out, even though the other x-litellm-* headers still appeared. The non-streaming success path now recovers the cost from the logging object when the response cannot carry _hidden_params, preferring the value already stored in model_call_details and recomputing from the same calculator only when it has not been stored yet. Object responses (ModelResponse, ResponsesAPIResponse) keep their existing behavior, so chat/completions, /responses, and the Anthropic error path that intentionally emits a zero cost are unaffected. Streaming stays out of scope because the header is emitted at stream start, before the cost is known. * fix(proxy): also recover response cost header for /generateContent responses with _hidden_params (LIT-4076) * fix(proxy): compute generateContent response cost synchronously so cost header is emitted (LIT-4076) * fix(lint): suppress BLE001 on generate_content cost normalization guard The defensive blind except keeps cost normalization from ever breaking the response path; mark it noqa so it does not breach the strict-rule budget. |
||
|
Krrish Dholakia
|
ea7be19225
|
fix: skip health check for semantic auto_router deployments (#31668)
* fix: skip health check for semantic auto_router deployments
auto_router/<name> deployments are semantic meta-routers that select among
real LLM deployments at request time. They have no LLM endpoint to probe.
The health check was passing model=auto_router/router_1 to get_llm_provider(),
which raised BadRequestError: "Unmapped LLM provider for this endpoint" because
auto_router is not a real LLM provider, causing these deployments to always
appear unhealthy and curl requests to hang.
Detect semantic auto_router deployments in _run_model_health_check and return
{} (healthy) without calling litellm.ahealth_check. Sub-strategies
(complexity_router, adaptive_router, quality_router) are excluded from this
fast path and continue to be health-checked normally.
* ci: trigger circleci
|
||
|
yucheng-berri
|
10849c880b
|
fix(guardrails): scan file and document attachments with Model Armor (#31655)
The Model Armor guardrail only sent text extracted from user messages to sanitizeUserPrompt, so harmful content inside attached PDFs, Office docs, and CSVs reached the LLM unscanned. A file-only message had no extractable text, so the pre-call and moderation hooks returned early and the document was never submitted to Model Armor at all. Wire inline document/file scanning into async_pre_call_hook and async_moderation_hook. extract_file_attachments walks message content blocks (OpenAI type:file file_data and Anthropic type:document source), decodes the base64 bytes, maps the MIME type to a Model Armor byteDataType, and skips remote URLs, bare file_id references, oversize files past the 4 MB limit, and unsupported types. Each attachment is sent through the byte API and a MATCH_FOUND blocks the request before it reaches the LLM. Resolves LIT-4084 |
||
|
yuneng-jiang
|
494d04c2a2
|
Merge pull request #31471 from BerriAI/litellm_veria_218_personal_key_metadata
fix(proxy): reject team-scoped object_permission on personal keys for non-admins |
||
|
tin-berri
|
ec808edece
|
fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint (#31657)
* fix(mcp): resolve per-user OAuth identity authoritatively at the token endpoint The OAuth token endpoint stored a user's per-server token under the identity returned by _extract_user_id_from_request, which read only the Authorization header and did getattr(cached, "user_id") on a raw user_api_key_cache lookup with no model_type rehydration and no DB fallback. That silently returned None in two common cases on a multi-replica gateway: the LiteLLM key arrives on x-litellm-api-key (what MCP clients such as Claude Desktop and Claude Code send) rather than Authorization, and a cross-replica cache hit deserializes to a plain dict rather than a UserAPIKeyAuth, so getattr finds no attribute. When it returned None the token was not persisted. This was survivable until the authorization_code v2 migration began stripping the caller's Authorization for migrated per-user OAuth servers and routing the preemptive 401 existence check through the stored token, so a persist miss now hard-fails: the egress challenges with 401 on every reconnect (the client sees "rejected them on reconnect" or a successful connect with zero tools). Resolve identity through get_key_object, the canonical resolver that reads the cache with model_type and falls back to the DB, and accept the key from x-litellm-api-key as well as Authorization. The silent persist skip is now a warning. The caller-Authorization stripping stays as is, since reinstating it would reopen the cross-user credential override it was added to prevent. * fix(mcp): reject blocked or expired keys when resolving the token-endpoint identity The OAuth token endpoint is unauthenticated, and get_key_object resolves a key row without the blocked/expiry checks the main user_api_key_auth pipeline runs (that pipeline is bypassed here). So a holder of a revoked or expired LiteLLM key could POST a valid upstream authorization code with that key in x-litellm-api-key/Authorization and write or overwrite the stored per-user OAuth token for that key's user. The cache-only resolver this replaced incidentally dropped blocked keys (blocking purges the cache entry), so moving to the authoritative cache-then-DB resolution removed that accidental shield. Validate the resolved key before trusting its identity: return None when blocked or expired, so the upsert is skipped. Deleted keys are already rejected, since get_key_object raises on a missing row. Regression tests cover the blocked and expired cases and fail without the guard. |
||
|
ryan-crabbe-berri
|
e195532c14
|
fix(proxy): count only active users toward license seat limit (#31227)
* fix(proxy): count only active users toward license seat limit SCIM-deactivated users (metadata.scim_active == false) are kept in LiteLLM_UserTable for audit and reactivation, but they were still counted toward the per-user license limit, so deactivating a user never freed a seat. Okta never sends a SCIM DELETE and Entra only hard-deletes well after deactivation, so deactivation has to be what frees the seat Add UserRepository.count_billable_users(), which counts every row except those where metadata.scim_active is false (absent, null, and true all count), and route the user-create license gate, the free-SSO 5-user cap, and the enterprise /user/available_users display through it. A separate litellm_active_users Prometheus gauge reports the billable count while litellm_total_users keeps its original meaning so existing dashboards are unaffected * fix(proxy): floor billable user count at zero count_billable_users() runs two separate count queries (total, then deactivated). Under a burst of deactivations between them, the deactivated count can momentarily exceed the earlier total and produce a negative result, which would flow into is_over_limit as a negative and show a negative seat count in the display and gauge. Clamp the result to zero so a transient race can never yield a nonsensical negative; the value self-corrects on the next call Addresses Greptile P1 on the PR * refactor(proxy): count teams via TeamRepository in available_users * style: ruff format changed files at line-length 120 |
||
|
Cursor Agent
|
5b029ecd08 | fix: preserve normalized mcp permissions on key regenerate | ||
|
yucheng-berri
|
971a1bedc7
|
fix(proxy): hard-reject CLI session token personal-key budget_limits (#31631)
Mirror the scalar `max_budget` guard in `_common_key_generation_helper` for the per-window check: a CLI session token caller (carrying `max_budget=None`) cannot set `budget_limits` on a personal key. Pass `team_table` into the helper so it can detect the personal-key shape; reject before the `delegation_ceiling is None` early return. Four new regression tests cover the personal-key reject, the team-key happy path, the team-key over-team-budget path, and the proxy-admin exemption. |
||
|
yucheng-berri
|
be6b28f25e
|
fix(proxy): reject non-finite budget_limits windows on /key/generate (#31630)
Enforce that every `budget_limits[*].max_budget` is a finite number; applies to every caller including proxy admin and runs before the role / ceiling checks. Six parametrized regression tests cover NaN / +inf / -inf for both non-admin and admin callers. |
||
|
Yassin Kortam
|
829bfebe0f
|
perf(auth): gather independent pre-call budget-enforcement reads (#31604)
increment_spend_counters was parallelized in #31578, but the dominant per-request cost under high concurrency is the pre-call budget enforcement in common_checks, which still ran a Redis-first get_current_spend per scope (team, team windows, key windows, org, tag, user, team member, end user) one sequential await after another inside the auth span. The per-scope reads target distinct counter keys with no cross-scope ordering dependency, so they now run concurrently under asyncio.gather. Key metadata.tags injection still runs before the gather so the tag budget check sees it, and every scope settles before the first error in scope-priority order propagates, preserving the previous rejection semantics. Resolves LIT-4090 |
||
|
yucheng-berri
|
7a1ba958f8
|
fix(proxy): gate non-admin /key/generate budget_limits and permissions (VERIA-392) (#31469)
/key/generate validated the caller's delegation ceiling against
data.max_budget only. The per-window entries in data.budget_limits
bypassed the check, so a non-admin caller could mint a key whose
1-day window vastly exceeded their own max_budget. The data.permissions
dict also went unvalidated for non-admin callers, so they could
self-grant capabilities like allow_pii_controls (and on Enterprise,
get_spend_routes).
Both gates now live in _common_key_generation_helper, covering
/key/generate and /key/service-account/generate. The existing empty
{} default on permissions still passes for non-admin callers.
|
||
|
michelligabriele
|
d7654d07ab
|
feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration (#31215)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration * test(proxy): add behavior scenarios for credential migration endpoints * fix(proxy): scan covered tables in encryption check, fix CI lint and route types * fix(proxy): migrate callback_settings credentials, clear CI lint/recursion gates, add encryption endpoint+CLI tests * fix(proxy): correct dry-run/real-run migrated vs residual-legacy counters in config and SSO walkers * fix(proxy): make callback-vars residual detection gate-independent in encryption check |
||
|
Sameer Kankute
|
8e30cfbeb1
|
feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents (#30950)
* feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents Bump a2a-sdk to 1.x and wire send/stream through compat conversions so the proxy accepts A2A 1.0 JSON-RPC while preserving 0.3 wire clients. Co-authored-by: Cursor <cursoragent@cursor.com> * Add user controlled protocol version in agents * Fix exeception mapping * Fix a2a base url * Add e2e test for a2a * Fix lint * Fix lint * fix(a2a): harden card version detection and header isolation coverage Use protocolVersion when inferring agent card wire format, assert distinct httpx cache keys in the header-isolation test, and suppress targeted basedpyright errors for optional SDK imports. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): suppress reportArgumentType for SDK compat types and fix streaming trace ID - Add pyright: ignore[reportArgumentType] to SendMessageSuccessResponse id= and result= args in _send_message, and SendStreamingMessageResponse root= in _stream_messages, where a2a-sdk compat types diverge from basedpyright's inferred signature, reducing the reportArgumentType count back within budget. - Fix streaming trace ID in astream_a2a_message to use str(request.id) when available instead of always generating a new uuid4(), restoring JSON-RPC request-ID correlation for observability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style(a2a): expand SendStreamingMessageResponse for black formatting Move pyright: ignore comment to the root= argument line so Black accepts the expanded multi-line form. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): fix 2 reportArgumentType errors without suppression - main.py: narrow logging_obj from object|None to Optional[Logging] via isinstance check before A2AStreamingIterator call, fixing the "Logging | object" argument type mismatch at line 699. - a2a_endpoints.py: extract response_dict with explicit isinstance(dict) guard before passing to normalize_jsonrpc_response, fixing the "LLMResponseTypes | dict[str, Any]" type mismatch at line 835. - Remove spurious pyright: ignore comments added in previous commits that were not suppressing the actual errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): rewrite upstream URL for 1.0 agent cards in getAuthenticatedExtendedCard 1.0 upstream agent cards store the endpoint URL in supportedInterfaces[0].url rather than a top-level url field. The previous guard only rewrote url when it existed at the top level, so after normalize_agent_card lowered a 1.0 card to 0.3 the upstream internal address leaked into the url field of the 0.3 response. Fix: rewrite both url and supportedInterfaces[0].url to the proxy address before calling normalize_agent_card, ensuring the upstream address is never visible to downstream clients regardless of the upstream card's wire format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: extend _served_version to all PascalCase methods; add direct httpx-client isolation proof - _served_version now checks `_PASCAL_TO_WIRE` membership instead of two hardcoded names, so GetTask/CancelTask/etc. are promoted to 1.0 wire format alongside SendMessage — prevents mixed wire formats mid-session - test_create_a2a_client_uses_fresh_httpx_client now asserts a2a_client_a._litellm_httpx_client is not a2a_client_b._litellm_httpx_client (direct proof that header bleed cannot occur), in addition to the cache-key inequality check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: id:0 silently dropped in version_convert; explicit continue in stream retry - version_convert.py: replace `request_id or ""` with `str(request_id) if request_id is not None else ""` in both _send_result_to and _stream_result_to; id=0 is valid JSON-RPC and must not be coerced to "" which breaks response correlation - main.py: add explicit `continue` after the A2ALocalhostURLError retry in _execute_a2a_stream_with_retry so the control flow (retry → next iteration → stream_succeeded guard) is unambiguous Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve a2a retry and discovery card urls * Fix black * Fix test * fix(a2a): avoid KeyError in discovery log after 0.3→1.0 card normalization When a 0.3-style agent card is normalized to 1.0, the top-level url key is replaced by supportedInterfaces; log the already-computed proxy_url instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): preserve taskId when lowering push notification config set params Flatten 1.x create envelope fields before parsing into TaskPushNotificationConfig so 1.0 clients forwarding to 0.3 upstream keep taskId and config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): ignore unknown fields in message/send proto fallback ParseDict in _build_message_send_params now matches other inbound paths so 1.0 clients with extra proto fields are not rejected with -32602. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): normalize tasks/list params and response across protocol versions Convert list task entries on the response path and lower ListTasksRequest params including status filters when forwarding 1.0 clients to 0.3 upstream. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): avoid reportArgumentType in _lower_list_tasks_params; use local var instead of _parse return Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(a2a): drop private SDK symbol in tasks/list status lowering _lower_list_tasks_params imported _CORE_TO_COMPAT_TASK_STATE, a private a2a-sdk symbol that could disappear on a patch release and silently break status-filter lowering. Derive the 0.3 wire string from the public protobuf enum name instead (TASK_STATE_<NAME> maps to the 0.3 value once the prefix is dropped and underscores become dashes) and validate the result against the 0.3 TaskState enum's own values via a fully-typed pure helper. Behavior is unchanged for every state; unspecified or unrecognized states still drop the filter. Adds parametrized regression tests covering dashed wire values (input-required, auth-required) and the unspecified drop. * fix(a2a): drop redundant push-notification envelope key; unify MessageToDict import _flatten_create_push_notification_params used `config or pushNotificationConfig`, which short-circuits so a co-present pushNotificationConfig key was never popped and leaked into the flattened params. Pop both keys unconditionally and prefer config when present. Adds a regression test on the helper that fails on the old leak. Also import MessageToDict from a2a.compat.v0_3.conversions in _lower_list_tasks_params to match every other conversion helper in the module instead of pulling it straight from google.protobuf.json_format. * fix(a2a): reject invalid message/stream params early with -32602 _handle_stream_message built MessageSendParams lazily inside the stream_response() generator, so malformed 1.0 params surfaced as a generic -32603 after the 200 status line was already committed. The non-streaming path validates up front and returns -32602 (Invalid params). Validate eagerly before returning the StreamingResponse and emit -32602 on failure so both paths reject malformed params identically. Adds a regression test asserting the streamed error code is -32602. * fix(a2a): raise clear error when non-streaming send ends on an update event _send_message fed the SDK iterator's last event straight into SendMessageSuccessResponse, whose result only accepts Message or Task. A non-standard upstream whose final event is a TaskStatusUpdateEvent or TaskArtifactUpdateEvent made the response construction raise an opaque pydantic ValidationError. Guard the converted result and raise a clear RuntimeError instead, consistent with the no-response guard above it. Adds regression tests for the Message happy path and the update-event rejection via an injected fake client. * test(a2a): lock in clean merged agent-card URL without PROXY_BASE_URL Regression coverage proving _build_merged_agent_card produces no double slash in supportedInterfaces[0].url when PROXY_BASE_URL is unset and request.base_url carries a trailing slash. get_custom_url routes through join_paths, which rstrips the base, so the f-string join stays clean. * style(a2a): modernize type annotations to satisfy strict ruff budget After merging the black->ruff-format migration from base, the A2A files owned by this PR still used Optional[X]/quoted annotations that pushed UP037/UP045 over their lowered ceilings. Convert to X | None, drop the now-unnecessary quoted local annotation in _send_message, and remove the imports left unused by the rewrite. Type semantics are unchanged. * style(a2a): type a2a_endpoints dict params as dict[str, Any] The merge with the formatter-migration baseline tightened the reportUnknownArgumentType ceiling; bare dict annotations made every value Unknown and pushed the codebase total over cap. Annotate the JSON-RPC params, body, metadata, and litellm_params dicts as dict[str, Any] so their values are typed, dropping the unknown-argument count back under the ceiling. No behavior change. * fix(a2a): guard localhost retry against a missing agent card handle_a2a_localhost_retry rewrote the card URL and called create_client with whatever agent_card it received. The caller resolves the card from the SDK client (Optional), so a None card reached set_agent_card_url and create_client, surfacing an opaque SDK error instead of a clear one. Add an early RuntimeError guard mirroring the httpx-client check, drop the now always-true card None-check on the stash line, and cover it with a regression test. * style(a2a): disable reportUnknownArgumentType in a2a-sdk boundary modules The lint env type-checks without the optional a2a-sdk/protobuf installed, so every call into the protobuf-generated compat conversions counts as an Unknown-typed argument and the new A2A code pushed the codebase reportUnknownArgumentType total over its ceiling. These three modules are the A2A SDK boundary; turn the rule off file-wide with a documented reason instead of scattering dozens of per-line ignores across every SDK call. * fix(a2a): tolerate unknown fields when lowering 1.0->0.3; align streaming trace id Two issues greptile flagged: version_convert: the 1.0->0.3 lowering paths (_send_result_to, _task_to, _stream_result_to) called ParseDict without ignore_unknown_fields=True, so a 1.0 upstream response carrying vendor extensions raised and best-effort fell back to passing the un-lowered 1.0 shape to a 0.3 client. Set the flag to match the agent-card path and every inbound path; unknown fields are now dropped and the result is correctly lowered. main.py: asend_message_streaming derived X-LiteLLM-Trace-Id from the JSON-RPC request id, unlike asend_message which uses the logging object's litellm_trace_id. Prefer the logging trace id (then request id, then a uuid) so streamed and non-streamed calls correlate under the same trace. Adds regression tests for both, including the stream-event lowering path. * style(a2a): apply ruff format to a2a protocol and proxy modules Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
2cf565ae28
|
test(batches): add 1:1 test file scaffold for batches component paths (#30529)
* test(batches): add 1:1 test file scaffold for batches component paths Co-authored-by: Cursor <cursoragent@cursor.com> * Add harness test for create batch endpoint * Add retrieve endpoint harness tests * Add list endpoint harness tests * Add cancel endpoint harness tests * Add cancel endpoint harness tests * Add test for litellm/batches/main.py * Add test for litellm/tests/test_litellm/batches/test_batch_utils.py * Add handler and transformation tests for all providers * Fix: run batches tests in cicd * fix(tests): remove azure/__init__.py that shadowed azure namespace package Adding __init__.py to tests/test_litellm/llms/azure/ caused pytest to insert tests/test_litellm/llms/ into sys.path[0], making our empty azure/ dir shadow the real azure-identity namespace package. Any test that patched azure.identity.* would then fail with AttributeError. * style(tests): apply ruff format to test_batch_utils.py Base migrated the formatter from black to ruff format (#31317); reformat the batches scaffold test file to match. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
a04321d2e1
|
test(videos): add 1:1 test file scaffold for videos component paths (#30631)
Keep only video test files and CI workflow entries; drop unrelated production code and non-video test changes from this branch. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
yucheng-berriai
|
f2d7cb152a |
refactor(proxy): type object_permission dict with ObjectPermissionDict
Replace bare Optional[dict] on the object_permission validator surfaces with a typed TypedDict mirror of LiteLLM_ObjectPermissionBase. The TypedDict shape matches the Pydantic model field-for-field and supports .get() and item assignment, so the mutation in _rewrite_object_permission_mcp_identifiers continues to work at runtime (TypedDict is a plain dict). Propagated through the surfaces this PR touches: _object_permission_to_dict, _validate_mcp_servers_for_key_update, validate_key_mcp_servers_against_team, validate_key_search_tools_against_team, validate_key_vector_stores_against _team, the five _extract_requested_* helpers, and the two _rewrite_object_permission_mcp_* mutators. attach_object_permission_to_dict, handle_update_object_permission_common, and _set_object_permission keep their wider dict typing because they handle the full key/team data_json, which is a superset of ObjectPermissionDict and pre-dates this PR. No behavior change. 373 tests pass; ruff strict + type discipline gates green. |
||
|
yucheng-berriai
|
2a5790fe55 |
fix(proxy): reject team-scoped object_permission on personal keys for non-admins
Non-admin callers could create or update a personal key (no team_id) with arbitrary access_group_ids, mcp_toolsets, vector_stores, or search_tools in object_permission. The server persisted the values without ownership validation; runtime authorization then trusted the IDs because they were stored on the key, allowing cross-tenant access to other teams' restricted models, MCP toolsets, and vector stores. The personal-key gate now mirrors the team-key path. enforce_member_can_assign_access_groups raises 403 for non-admin teamless callers. validate_key_mcp_servers_against_team rejects non-empty mcp_toolsets on personal non-admin keys. A new validate_key_vector_stores_against_team enforces the same rule for vector_stores. validate_key_search_tools_against_team gains the same gate for search_tools. The four validators are wired into /key/generate, /key/update, and /key/regenerate. Proxy admins keep their existing carve-out across all fields; team keys are unaffected. Endpoint-level regression coverage lives in tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py (six new parametrised cases through generate_key_fn and _validate_update_key_data) and helper-level coverage in tests/test_litellm/proxy/management_helpers/. Deleting any of the validator calls in _common_key_generation_helper or unmoving the enforce gate in _validate_update_key_data breaks the suite. |
||
|
ryan-crabbe-berri
|
234263fdda
|
fix(router): persist global retry_policy via /config/update (#29540)
* fix(router): persist global retry_policy via /config/update (LIT-3152)
The Admin UI Model Retry Settings tab POSTs
{router_settings: {retry_policy: {...}}} to /config/update, but the
field was dropped on two write-side layers so it never reached the
router. UpdateRouterConfig did not declare retry_policy, so
dict(exclude_none=True) stripped it before the DB upsert. And even when
fed directly, Router.update_settings had no "retry_policy" entry in
_allowed_settings, so the assignment was a silent no-op. The DB row
stayed at {"model_group_alias": {}}, llm_router.retry_policy stayed
None, and the UI fell back to defaultRetry = num_retries = 2 on refresh.
Declare retry_policy on UpdateRouterConfig as a plain dict, and add a
retry_policy branch to update_settings that coerces dict payloads to
RetryPolicy before setattr, mirroring Router.__init__. get_settings
already lists retry_policy, so reads work once writes land.
* fix(router): guard retry_policy type in update_settings
Mirror Router.__init__ semantics in update_settings: only assign
retry_policy when it is None or a RetryPolicy (after dict coercion).
Previously a non-dict, non-RetryPolicy value (e.g. a YAML typo like
retry_policy: 5 flowing through /config/update) was stored verbatim,
deferring the failure to request time in get_num_retries_from_retry_policy
instead of being dropped at write time.
* refactor(ui): harden Model Retry Settings flow and validate retry_policy at the boundary
Types UpdateRouterConfig.retry_policy as RetryPolicy and model_group_retry_policy as Dict[str, RetryPolicy] so /config/update validates the payload and rejects malformed counts instead of silently persisting them; the apply path in update_settings keeps coercing the stored dict back to RetryPolicy
Makes the Model Retry Settings tab the single owner of retry_policy and model_group_retry_policy so the generic Router Settings page no longer renders or writes them, replaces the fire-and-forget save with a react-query mutation that only shows the success toast after the write resolves, surfaces real errors, disables Save while in flight, and re-reads authoritative state on success, and sends both the global and per-group policies atomically so edits in the inactive scope are no longer dropped
Decouples the retry-scope selector from the All Models filter and defaults it to Global, seeds the displayed default from num_retries (falling back to 2), and gives per-group rows real inherit semantics so an empty input shows the global value as a placeholder with a Reset control, keeping 0 ("no retries") distinct from inheriting the global value
* fix(keys): align router_settings examples with typed RetryPolicy and resync UI artifacts
model_group_retry_policy is now Dict[str, RetryPolicy], so the {"max_retries": 5} sample in the key-generate test and the /key/generate and /key/update docstrings no longer validate; they now use a valid {"gpt-4": {"RateLimitErrorRetries": 5}} shape.
Regenerated eslint-metrics.json (no-explicit-any drifted 2027 -> 2026) and schema.d.ts (new RetryPolicy schema, retry_policy field, model_group_retry_policy value type) so the UI build and api-types-sync checks pass
* test(router): pin retry_policy persistence end to end (LIT-3152)
The existing retry_policy tests exercise UpdateRouterConfig and Router.update_settings in isolation, so they would all still pass if a regression flipped ConfigYAML.router_settings back to a loose dict or stopped add_deployment from applying the stored row. This drives the real handler chain an Admin UI save triggers: update_config writes the LiteLLM_Config row, the apply path forwards it to the live router, and get_config serializes it back, pinning retry_policy across persist, apply, and read-back.
* fix(teams): use valid model_group_retry_policy example in router_settings docstring
Same stale {"max_retries": 5} example the key endpoints carried; model_group_retry_policy maps a model group to a RetryPolicy, so the team /team/new and /team/update docs now show {"gpt-4": {"RateLimitErrorRetries": 5}}. Regenerated schema.d.ts to match.
* fix(ui): load retry settings via deferred fetch to satisfy set-state-in-effect
The Model Retry Settings effect called loadRetrySettings synchronously; eslint-plugin-react-hooks (react-hooks/set-state-in-effect) traces into it and flags the setState calls, failing frontend-lint. Split the loader into fetchRouterSettings + applyRouterSettings and run the fetch in an inline async IIFE with a cancellation flag, so state is applied in the post-await callback rather than on the effect's synchronous path. Behavior is unchanged and onSuccess still refreshes via loadRetrySettings.
* fix(ui): match CI rendering of RateLimitError 429 docstring in generated schema
gen:api run on a dev env (python 3.13 / newer fastapi) rendered the RateLimitError response description with 4-space indentation, but CI regenerates it with 8-space under its frozen python 3.12 toolchain, which is the canonical committed form. The Check UI API Types Sync job regenerates and diffs, so restore that block to the CI rendering; verified byte-identical to the pre-existing committed version.
* fix(ui): pin RateLimitError 429 docstring to CI's frozen schema rendering
Base #29619 regenerated schema.d.ts on a newer FastAPI that renders the RateLimitError response description at 4-space indent, but the Check UI API Types Sync job regenerates under the frozen python 3.12 toolchain, which renders 8-space. Merging base pulled in the 4-space form; restore the 8-space rendering so the generated types match what CI produces (verified byte-identical to the pre-#29619 committed form), which also corrects the base drift once this PR merges.
|
||
|
ryan-crabbe-berri
|
ac56320f26
|
fix(agents): show an agent's attached virtual key in the UI (#29619)
* fix(agents): show an agent's attached virtual key in the UI
The A2A agent detail view never surfaced which virtual key was attached to
an agent, so after assigning a key during agent creation there was no way to
see it again. Surface the attached key(s) in the agent detail view, derived
from the key table's agent_id foreign key the same way spend is already
joined into the agent response.
Backend adds an agent_id filter to /key/list (mirrors team_id) and enriches
GET /v1/agents and GET /v1/agents/{id} with a non-secret key summary (alias,
masked key_name, hashed token id). The frontend renders a Virtual Keys
section in the agent detail view that lists the agent's keys and links
through to the key detail, and the list view drops its fetch-500-keys-and-
filter-client-side workaround in favor of the enriched response. The orphaned
AgentCard and AgentCardGrid components, left behind when the agent list
switched from a card grid to a table, are removed
* fix(agents): redact attached virtual keys for non-admins
_attach_keys_to_agents joins keys onto the agent response by agent_id with
no caller scoping, but _redact_sensitive_agent_fields never cleared the new
keys field. A non-admin able to view an agent therefore received the alias,
masked name, and hashed token of every key attached to it, including keys
owned by other users or teams; the old client-side path used the scoped
key list, so this was a visibility regression. Clear keys in the redaction
path so only admins see attached-key metadata.
Adds an endpoint-level regression test asserting keys is populated for admins
and null for non-admins, and a list-view test covering the Active vs Needs
Setup badge that lost coverage when the agent card tests were removed.
* fix(agents): satisfy strict lint and resync key/list types
- use builtin list/dict generics in the new agent key helpers to stay
under the UP006 strict-rule ceiling
- swap @tremor/react for antd Typography in agent_virtual_keys (tremor is
being phased out; the new component was the only unsuppressed import)
- regenerate schema.d.ts so the /key/list agent_id query param is typed
* style(agents): prettier-format key hook test and agent_info
|
||
|
tin-berri
|
5963b9320f
|
feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31493)
* feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5) The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss. * feat(mcp): DualCache-backed token cache backend (step 1b §1.5) The cross-replica TokenCacheBackend implementation that plugs into the foundation's CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected; a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss. * feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5) The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis. * feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5) The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh), release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired / not-held so a cache blip causes an extra refresh, never a crash on the resolve path. * feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5) Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh coordinator when Redis is wired, falling back to the foundation's in-process defaults on a single replica. Layers the cross-replica path on top of the single-replica dispatch store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(mcp): refresh on lock-backend error instead of serving a stale token The cross-replica refresh coordinator elected refreshers with a boolean acquire: a Redis transport error was caught and returned as False, which is indistinguishable from "another worker holds the lock". On a total Redis outage every worker therefore took the wait-then-reread branch and served the still-expired token upstream (the upstream then 401s), even though the lock and coordinator docstrings claimed a Redis blip "degrades to an extra refresh". Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the coordinator can tell a busy holder from a dead backend, and refresh anyway on ERROR. This single-flight lock is a load optimization, not a correctness mutex, so failing open is correct: it degrades a lock-backend outage to the no-coordinator behavior (an extra refresh), never a stale bearer. Add a regression test asserting an acquire error refreshes rather than re-reading the expired token, and update the docstrings to match. * style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format * fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed The cross-replica coordinator's losers re-read the token the winner persisted. If the winner's refresh failed, the store still holds the expired token, so the loser re-read it and RefreshingTokenStore handed that expired bearer to the caller (the upstream then 401s) instead of the re-auth challenge the winner returned via None. Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a re-read that is still expired surfaces None so the arm challenges. This only affects the loser path; the winner's freshly refreshed token is returned directly by the coordinator and is unaffected. * fix(mcp): log per-user token decrypt failures at debug, matching v1 When a cached blob cannot be decrypted (e.g. after a salt or master-key rotation) the codec logged a full traceback at error level, since decrypt_value_helper defaults to exception_type=error. v1's MCPPerUserTokenCache passed exception_type=debug on the same path. The blob is ciphertext so this is log noise only, but matching v1 avoids error-level traceback spam on stale entries after a key rotation * fix(mcp): namespace the refresh lock key and fence its release with a token The Redis lock wrote its key through the raw client from init_async_client(), bypassing RedisCache's namespace, so two deployments sharing one Redis collided on mcp:refresh_lock:<user>:<server> for any overlapping (user, server) and a colliding deployment skipped the refresh and challenged its own users. The lock now runs every key through an injected namespace_key wired to RedisCache.check_and_fix_namespace, matching the namespace its token cache already uses release() also deleted the key unconditionally, so a holder whose lock PX-expired and was re-acquired by another worker could delete the new holder's lock and let a third worker run a duplicate refresh, recreating the rotating refresh_token race. acquire now writes a unique per-acquisition token generated by the coordinator and release deletes only when the key still holds that token, via a compare-and-delete Lua script Adds regression tests: release with a stale token is a no-op while the owner's release deletes; keys are namespaced before reaching Redis; the coordinator acquires and releases with the same token * fix(mcp): fail open when the per-user token cache delete errors DualCache swallows get/set errors internally but not delete, and the Redis layer underneath re-raises through its circuit breaker. So a Redis outage on the delete() path escaped CachedOAuthTokenStore.fetch()'s unauthorized branch (which deletes before returning None) and invalidate(), turning a cache blip into a 500 instead of the v1-style fallback. Catch in the backend so delete degrades to the TTL-bounded stale entry like get/set already do. * style(mcp): reformat outbound-credentials files to line-length 120 The merge from staging brought in ruff's line-length 120, but these two PR-authored files were still wrapped at the old width, so the diff-scoped ruff format --check in CI flagged them. Pure reformatting; no behavior change. * fix: harden mcp oauth redis refresh coordination * fix(mcp): make the per-user token cache backend airtight on boundary failures get/set now degrade a cache or codec failure to the safe value (miss / no-op) in the backend itself rather than relying on DualCache and decrypt_value_helper happening to swallow internally, matching delete() and v1's MCPPerUserTokenCache. This upholds the layer's boundary-failure-is-a-miss contract regardless of the injected collaborators, so a Redis outage or an undecryptable entry reads as a cache miss that re-reads the DB instead of a 500. Adds contract tests for the cache raising on get/set/delete and the codec raising on encode. * test(mcp): pin per-user cache get() to a miss when decrypt raises Greptile's out-of-diff repro had the decrypt reject a blob with ValueError (bad ciphertext after key rotation); cover that exact raise path, not just the decrypt-returns-None case, so get() is regression-locked to read it as a miss. * refactor(mcp): use frozen dataclasses for the trivial DI constructors Replace the hand-written self._<arg> = arg constructors on OAuthTokenCacheCodec, RedisRefreshCoordinator, RedisDistributedLock, and DualCacheTokenCacheBackend with frozen slotted dataclasses, matching the rest of this layer. Fields take the former parameter names so the constructor API (and the tests' keyword args) are unchanged; KW_ONLY preserves the keyword-only collaborators. * fix: serialize lazy per-user oauth store rebuild * fix(mcp): stop losers challenging mid-refresh by decoupling wait from lease TTL wait_timeout_seconds defaulted to the same 10s as lock_ttl_seconds, but the holder renews its lease while a slow token endpoint runs, so a loser waiting past 10s bailed and re-read the still-expired DB token, challenging the user even though a valid refresh was in flight. Bound the holder's renewal with a refresh budget so its lock-hold is finite, and set the loser's wait to outlast that budget (refresh_budget_seconds + one lease tail) so a loser only re-reads once the holder has finished or its bounded lease has lapsed, never mid-refresh. * fix: allow concurrent lazy OAuth fetches without Redis --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Yassin Kortam
|
b2e708d5ae
|
feat(prometheus): add per-team litellm_team_members_metric gauge (#31506)
Emit litellm_team_members_metric on every team member add and delete, labelled by team and team_alias and set to the team's authoritative member count. Because it is set from the current membership rather than incremented or decremented, it tracks the count up and down, never goes negative, and self-corrects on the next change after a proxy restart. Bulk member add is covered for free since it delegates to team_member_add, and the helper no-ops when the Prometheus callback is not registered. Resolves LIT-3082 |
||
|
Yassin Kortam
|
1883f975e2
|
fix(proxy/auth): honor user_api_key_cache_ttl for management-object cache writes (#31504)
general_settings.user_api_key_cache_ttl was ignored for every management-object write into user_api_key_cache. The configured value is propagated to the cache's default_in_memory_ttl at startup, but DualCache only applies that default when no explicit ttl kwarg is passed, and every management-object writer passed ttl=DEFAULT_MANAGEMENT_OBJECT_IN_MEMORY_CACHE_TTL (60s), which always won. So keys, teams, users, budgets, object permissions, vector stores, JWT user syncs and MCP caches all expired after 60s regardless of the setting. Adds get_management_object_ttl(cache) in user_api_key_cache.py, which returns the configured default_in_memory_ttl and falls back to the 60s constant only when no default is set, and routes every management-object writer through it. The helper takes a DualCache so it works at the many call sites that are typed UserApiKeyCache but exercised with a bare DualCache. Also covers the spend-update writeback in update_cache (async_set_cache_pipeline), which hardcoded ttl=60 on the same key/user/team objects and reset an active key's cache entry back to 60s on every priced request, so the configured TTL was never observed for keys receiving traffic. Resolves LIT-3338 |
||
|
Yassin Kortam
|
63490655ad
|
fix(pass_through): log pre-call guardrail blocks at WARNING, not ERROR with a traceback (#31500)
A pre-call guardrail block on a pass-through endpoint (e.g. OpenAI moderation flagging disallowed content) was logged at ERROR level with a full stack trace, even though the guardrail is working as designed and the client correctly receives the 4xx. The generic except in pass_through_request logged every exception via verbose_proxy_logger.exception(), so an intentional block produced scary traceback noise for operators tailing logs. Branch on the existing CustomGuardrail._is_guardrail_intervention classifier (the same predicate pipeline_executor already uses) so guardrail interventions log once at WARNING without a traceback while genuine failures keep their ERROR and traceback. This covers every guardrail that signals a block through the shared typed exceptions or an HTTPException 400, not just OpenAI moderation, and leaves the client-facing response unchanged. Resolves LIT-3538 |
||
|
Yassin Kortam
|
c33a7f8757
|
fix(proxy): cancel upstream LLM stream when client disconnects during time-to-first-token (#31499)
create_response buffers the first streamed chunk (to detect error-only streams) before handing the StreamingResponse to Starlette. Starlette only starts listening for client disconnects once it is serving that response, so a disconnect during a long time-to-first-token left the upstream LLM call running until the request timeout. This races the first-chunk fetch against an http.disconnect monitor; on disconnect it cancels the fetch, which propagates into async_streaming_data_generator's cleanup (records the 499 and closes the upstream stream), and returns a 499. Resolves LIT-3568 |
||
|
Yassin Kortam
|
437acc9b09
|
perf(proxy): bound event-loop blocking from oversized requests (#31497)
Skip token counting in Router._pre_call_checks when no deployment in the group declares max_input_tokens, and skip the full-body surrogate-repair regex in _read_request_body above a configurable size, raising the existing 400 immediately. Resolves LIT-3541 |
||
|
Yassin Kortam
|
80d3b69d9c
|
fix(pass-through): remove stale routes by key so the registry stops growing every reload (#31314)
The 30s add_deployment_job re-runs initialize_pass_through_endpoints, which
re-registers every config/DB pass-through endpoint. Endpoints without a
persisted id get a fresh uuid each cycle, so their route key
("{id}:{type}:{path}:{methods}") changes every reload. The stale-route cleanup
called remove_endpoint_routes(route_key), but that helper matches entries by
endpoint_id, so it never matched a route key and never deleted anything. The
registry grew by one entry per route per reload, turning the per-cycle cleanup
and the per-request is_registered_pass_through_route scan into a CPU sink that
eventually pins a core and slows every endpoint.
Pop the stale key from the registry directly in O(1). openai_routes is left
alone: its append is path-deduped and the path is still owned by the live
endpoint re-registered under a new id in the same cycle.
Resolves PERF-13
|
||
|
tin-berri
|
4b398ef6d4
|
feat(mcp): migrate authorization_code MCP to the v2 resolver (single-replica) [1/2] (#31473)
* feat(mcp): implement the authorization_code resolver arm
Resolve a user's authorization_code token through the injected OAuthTokenStore: present ->
Authorization: Bearer <access_token>; absent -> the RFC 9728 WWW-Authenticate OAuth challenge;
store unavailable -> the same challenge (not a 500), since a transient outage is not a definite
absence. UpstreamCredentialProvider gains the oauth_token_store collaborator (fail-closed null
default); per-subject isolation comes from keying the fetch on subject_id. Not live until
to_server_spec maps authorization_code and a v1-backed token source is wired (next steps).
* feat(mcp): v1-backed OAuth token source for authorization_code
V1PerUserTokenStore reads the user's stored access token through v1's mcp_per_user_token_cache
(Redis-backed, encrypted) and wraps it in an OAuthToken. v1 holds only the access token (its
cache TTL is the lifetime), so no expires_at/refresh_token yet; the v2 cache holds it for its
default TTL and the OAuth challenge drives re-auth once v1's cache drops it. Additive: nothing
wires it yet, so no behavior change. Step 1b swaps it for a v2-native token store behind the
OAuthTokenStore seam.
* style(mcp): modern type annotations in the authorization_code arm and source
* refactor(mcp): share v1's OAuth egress core; make V1PerUserTokenStore refresh-capable
Extract v1's per-user OAuth egress (Redis cache, else DB read with the refresh_token grant, then
re-cache) from _get_user_oauth_extra_headers_from_db into resolve_user_oauth_access_token in db.py;
the v1 header builder is now a thin wrapper over it and its callers are unchanged.
V1PerUserTokenStore (the v2 OAuthTokenStore adapter) resolves through that same core via an injected
server lookup, so the authorization_code arm injects exactly the token v1 would, with the same silent
refresh, rather than a Redis-only read that can never refresh. One resolution implementation, two thin
adapters (header dict and OAuthToken). Behavior-preserving: the existing v1 egress tests pass
unchanged, and the arm is not wired into the live path yet (that lands with to_server_spec + the
manager).
* feat(mcp): route oauth2 per-user (authorization_code) servers through the v2 resolver
to_server_spec maps an oauth2 server to AuthorizationCodeConfig when it relies on per-user tokens
(needs_user_oauth_token and not delegate_auth_to_upstream); client_credentials (M2M), delegated
upstream OAuth, token exchange, and SigV4 still defer to v1. The manager injects V1PerUserTokenStore
(resolving through v1's shared egress core) into the credential provider. The v2 path is live but
still defers to a token v1 places in extra_headers; the cutover that makes v1 step aside lands next,
alongside the unified challenge.
* feat(mcp): per-server fail-closed OAuth challenge at the v2 egress
When an authorization_code server has no usable per-user token, the arm returns a semantic
unauthorized and the graft builds the 401 where the full MCPServer is in hand: a relative,
per-server RFC 9728 resource_metadata pointer (/.well-known/oauth-protected-resource/mcp/{name})
that names the server's own authorization server, instead of the resolver's earlier root pointer
which resolved to the gateway's generic PRM. Relative, so it is correct behind a reverse proxy
without request context. The listing-phase 401 still emits the RFC 8414 authorization_uri form;
both now target the same server, so the remaining difference is cosmetic and unifies in a later PR.
* feat(mcp): cut the call_tool egress over to v2 for authorization_code servers
_resolve_oauth2_headers_for_tool_call steps aside (builds no header) when to_server_spec maps the
server, so the v2 resolver drives the token-present case instead of being shadowed by a token v1
places in extra_headers. Non-migrated oauth2 (delegate, client_credentials) and BYOK still build
their header on v1. With this, v2 owns the authorization_code egress end to end: inject the
refreshed per-user token when present, raise the per-server fail-closed 401 when absent.
* feat(mcp): cut the tools/list connection over to v2 for authorization_code servers
The listing connection's per-user OAuth header is no longer built by v1 for migrated servers; the
v2 resolver drives it at connect time, ending the double-resolution where v1 built the token into
extra_headers and the v2 graft then deferred to it. Safe because the preemptive 401 (in the
streamable-http and SSE handlers) already challenges a missing token before the listing connection
runs, so the connection is only reached with a token present. Non-migrated oauth2 (delegate) and
the rest still build their header on v1. With this, resolve_credentials' result is honored on every
authorization_code upstream path: tool calls and listing.
* feat(mcp): route the preemptive 401 existence check through the v2 resolver
The discovery-phase 401 no longer calls v1's _get_user_oauth_extra_headers_from_db to decide
whether a migrated server has a token; it asks the v2 resolver via a new has_user_oauth_token
manager method (to_server_spec + to_subject + resolve_credentials, Ok means a token exists). With
this, every authorization_code resolution runs through the v2 resolver: the call_tool egress, the
listing connection, and the discovery challenge. Delegate servers short-circuit before the check
(the client completes PKCE with the upstream). The challenge itself still emits the RFC 8414
authorization_uri form; the format unification stays a follow-up.
* refactor(mcp): extract the authorization_code arm into a helper
Mirror the api_key arm's structure: the inline AuthorizationCodeConfig body moves into
_authorization_code(subject, server), keeping resolve_credentials a flat one-line-per-arm dispatch.
The helper is annotated with the concrete StaticHeaderAuth it returns rather than the abstract
httpx.Auth (which api_key uses) because a new method carrying the unresolved httpx.Auth return
would add reportUnknownMemberType; the concrete type is both precise and budget-neutral.
* fix(mcp): emit the canonical WWW-Authenticate header name in the OAuth challenge
raise_user_oauth_challenge emitted the header lowercase while the sibling raise_public and every
resource_metadata (RFC 9728) emitter use the canonical WWW-Authenticate; align it. HTTP header names
are case-insensitive on the wire so this is cosmetic for compliant clients, but it keeps the challenge
builders consistent and matches RFC 6750.
* feat(mcp): v2-native per-user token read store (step 1b inner store)
Reads the user's persisted authorization_code credential and returns a typed OAuthToken (access
token, epoch expiry, refresh token), validating the decoded blob at this boundary so no Any leaks
past it. The raw inner store that RefreshingTokenStore/CachedOAuthTokenStore wrap; the DB read +
decode collaborator is injected so it stays testable. Not yet wired - V1PerUserTokenStore is still
the composition-root store until the refresher and cross-worker cache land.
* feat(mcp): v2-native authorization_code token refresher (step 1b)
The refresh_token grant for the authorization_code mode: POSTs the RFC 6749 refresh_token grant to
the server's token endpoint, persists the rotated triple, and returns the new typed OAuthToken for
RefreshingTokenStore to cache. HTTP post and persist are injected so the grant + response parsing
are testable without a live IdP/DB. Also extends the TokenRefresher seam with (user_id, server_id),
which the foundation's refresh(token) lacked but the grant (server config) and persist (key) need.
* feat(mcp): wire the v2-native per-user OAuth store into the resolver (step 1b piece 4)
Assemble Cached(Refreshing(V2PerUserTokenStore)) at the composition root and replace
V1PerUserTokenStore in mcp_server_manager. The chain is built lazily on first fetch (its cache/DB/
Redis collaborators are LiteLLM globals not ready at import); when Redis is wired it uses the
cross-replica path (DualCache cache + SET NX PX coordinator), else the in-process defaults. The DB
read, refresh-grant POST, and persist acquire their globals per call like v1. authorization_code
resolution now reads/refreshes through the v2-native lifecycle, not v1's core.
* refactor(mcp): delete the unwired V1PerUserTokenStore adapter (step 1b piece 5)
Piece 4 replaced V1PerUserTokenStore with the v2-native chain at the composition root, leaving the
adapter with no callers, so remove it and its test. The shared v1 read/refresh core
(resolve_user_oauth_access_token and friends) stays - delegate's egress in server.py still uses it -
and comes out with the delegate migration.
* fix(mcp): green CI for authz_code dispatch (format + UTC expiry + v2-seam tests)
- ruff format per_user_oauth_store.py (clears the lint check)
- v2_token_store._iso_to_epoch: anchor a tz-naive expiry to UTC before
.timestamp(), matching v1's db.py _remaining_token_seconds (Greptile P1) so a
non-UTC host doesn't read the expiry as local time and skew refresh timing
- test_mcp_stale_session: repoint the 3 discovery tests off the removed v1
_get_user_oauth_extra_headers_from_db onto the v2 has_user_oauth_token seam;
the delegate test now asserts the existence check is never consulted (delegate
short-circuits to the resource_metadata 401 before any token lookup)
- test_mcp_server_manager: repoint test_deferred_mode_uses_v1_auth_value at M2M
(oauth2 client_credentials), which is still a deferred mode, since per-user
oauth2 (authorization_code) now routes to the v2 resolver
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): caller Authorization must not override the stored per-user OAuth token
A caller with a valid x-litellm-api-key could include their own
"Authorization: Bearer <chosen>" header and have the proxy execute tools against
that bearer instead of the user's stored OAuth credential. For a v2-migrated
authorization_code server the caller's Authorization was seeded into
extra_headers, and the graft's apply-if-absent then dropped the resolved
per-user token in its favor. v1 prevented this by overwriting a stale client
Authorization with the stored token; this restores that precedence on both
egress paths (connect + call_tool).
- _should_strip_caller_authorization: also strip for migrated per-user OAuth
(authorization_code) servers - the v2 resolver injects the stored token, so a
caller-forwarded Authorization must not be forwarded upstream. Delegate /
pass-through (to_server_spec is None) keep forwarding the caller's bearer.
- both seed sites (_prepare_mcp_server_headers, _call_regular_mcp_tool) drop only
the Authorization from the caller's oauth2_headers (via _without_authorization),
keeping any other forwarded header and any hook/static Authorization (which
still wins, as in v1).
- regression test for the call_tool path; updated the two tests that asserted the
old (vulnerable) forwarding to assert the secure behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): preserve recorded OAuth scopes across authorization_code refresh
When a refresh response omits `scope` (RFC 6749 §5.1, where omission means unchanged), the v2 refresher persisted scopes=None and overwrote the user's recorded grant. v1 carried the prior scopes forward via `or cred.get("scopes")`; the v2 path lost that because OAuthToken did not model scopes
OAuthToken now carries scopes, V2PerUserTokenStore populates them on read, and AuthorizationCodeRefresher carries them forward for both the persisted write and the returned/cached token, so repeated refreshes do not erode them. A present `scope` in the response still replaces the prior grant
Adds regression tests: a refresh omitting `scope` preserves the prior scopes, and a present `scope` overrides them
* fix(mcp): keep user token in authorization_code tools preview
After to_server_spec maps oauth2 onto the v2 resolver, the interactive tools preview for an unsaved authorization_code server read the per-user token store, found nothing, and fail-closed with a 401, so the create/test tab could no longer list tools
The preview now routes the just-authorized token (forwarded in oauth2_headers) through mcp_auth_header, so _create_mcp_client takes the per-request-override v1 path and uses it directly, matching v1's preview. Gated to the v2-mapped oauth2 case; M2M, delegate/passthrough, and token-exchange keep their existing preview path
Adds tests: interactive oauth routes the forwarded token to mcp_auth_header, M2M and token-exchange do not
* fix(mcp): stop caller-supplied auth from overriding stored authorization_code tokens
A caller-supplied per-request override (mcp_auth_header / x-mcp-auth / x-mcp-<alias>-authorization) disabled the v2 resolver in _create_mcp_client for any spec, so an authenticated user with a stored authorization_code token could force an arbitrary upstream bearer and bypass the stored credential and its save-time validation. _create_mcp_client now keeps the v2 spec for authorization_code and ignores the override; other modes keep the client-side-credentials override
The create/test tools preview no longer relies on that override path. It resolves the just-authorized, not-yet-persisted token through the v2 resolver via a one-shot PresentedOAuthTokenStore passed as cred_provider - the same path runtime uses for the stored token - so preview and runtime resolve identically. This replaces the mcp_auth_header routing added earlier
Adds tests: a caller override cannot bypass the v2 resolver for authorization_code; the interactive preview resolves via the presented store rather than a caller header; M2M and token-exchange build no presented provider
* feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (#31474)
* feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5)
The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so
encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token
and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived
refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token
always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss.
* feat(mcp): DualCache-backed token cache backend (step 1b §1.5)
The cross-replica TokenCacheBackend implementation that plugs into the foundation's
CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's
shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a
token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected;
a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss.
* feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5)
The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET
NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the
token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The
lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read
and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis
SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis.
* feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5)
The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic
SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh),
release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's
RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired /
not-held so a cache blip causes an extra refresh, never a crash on the resolve path.
* feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5)
Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh
coordinator when Redis is wired, falling back to the foundation's in-process defaults on a
single replica. Layers the cross-replica path on top of the single-replica dispatch store.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(mcp): refresh on lock-backend error instead of serving a stale token
The cross-replica refresh coordinator elected refreshers with a boolean acquire:
a Redis transport error was caught and returned as False, which is
indistinguishable from "another worker holds the lock". On a total Redis
outage every worker therefore took the wait-then-reread branch and served the
still-expired token upstream (the upstream then 401s), even though the lock and
coordinator docstrings claimed a Redis blip "degrades to an extra refresh".
Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the
coordinator can tell a busy holder from a dead backend, and refresh anyway on
ERROR. This single-flight lock is a load optimization, not a correctness mutex,
so failing open is correct: it degrades a lock-backend outage to the
no-coordinator behavior (an extra refresh), never a stale bearer.
Add a regression test asserting an acquire error refreshes rather than
re-reading the expired token, and update the docstrings to match.
* style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format
* fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed
The cross-replica coordinator's losers re-read the token the winner persisted.
If the winner's refresh failed, the store still holds the expired token, so the
loser re-read it and RefreshingTokenStore handed that expired bearer to the
caller (the upstream then 401s) instead of the re-auth challenge the winner
returned via None.
Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a
re-read that is still expired surfaces None so the arm challenges. This only
affects the loser path; the winner's freshly refreshed token is returned
directly by the coordinator and is unaffected.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "feat(mcp): cross-replica single-flight refresh for the v2 per-user OA…" (#31492)
This reverts commit
|
||
|
yuneng-jiang
|
e4aedb0342
|
Merge pull request #31391 from BerriAI/litellm_multipart_file_upload
fix(passthrough): forward all multipart files with repeated field names |
||
|
Krrish Dholakia
|
99b1a323c1
|
feat(guardrails): add headroom guardrail for message compression (#31407)
* feat(guardrails): add headroom guardrail for message compression Adds a headroom guardrail that compresses request messages via POST /v1/compress before they reach the LLM. The guardrail implements apply_guardrail so it runs on the unified guardrail path; it receives pre-built structured_messages (OpenAI format) from the translation layer, calls the headroom compression service, and returns the compressed messages as structured_messages. Set x-headroom-bypass: true on the request to skip compression. Also adds structured_messages write-back support to the OpenAI and Anthropic translation handlers: when apply_guardrail returns structured_messages, those are written to data["messages"] directly (OpenAI) or reverse-translated via anthropic_messages_pt (Anthropic) instead of falling through to the existing text-patch path. This is a prerequisite for any guardrail that needs to replace the full message list rather than patch individual text spans. * fix(guardrails/headroom): add @log_guardrail_information to populate guardrail_information in spend logs * style: fix ruff format violations * fix(lint): replace deprecated typing aliases with builtin generics (UP006/UP037) * fix(guardrails): only write back structured_messages when guardrail actually changed them * fix(guardrails/headroom): raise 502 when compression returns empty message list * fix(guardrails/headroom): catch transport errors and fix stale debug log * fix(guardrails/anthropic): strip system messages before anthropic_messages_pt reverse-translation * fix(guardrails/anthropic): strip cache_control from thinking blocks after write-back * debug(headroom): add INFO logging to trace guardrail execution * debug(headroom): use print() for immediate visibility * debug(headroom): print request_data keys to diagnose metadata dict mismatch * fix(guardrails/anthropic): propagate guardrail info to logging_obj.metadata for spend log * fix: use model_call_details litellm_params metadata on Logging object * fix(guardrails/anthropic): write guardrail info to litellm_params attr not model_call_details copy * fix: read slg_info from litellm_metadata when metadata key absent * fix: write slg_info to both litellm_params attr and model_call_details copy * chore: remove debug prints; fix now verified end-to-end * refactor(guardrails): move spend-log sync to shared helper in custom_guardrail.py - Add _sync_guardrail_info_to_logging_obj in custom_guardrail.py; call it from both async and sync wrappers in @log_guardrail_information, fixing guardrail_information=null in spend logs for all passthrough routes (/v1/messages, /v1/responses, etc.) in one place - Remove the 35-line inline sync block from the anthropic translation handler - Wrap response.json() in try/except in headroom.py to 502 on HTML/truncated responses - Drop redundant headers.get(BYPASS_HEADER.lower()) — header key already lowercase - Add regression tests for _sync_guardrail_info_to_logging_obj * fix(lint): reduce _sync_guardrail_info_to_logging_obj complexity below C901 threshold * fix(lint): simplify _sync_guardrail_info_to_logging_obj to reduce McCabe complexity * fix(lint): extract _append_slg_to_litellm_params to reduce McCabe complexity * fix(lint): extract _write_back_structured_messages to reduce process_input_messages complexity |
||
|
Mateo Wang
|
4157f3b580
|
fix(passthrough): schedule spend logging via durable logging worker (#31485)
Pass-through success logging was scheduled with a bare asyncio.create_task whose return value was discarded, for non-streaming HTTP, streaming, and the vertex live websocket paths. The event loop keeps only a weak reference to such tasks, so under GC or load the task can be collected before it finishes writing the SpendLogs row; a request then returns 2xx to the caller yet never produces a costed spend log. This is the most likely cause of the flaky vertex passthrough e2e test and a rare real source of unbilled pass-through spend. Route these coroutines through GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue instead, matching how the SDK completion path already enqueues async logging. The worker holds a strong reference in its _running_tasks set and drains on shutdown via flush/stop/clear_queue and the atexit handler, so the write can no longer be dropped mid-flight. |
||
|
tin-berri
|
2e69708ef8
|
feat(mcp): shared OAuth token foundation - challenge, store seam, expiry-aware cache, single-flight refresh (#31275)
* feat(mcp): let CredError.of_unauthorized carry a 401 challenge The unauthorized case becomes a structured Unauthorized (detail + optional WWW-Authenticate header + optional structured body) instead of a bare string, and raise_public emits the header and body when present. This lets a mode reproduce a rich 401 challenge (e.g. BYOK's provisioning prompt) through the generic resolver edge. of_unauthorized's new params are keyword-only and default to None, so existing callers and the summary string are unchanged. * fix(mcp): make Unauthorized a frozen dataclass to keep the type budget flat CredError's unauthorized payload was a pydantic BaseModel, whose base resolves as unknown in this repo's basedpyright (every model in the file trips reportUntypedBaseClass plus an unknown model_config), so the tagged-union case read as unknown and the public edge's challenge access added reportUnknownMemberType errors over the per-rule ceiling. A frozen dataclass is fully typed here, so error.unauthorized resolves directly with no cast or accessor and the per-rule basedpyright counts match base. * feat(mcp): OAuth token store seam + expiry-aware cache for authorization_code Lay the foundation for the authorization_code resolver arm: OAuthToken (access_token, expires_at, refresh_token), the OAuthTokenStore Protocol seam, TokenStoreUnavailable for outages, and CachedOAuthTokenStore, an expiry-aware cache that serves a token only while unexpired, caches the "not authorized" None for a default TTL, and propagates a store outage without caching it. Mirrors the BYOK store/cache pattern, adapted for tokens. Refresh and distributed single-flight are deferred to the hardening step. * feat(mcp): proactive token refresh with self-cleaning single-flight Add TokenRefresher (a mode-supplied seam: mint a fresh token from an expired one and persist it) and RefreshingTokenStore: when the stored token is near expiry, the first caller refreshes while concurrent callers await the same in-flight task and share its result, so the IdP is not stampeded. The task self-cleans (a done-callback drops its entry), so the map is bounded by in-flight refreshes rather than by distinct users/servers, and is detached from the caller so a cancelled caller does not abort the refresh. An expired token the refresher cannot renew surfaces as None so the arm challenges, never a stale bearer; it composes under CachedOAuthTokenStore. OAuthToken's repr masks the access/refresh tokens so a stray log cannot leak them. Cross-replica single-flight (Redis) and reactive-401 refresh are the later distributed hardening. * style(mcp): modern type annotations (dict/tuple/X | None) + sorted imports in the token modules * refactor(mcp): FIFO cache eviction, fix stale single-flight comment + refresh_token docstring * refactor(mcp): cache positive tokens only, matching v1 (no negative caching) CachedOAuthTokenStore no longer caches the "not authorized" None result; every miss re-reads the inner store. v1's per-user token cache never caches misses, so a token written by the OAuth flow is visible on the next request without an invalidation hook, and uniformly across replicas since the in-process cache holds no stale None to clear. invalidate() now only covers rotation or revocation of a cached token. Negative caching (with distributed invalidation) can return later if a slow DB-backed v2-native source makes per-miss reads expensive. * fix(mcp): default OAuth expiry skew to 60s, the industry standard The proactive token-refresh / cache-expiry buffer defaulted to 30s, which is an outlier among OAuth clients. Spring Security uses 60s as both its JWT clock-skew tolerance and its refresh buffer, and 60s sits inside RFC 7519's "a few minutes" leeway while preserving nearly all of a typical token's life; 30s was untested, so pin the default with two boundary-probe regression tests. * refactor(mcp): thread user_id/server_id through the TokenRefresher seam The refresh seam took only the OAuthToken, but a refresher needs the server's config (token endpoint, client credentials, scopes) to run the grant and the (user_id, server_id) key to persist the minted token, neither of which is derivable from the token. Widen TokenRefresher.refresh to (user_id, server_id, token) and pass them through from RefreshingTokenStore so each stacked mode PR plugs into the final seam rather than forcing a later signature change across the stack. * feat(mcp): inject cache-backend and refresh-coordinator seams (cross-replica token caching) Make CachedOAuthTokenStore's storage and RefreshingTokenStore's single-flight injectable so a cross-replica deployment can back them with Redis without touching the resolver. The defaults preserve today's behavior exactly: InMemoryTokenCacheBackend (the bounded per-process dict) and InProcessRefreshCoordinator (the asyncio single-flight). A distributed deployment injects a shared DualCache-backed backend and a SET NX PX coordinator. invalidate() is now async (the backend may be). The cache stores via the backend with a TTL derived from the token's expiry; the coordinator threads a reread callback for the cross-replica case (losers re-read the persisted token) that the in-process default ignores. * fix: reread oauth token before refresh --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Yassin Kortam
|
c14329128b
|
fix(guardrails): match policy-pipeline block response to direct guardrail attachment (#31421)
When a guardrail blocked a request through a flow-builder policy pipeline, the proxy discarded the guardrail's own exception and synthesized a generic guardrail_pipeline_error response, so the same guardrail produced a different HTTP response and trace span depending on whether it was attached directly or via a policy. The pipeline now carries the guardrail's original exception and re-raises it verbatim on block, enriching it with the blocking guardrail's name and mode exactly as the direct path does, so the two attachment methods are indistinguishable to clients and tracing. The generic pipeline error remains only as a fallback for blocks with no underlying exception (e.g. a guardrail that could not be found). Resolves LIT-4041 |
||
|
Yassin Kortam
|
ce658367a4
|
fix(auth): cache auth-path team object under canonical team_id key (#31418)
The auth builder cached the team object under the raw `valid_token.team_id`,
while `get_team_object`, `_cache_team_object`, and `_update_team_cache` all read
and write under `team_id:{id}`. The raw-key write was therefore never served
back, and on a non-team (personal) key, whose team_id is None, the original
unguarded version passed a None key straight to the cache layer; the in-memory
cache tolerates None keys but Redis rejects them with a NoneType key error, so
with `enable_redis_auth_cache: true` the team object never reached the L2 cache
and every request fell back to Postgres.
Write under the canonical `team_id:{id}` key, keeping the existing guard that
skips the write when team_id is None. Add a regression test that drives the real
auth builder for a team-scoped key against an in-memory cache and asserts the
team object is served back under `team_id:{id}` and never under the raw team_id
or a None key.
Resolves LIT-4000
|
||
|
Yassin Kortam
|
f2fa23b0ec
|
fix(guardrails): instrument during-call and post-call guardrail latency (#31414)
litellm_guardrail_latency_seconds was only emitted for pre-call guardrails. during_call_hook and post_call_success_hook ran guardrails without recording any latency, so during-call and post-call guardrail time was invisible in the metric and leaked into litellm_overhead_latency_metric, making the documented "subtract guardrail latency from overhead" workaround under-report total guardrail time. Extract the find-the-PrometheusLogger-and-record step into _emit_guardrail_metrics and add _run_guardrail_with_metrics, a single wrapper that times a guardrail coroutine, classifies its outcome (success / intervened / error), enriches any raised HTTPException, and records the latency under the given hook_type. Route the pre-call emit, during_call_hook, and post_call_success_hook through it so every guardrail phase contributes to the metric the same way. Resolves LIT-3999 |
||
|
Yassin Kortam
|
7209e139d6
|
fix(spend): fold logs-tab total into the page query to avoid a separate COUNT(*) (#31423)
The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) ran a standalone SELECT COUNT(*) before the page query to compute total_pages. On sharded engines like YugabyteDB a COUNT(*) is a distributed RPC that contacts every tablet leader and aggregates partial results regardless of row count, so it hits the distributed RPC timeout and the logs tab 500s even on a one-minute window with a couple of rows. The startTime range cannot prune tablets because rows hash to tablets on request_id, not startTime. Fold the count into the same scan as the page data with COUNT(*) OVER () and read total off the returned rows, dropping the helper column before serialisation. One distributed scan per page load instead of two; the response shape is unchanged. An empty page carries no count row, in which case the total is zero. Resolves LIT-4027 |
||
|
Yassin Kortam
|
f55d13ebba
|
fix(team): persist budget_duration on /team/member_add member budgets (#31443)
/team/member_add could not set budget_duration on an individual member budget. add_new_member created the budget row with only max_budget and allowed_models, and TeamMemberAddRequest had no budget_duration field, so a member added with an explicit per-member budget while the team ran a recurring member budget got a lifetime cap instead of a recurring allowance. Thread budget_duration from TeamMemberAddRequest through _process_team_members into add_new_member, and pull the member-budget resolution into a helper that writes budget_duration plus a computed budget_reset_at. When only a budget_duration is supplied and the team has a default member budget, the default is cloned and its reset window overridden so the member keeps the default's max_budget rather than becoming uncapped; a duration with no team default creates a window-only budget. Invalid durations are rejected with a 400 before any DB write, symmetric with /team/member_update. The available-team self-join bypass only grants the ability to join, so reject per-member budget and model controls (max_budget_in_team, budget_duration, allowed_models) for non-admin self-join callers in _validate_team_member_add_permissions, before any DB write. Otherwise a self-joining non-admin could set their own cap, reset window, or model scope past the team default; admins, team admins, and org admins are unaffected and a clean self-join still inherits the team default budget. Resolves LIT-4052 |
||
|
yucheng-berri
|
e99151bb95
|
feat(guardrails): make the Generic Guardrail resilient to built-in tools and errors (adopted from #31286) (#31461)
* fix(guardrails): stop Generic Guardrail API 500 on built-in tools
Requests carrying built-in tools (code_interpreter, file_search, ...) crashed
the Generic Guardrail with a 500. GenericGuardrailAPIRequest.tools validated
each tool against ChatCompletionToolParam, whose base TypedDict requires a
function block, so a tool like {"type": "code_interpreter"} raised a Pydantic
ValidationError before the request was ever sent.
Type the field with a permissive GuardrailToolParam model (type required,
extra=allow) so built-in tools validate and their config is forwarded to the
guardrail intact instead of being stripped.
* feat(guardrails): add complete fail-open (fail_on_error) to Generic Guardrail
The Generic Guardrail already honored unreachable_fallback, which fails open
only on network-unreachable errors. This wires up the existing generic
fail_on_error config (so far implemented only by Model Armor) so that
fail_on_error=false degrades any guardrail error to a critical-log warning and
lets the request proceed as if the guardrail were absent.
Only a valid guardrail response can act: a parsed BLOCKED decision still raises,
while endpoint errors, malformed responses, and internal serialization or
validation errors all fall through when fail_on_error=false. To cover that last
class, the request construction now runs inside the protected block, so the kind
of validation error that previously surfaced as a 500 is caught here too.
Defaults to true (fail closed), matching today's behavior; turning it off is an
explicit availability-over-security choice and is logged at critical level on
every bypass.
* test(guardrails): cover fail_on_error on the response path
The existing fail_on_error tests all drive the request path. Add response-path
(input_type=response) coverage: an endpoint error proceeds unchanged under
fail_on_error=false, and a valid BLOCKED decision still raises. Guards against a
future regression that special-cases input_type in the error handling.
* style(guardrails): black-format the fail-open guard expression
CI runs black (line-length 88) over litellm/; the unreachable_fail_open
assignment exceeded it. Wrap it to satisfy the formatter.
* fix(guardrails): validate tools into GuardrailToolParam at the call site
Changing the request field to List[GuardrailToolParam] left the construction
passing List[ChatCompletionToolParam] (list is invariant), which tripped the
basedpyright reportArgumentType budget gate. Validate each tool explicitly,
which is what Pydantic did implicitly, so the types line up with no Any or
suppression and the serialized payload is unchanged.
* fix(guardrails): make fail-open log message accurate for non-network errors
The fail-open path is now shared by fail_on_error, so it fires for any guardrail
error, not just unreachability. The log said 'unreachable' even for an HTTP 400
or a malformed response; reword to 'error' (the status code and exception are
already logged). Addresses the Greptile review's only finding.
* fix(guardrails): align GenericGuardrailAPIResponse.tools with GuardrailToolParam
Greptile flagged that the request side moved to GuardrailToolParam but the
response side still annotated tools as List[ChatCompletionToolParam], which
mandates a function block and contradicts the new built-in-tools support.
Update the response annotation (and the now-unused import) so the two sides
agree. Runtime is unchanged; from_dict stores the raw dicts and the only
consumer assigns through to GenericGuardrailAPIInputs without inspecting
the elements.
---------
Co-authored-by: Itay Ovadia <itay@sun.security>
|
||
|
Sameer Kankute
|
133da06aa3
|
chore: litellm oss staging (#31185)
* fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped
The Total Tokens Over Time and Total Requests Over Time AreaCharts on the
Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow
once totals pass the hundred-million mark — leading digits of labels like
"100.00M" / "4500.00M" got clipped against the chart edge. The requests
chart was worse: it formatted with toLocaleString(), so billion-scale
request counts produced "1,000,000,000" (13 chars) and overflowed
immediately.
Fix in two places so neither alone has to carry the whole margin:
- activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and
switch the requests chart to the shared valueFormatter so it uses the
same compact k/M/B suffixes as the tokens chart.
- value_formatters.tsx: add a >= 1e9 branch to valueFormatter /
valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping
every formatted label at most 7 chars.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* docs(readme): add Deploy on AWS/GCP with Terraform section
Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.
Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): add 1-click deploy buttons for AWS + GCP
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.
AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): move AWS + GCP deploy buttons next to Render button
* docs(readme): unify deploy button sizes and badge styles
* docs(readme): bump deploy button height to 48 to match Render/Railway
* docs(readme): bump AWS/GCP badge height to compensate for SVG padding
* docs(readme): bump AWS/GCP badge height to 72
* docs(readme): bump AWS/GCP badge height to 84
* fix(readme): make deploy buttons same height (48px)
https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc
* docs(readme): flag GCP project ID substitution in image_registry
* docs(readme): equalize deploy button heights and fix Cloud Shell button font
GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.
Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
* docs(readme): collapse Railway deploy anchor to a single line
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
* Add Claude Fable 5 cost map entries as a data-only hotfix
Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano
Three bugs in model_prices_and_context_window.json:
1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens
were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K
max output, but the values were set as max_input=128000,
max_tokens=272000. This caused token limit errors when sending
prompts over 128K tokens to GPT-5 Pro.
2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was
272000, but GPT-5.4 Mini shares the same 1,050,000 token context
window as GPT-5.4. This was inconsistent with the azure/ variants
which already correctly had 1,050,000.
3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini,
max_input_tokens was 272000 instead of 1,050,000.
Source: OpenAI model documentation and contextwindows.dev which
aggregates official context window sizes.
Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini
should be 400K; their 272K values are correct per OpenAI docs)
* fix: also correct max_output_tokens for gpt-5-pro (272000→128000)
Per reviewer feedback, max_output_tokens was left at 272000 while
max_tokens was corrected to 128000, causing an internal inconsistency.
Both should be 128000 per OpenAI docs.
* fix(cost): price gpt-image generated output tokens as image tokens (#31147)
The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return
usage with no output token breakdown — litellm's `ImageUsage` has no
`output_tokens_details` field — so generated-image OUTPUT tokens were priced at
the text rate (`output_cost_per_token`) instead of the image rate
(`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x
undercount on the dominant cost component (image output is ~74% of spend). This
also affects azure gpt-image, which shares this calculator.
The OpenAI gpt-image cost calculator re-implemented usage handling instead of
reusing `calculate_image_response_cost_from_usage`, the shared helper that
azure_ai/gemini/vertex_ai already use. That helper classifies generated output
tokens as image tokens when the provider does not itemize output, and splits
text/image when it does.
Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage`
(pre-transformed chat Usage objects are still costed directly). Adds a regression
test for the no-breakdown ImageUsage case (gpt-image-2).
* fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098)
A bare application-inference-profile ARN passed as bedrock/arn:... fell
through to the invoke route, which cannot derive a provider from the
opaque profile id and raised 'Unknown provider=None'. The converse route
needs no provider, so detect these ARNs in get_bedrock_route and route
them to converse, matching the behavior of the already-documented
bedrock/converse/arn:... workaround.
Explicit invoke/ prefixes still win, and they remain a dead end for these
ARNs by design (no provider derivable). System-defined inference-profile
ARNs that embed a known model, and other opaque ARN types
(provisioned-model, imported-model, custom-model-deployment) that are
frequently invoke-only, are deliberately left on their current routes;
tests guard both boundaries.
* fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060)
_add_tool_choice_required_message appended the "select a tool" prompt to
the caller's messages list in place, so transform_request corrupted the
caller's conversation history and appended a duplicate prompt on every
retry. Build and return a new list instead so the call stays idempotent.
Adds a regression test asserting the input messages list is unchanged
across repeated transform_request calls.
Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com>
* fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996)
gpt-4o-transcribe and compatible ASR backends return a diarized_json
response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8.
TranscriptionUsageDurationObject typed seconds as int, so parsing the
response raised a pydantic ValidationError (int_from_float). That error
surfaces as an APIConnectionError which the router treats as retryable, so
it keeps re-calling the upstream (200 every time) until the upstream
rate-limits and returns 429 to the caller.
OpenAI specs this field as a float (see openai SDK UsageDuration.seconds),
so widen seconds to float. With the parse succeeding there is no exception
left to retry, which removes the loop.
Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com>
* fix(deepseek): drop non-function tools before chat completions call (#30910)
* fix(deepseek): drop non-function tools before chat completions call
DeepSeek's /chat/completions only accepts tools of type "function".
Requests bridged from /v1/responses can carry responses-API-native tool
types, for example a Codex CLI tool typed "namespace", which DeepSeek
rejects with "unknown variant 'namespace', expected 'function'" so the
whole request fails (issue #30722).
Filter unsupported tool types in the DeepSeek request transform so the
function tools still go through; when nothing callable remains, also drop
the now-dangling tool_choice and parallel_tool_calls
Fixes #30722
* test(deepseek): cover async tool filtering and document tool_choice assumption
Add an async_transform_request regression test so the sync and async tool
filtering paths cannot silently diverge, and document in _drop_unsupported_tools
that only non-function tools are dropped, so a function-named tool_choice always
references a surviving tool
* feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840)
* feat(ui): surface team budget on key overview when key has no own budget (#30801)
* feat(ui): surface team budget on key overview when key has no own budget
* fix(ui): replace IIFE with derived variable and use find() for team budget display
* fix(anthropic): emit replayable streaming thinking blocks (#31022)
* feat(proxy): read cold-storage prompts back in the logs detail view (#30364)
* feat(proxy): read cold-storage prompts back in the logs detail view
When a deployment offloads prompts and responses to cold storage instead of
Postgres, the spend-log row holds only "{}" placeholders plus a
metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed
nothing. The detail endpoint only read the placeholder columns and never
fetched the object back.
Resolve the payload per row based on actual content, not a config flag: if
Postgres has content, return it; otherwise read the exact stored object key and
fetch from the configured cold storage backend through ColdStorageHandler.
Reading the persisted key is a single GET. The key embeds a microsecond
timestamp that cannot be reconstructed from the millisecond-precision startTime
column, and listing the day's prefix to match on request_id would be too
expensive for this per-open path.
Also teach the detail drawer's pretty-view parser to accept a bare messages
array. The cold storage payload carries the prompt as a top-level messages list
with no proxy_server_request, so without this the output rendered while the
input stayed blank.
ColdStorageHandler gains an optional injected logger so the resolver can be unit
tested without monkeypatching. Postgres-stored prompts are unaffected: the fast
path returns the existing columns and the request-body object still renders the
same way.
* Update litellm/proxy/spend_tracking/spend_management_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure
Add unit tests for ColdStorageHandler (injected logger, graceful None when no
logger is configured, and resolution of a configured logger from the callback
registry) and a regression test asserting a cold storage backend exception
degrades to the Postgres values instead of surfacing a 500.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068)
* fix(mavvrik): advance metricsMarker after upload + fix scheduler startup
Two bugs fixed:
1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a
successful GCS upload, so metricsMarker stayed at 0 and every daily run
re-exported the same dates in an infinite catch-up loop.
Fix: add _update_metrics_marker(date_epoch) called at the end of deliver()
after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS
file is already committed). A 410 raises consistent with the rest of the
destination.
2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call
has triggered lazy instantiation of MavvrikFocusLogger, so it found no
logger instance and silently skipped registering the daily export job.
Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call
_init_custom_logger_compatible_class to force instantiation before
the APScheduler job is registered.
* fix(mavvrik): catch up from earliest window when metricsMarker=0
When the connector is freshly registered, metricsMarker=0 parses to None.
The catch-up block was guarded by `if last_ingested and ...` which skipped
it entirely for None, so only yesterday was exported instead of the full
_MAX_CATCHUP_DAYS window.
Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup).
The existing > 7 day warning only fires for non-None markers that are old.
* fix(mavvrik): use now as end_time for yesterday's export window
LiteLLM_DailyUserSpend rows for a given date get their updated_at
bumped by the spend flush job throughout the next morning. The core
database query filters on updated_at, so capping end_time at midnight
(yesterday + 1 day) missed any spend rows flushed after midnight.
Fix: pass now (cron fire time) as end_time for the daily "yesterday"
window so all fully-settled rows are captured regardless of when the
flush job ran.
Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per
row in the exported FOCUS CSV.
* fix(mavvrik): also use now as end_time for catch-up windows
* fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class
Calling it with only logging_integration raised TypeError at proxy startup
because internal_usage_cache and llm_router have no defaults. Also fix test
name to reflect the actual status code (5xx not 4xx) used in the mock.
* ci: retrigger CI run
* feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757)
* Add optional `instruction` passthrough to the rerank API
vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction`
field (folded into the model's chat_template_kwargs and consumed by the
chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently
dropped it: RerankRequest / OptionalRerankParams had no such field, so the
outgoing body was rebuilt without it.
Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(),
get_optional_rerank_params, and the hosted_vllm transformation into the
request body, only when non-None. When callers omit it, model_dump(exclude_none)
drops the field and the outgoing request is byte-for-byte unchanged — fully
backward-compatible. (DeepInfra already forwards `instruction` via
non_default_params; this formalizes the field in the shared types.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: thread `instruction` as a typed param + cover rerank_utils
Per PR review (greptile P2 + codecov):
- Make `instruction` a typed, named argument on the rerank provider interface
instead of recovering it from the opaque `non_default_params` blob. Adds
`instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params`
and every provider override, and forwards it explicitly from
`get_optional_rerank_params`. hosted_vllm now reads the named param directly.
It is still also surfaced in `non_default_params` so providers that read it
there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction`
as a named param rather than leaving it in **kwargs.
- Add get_optional_rerank_params unit tests (present + absent) to cover the
previously-uncovered threading line flagged by codecov.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scan rerank `instruction` through request guardrails
The rerank guardrail translation (CohereRerankHandler.process_input_messages)
only scanned `query`, so the newly added `instruction` field reached the
backend model unscanned. Since instruction-aware rerankers (hosted vLLM /
Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller
could place content there to bypass configured rerank request guardrails.
Generalize the handler to scan every user-controlled text field (`query` and
`instruction`) in one apply_guardrail call and write each sanitized value back
by index. Query-only requests are unchanged (single-element list at index 0);
non-string fields are left untouched. Adds tests covering instruction
scanning, PII masking write-back, and the non-string case.
Addresses the Veria AI security review on PR #30757.
* test: narrow Optional results before len() to satisfy basedpyright budget
The lint gate (basedpyright delta-vs-base budget) flagged one new
reportArgumentType: len(result.results) where results is
List[RerankResponseResult] | None. Assert results is not None first to
narrow the type before len()/indexing.
* fix: read rerank `instruction` from kwargs to satisfy basedpyright budget
The basedpyright delta-vs-base gate flagged one new reportArgumentType: the
Router forwards rerank calls via an untyped `**kwargs` unpack
(`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a
typed named param on the public `rerank`/`arerank` entrypoints made pyright
check that key against `str | None`, adding an error at router.py with no real
safety gain. Read `instruction` from kwargs in `rerank` instead.
It remains fully typed where it matters - threaded as a typed argument through
`get_optional_rerank_params` and each provider's `map_cohere_rerank_params`
(the original Greptile P2 ask). Whole-repo reportArgumentType is back to the
base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(github_copilot): synthesize empty choices at the provider seam (#30929)
Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with
choices=[], either carrying Anthropic-native content blocks or, for the
max_tokens=1 probe Claude Code sends, no content at all. github_copilot
is dispatched through the OpenAI SDK handler, which calls
convert_to_model_response_object directly and never invokes
GithubCopilotConfig.transform_response, so the empty-choices guard there
surfaced as a 500
Instead of synthesizing choices inside the shared
convert_to_model_response_object (which would silently turn empty choices
into a fabricated success for every provider), add a no-op
transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig
overrides it to synthesize choices from Anthropic-native content, reusing
its existing parsing, and the OpenAI SDK handler routes its parsed
response through the hook before generic conversion. The core utility
keeps treating empty choices as an error for all other providers
Fixes: https://github.com/BerriAI/litellm/issues/30927
Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com>
* fix(router): stop fallback lookups from mutating the router fallbacks config (#30624)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in
|
||
|
Sameer Kankute
|
687a62e561
|
fix(cli): mint per-session agent credential on lite login (#31072)
* fix(cli): mint per-session agent credential on lite login
The `lite login` command was producing a shared UI session token that broke agent use in three ways: a $0.25 budget cap (from max_ui_session_budget) that killed agent sessions in minutes, a fixed identity "cli-jwt-token" shared across every user preventing per-session spend attribution, and auth gated behind EXPERIMENTAL_UI_LOGIN so the token was rejected on default deployments.
This fixes all three. Each login now generates a unique cli-session-{uuid} token with no per-key budget cap (enforced via shared team/user counters instead), and the decrypt path activates for any non-sk- token without requiring EXPERIMENTAL_UI_LOGIN.
* fix(cli): address review feedback on EXPERIMENTAL_UI_LOGIN gate and e2e test
Restore EXPERIMENTAL_UI_LOGIN=false as an explicit opt-out: operators who set it to false keep the old boundary; unset (new default) and true both attempt NaCl decryption, which fails closed for non-blob tokens.
In the e2e test: replace the silent Redis fallback with pytest.skip so a missing Redis instance is explicit rather than silently degrading to a directly-minted token. Write the seeded flow back as JSON (proxy reads it via json.loads on cache fetch) instead of Python repr, and build the updated flow immutably.
* fix(key-management): cap CLI session token delegation budget to team ceiling
A CLI session token intentionally carries max_budget=None to avoid a per-session LLM spend cap. The key-generation delegation check (GHSA-q775-qw9r-2r4g) previously skipped non-admin callers with max_budget=None, treating them as having unlimited delegation authority. This allowed any internal user with a lite login session to mint virtual keys with arbitrary budgets.
Adds is_session_token=True to UserAPIKeyAuth for CLI session tokens and uses the caller's team budget as the delegation ceiling in that case, so the effective limit is min(requested_budget, team.max_budget) rather than unbounded.
* chore: regenerate dashboard OpenAPI types
The is_session_token field added to UserAPIKeyAuth cascades to the
dashboard schema. Regenerate types from the updated OpenAPI spec.
* fix(key-management): block personal key budget delegation from CLI session tokens
When team_table is None (personal key, no team_id in request), the personal key
has no team-budget enforcement at request time. A session token therefore cannot
delegate any explicit max_budget for a personal key -- that would open a budget
bypass path. Block the request with a clear 400 directing the caller to use a
team_id instead.
* test(auth): add unit coverage for non-admin CLI session token production path
* fix(type-check): use model_validate in _return_user_api_key_auth_obj to fix reportArgumentType gate
UserAPIKeyAuth(**user_api_key_kwargs) spread triggers a basedpyright
reportArgumentType error for each named field in UserAPIKeyAuth because
the dict's inferred value type (str | Span | LitellmUserRoles | Unknown)
is not assignable to each field's specific type. Adding is_session_token:
bool introduced +2 more such errors, breaching the gate cap.
model_validate accepts an untyped dict without per-field argument checking,
which eliminates the +2 new errors and also ratchets down the pre-existing
333 errors at those call sites. basedpyright-code-budget.json is updated
to reflect the new lower baseline (1814, down from 1934).
* fix(type-check): ratchet down reportArgumentType baseline only
The previous lint-budget-update captured all baselines from the local
environment, raising many ceilings vs the merge-base and failing the
non-gating budget_ratchet_check. Restore staging's values for every
rule and only lower reportArgumentType (1934 -> 1814) to reflect the
reduction from switching to model_validate in _return_user_api_key_auth_obj.
* fix(auth): set max_budget on CLI session token to enforce max_ui_session_budget
CLI session tokens were missing max_budget, so _virtual_key_max_budget_check
had no per-session ceiling to enforce. Operators relying on max_ui_session_budget
could be bypassed for the full token lifetime. Mirrors the existing UI token path.
* revert(auth): remove max_ui_session_budget from CLI session token
max_ui_session_budget defaults to $0.25 and is sized for the UI chat
pane (10-min sessions). CLI sessions are 24-hour tokens for real work;
capping them at that ceiling would throttle users under their actual
user/team budget. Budget enforcement for CLI sessions is via the shared
user and team counters as originally intended.
* fix(auth): cap CLI session at max_ui_session_budget only when user and team have no budget
When neither the user nor their team has a budget configured, CLI sessions
were fully uncapped. The poll endpoint now looks up the real user and team
objects from DB; if both have no max_budget, it passes litellm.max_ui_session_budget
as the token's per-key ceiling. Users or teams that already have a budget
configured are unaffected and continue to rely on the shared counters.
* fix(auth): fix black formatting and update test mock for cli_poll_key budget lookup
The get_user_object and get_team_object async calls in cli_poll_key were
not mocked in the existing test, causing MagicMock await errors. Patch
both functions at the auth_checks module level. Also apply black formatting
to ui_sso.py which CI rejected.
* fix(auth): skip fallback budget cap when team lookup fails for cli session token
* test(auth): pin cli session budget cap to user/team budget presence
The session_max_budget fallback in cli_poll_key only applied
max_ui_session_budget when neither the user nor the resolved team had a
budget. The existing coverage exercised only the team-lookup-failure
branch. Add two regression tests: a user with a configured budget must
not receive the fallback cap, and a session with no user and no team
budget must fall back to max_ui_session_budget. Mutating either guard
out of the branch now fails these tests.
* fix: remove CLI poll session budget cap
* revert(auth): restore CLI session fallback budget cap
Bugbot autofix (
|
||
|
Sameer Kankute
|
2b496bc7f7
|
fix(proxy): restore wildcard expansion in /v1/model/info (#31444) | ||
|
Sameer Kankute
|
e5da5a3b6d
|
fix(proxy): skip model override when response has no model field (#31183)
* fix(proxy): skip OpenAI model override for search responses Search responses omit a model field by spec but still set model on the request for routing, which caused noisy errors and dict injection. * fix(proxy): drop redundant search-specific model override skip The silent return for responses without a model field already covers SearchResponse objects; remove the extra search type check. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): skip model override for dict responses without model key Dict-shaped responses (e.g. search) must not get a spurious model field injected when they never had one; only override when model is present. Co-authored-by: Cursor <cursoragent@cursor.com> * test(proxy): cover swallowed setattr failure in model override --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
7eacdd5258
|
chore: litellm oss staging 250626 (#31305)
* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes #30926) * style: format common_utils.py with black * fix(anthropic): extract api_base from litellm_params in batches/files validate_environment * fix(anthropic): scope Bearer key check to custom api_base endpoints * fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives The Anthropic streaming protocol emits `message_start.usage.output_tokens=1` as a placeholder cursor; the real cumulative output count only arrives in the final `message_delta` event. When a stream is cancelled before `message_delta` lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the `completion_tokens or token_counter(text=...)` fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed. Fix: track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content. Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct. Tests: - TestAnthropicCursorBug (6 cases) — pins the post-fix behavior - TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests still pass. * fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival Addresses both Greptile P2 threads on PR #30420: CLASS A — Anthropic-specific heuristic was applied globally ============================================================ The `completion_tokens == 1 and not saw_non_cursor_completion` reset lived in provider-neutral `streaming_chunk_builder_utils.py`. Any non-Anthropic provider that legitimately reports completion_tokens=1 in a single usage chunk (perfectly normal for short OpenAI / Bedrock / Vertex single-token replies with stream_options.include_usage=true) would have its value silently rewritten to 0 and re-billed via token_counter — producing a different number than what the provider actually charged. Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved from the first chunk's `_hidden_params` (the same field set by streaming_handler.py:722 on the live path). Unknown / missing provider is treated as non-Anthropic and skips the reset, so newer providers and custom plugins are also safe by default. CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies ============================================================ Previous condition was `usage_chunk_dict["completion_tokens"] > 1`, which never fires for an Anthropic stream where the model legitimately emits exactly one output token (e.g., "Yes."). Anthropic still sends message_start (output_tokens=1, the cursor) AND message_delta (output_tokens=1, the real value) — same value, but two distinct usage events. The old check couldn't tell that apart from a cancelled stream where only message_start landed. Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion` when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR (2) we've seen >=2 completion-bearing usage events (positive evidence that message_delta arrived). Cancelled cursor-only streams still have exactly one event and still hit the reset; cache chunks with completion_tokens=0 don't count toward the threshold. Tests ============================================================ - _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default "anthropic") so the gate is exercised by every existing test — none of them needed assertion changes besides the legitimate-single- token case, which now expects exactly 1 (was a fuzzy 0..3 range). - New: test_anthropic_cache_only_chunks_after_message_start_still_resets - New: test_non_anthropic_provider_completion_tokens_one_not_reset - New: test_unknown_provider_completion_tokens_one_not_reset 11/11 tests pass. * chore: add Co-authored-by trailer for attribution Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> * fix(anthropic): preserve messages cache usage * style(anthropic): format messages cache usage helper * fix(anthropic): accept integral float cache token counts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(anthropic): accept integral float cache token counts * test(anthropic): cover cache usage edge cases * fix(gemini): preserve thoughtSignature for server-side tool responses When Gemini API returns toolCall and toolResponse parts, they might have different thoughtSignatures. Previously, LiteLLM merged them into a single dict, overwriting the response's thoughtSignature with the call's. This fix extracts them separately and re-injects them correctly. TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * fix(gemini): address PR comments on thoughtSignature handling - Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature - Add missing assertions in existing tests - Add new unit tests for orphan-response signature handling TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * feat(mcp): include server alias and server_id in mcp_info response - Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint - Update rest_endpoints.py to surface alias from server config - Add test coverage in test_mcp_server.py and test_rest_endpoints.py Fixes #31015 * fix(proxy): reject non-finite spend via validate_finite_spend A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a shared finite-value guard, defined above the litellm.proxy.* imports to avoid the module-level cyclic-import warning. * fix(proxy): require admin for any /key/update spend, reject non-finite Gate the admin check on the presence of `spend` (not a value diff): the DB spend lags the live cross-pod counter, so an "unchanged" spend on the non-admin path let a key owner / team member overwrite the live counter below real usage. Also reject NaN/+-inf spend before the DB write. * fix(proxy): invalidate spend counter on /user/update spend change A direct spend change on /user/update wrote the DB row but left the warm cross-pod counter at the stale value, so enforcement kept reading the old spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB), and reject non-finite spend before the write. * fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244) The semantic cache's embedding model is a proxy Router alias whose AWS credentials (aws_role_name, aws_session_name) live only in the Router deployment's litellm_params. The sync embedding paths called litellm.embedding() directly, bypassing the Router, so they could neither resolve the alias nor assume the configured role; cross-account Bedrock semantic caching failed with "bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding during cache construction, while llm_router is still None. Fix A: make the sync paths mirror the already-correct async paths. A shared, dependency-injected helper (litellm/caching/_embedding_router.py) decides whether to route through llm_router.embedding(...) when the model is a Router deployment, else fall back to direct litellm.embedding(...). Redis and qdrant sync set_cache/get_cache now precompute the embedding and pass vector= to the backend, exactly as the async astore/acheck already do. Both async _get_async_embedding methods are unified onto the same helper and now forward the caller's full metadata instead of a hand-picked subset. Fix B (Redis only): defer redisvl index construction from __init__ into a lazy, memoized llmcache property, so the dimension-probe embedding fires on first cache use, after llm_router is wired. A failed build is not memoized, so a transient outage recovers on the next request. Known limitation: resolve_embedding_router gates on an exact model-name match (same as the shipped async path); wildcard/alias/team-public routes still fall back to direct embedding. Tracked as a follow-up. * fix(cache): harden embedding-router and shrink Any surface (review) Address review feedback on the semantic-cache aws-role fix (#28244): - resolve_embedding_router now skips deployment entries missing model_name instead of raising KeyError on a malformed model_list (Greptile P2); add a regression test that fails on the old direct-key access. - Replace the `**kwargs: Any` passthrough on the four cache _get_embedding / _get_async_embedding helpers with an explicit, typed `metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only ever consumed kwargs["metadata"], so this is behavior-preserving, makes the forwarded field obvious at the call site, and removes three bare-Any annotations (keeps the strict-rule ANN401 budget within ceiling). - Note in _build_llmcache that redisvl's dimension-probe embedding adds one extra billable embedding on the first cache request (Greptile P2). * fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist" Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash * test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression Adds the regression coverage codecov flagged on the two responses to completion bridge guard lines and the bedrock route-prefix helper. The handler tests drive both the sync and async fallback paths with litellm.completion and litellm.acompletion mocked, and assert the forwarded kwargs carry _skip_responses_api_bridge=True, so dropping either flag line fails the suite. The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids still do, exercising both branches of _model_has_route_prefix. Also aligns update_messages_with_model_file_ids model_id to Optional[str], matching its Responses API sibling, so the defensive model_info fallback no longer introduces a new reportArgumentType in completion(); the file-id lookup narrows model_id before the dict get * chore(ui): sync generated OpenAPI types for optional test_connection mode The test_model_connection mode body param default changed from chat to None so the mode is auto-detected from model capabilities, which makes the field optional in the proxy OpenAPI spec. Regenerate the committed schema so the dashboard types match: mode becomes optional and the description and default JSDoc follow the spec, keeping the Check UI API Types Sync gate green * refactor(bedrock): match all explicit route prefixes at path-segment boundary Migrates the remaining substring route checks to the existing _model_has_route_prefix helper so every explicit route token matches only as a leading path segment, consistent with get_bedrock_route and the mantle route. Covers _explicit_converse_route, _explicit_claude_platform_route, _explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route, _explicit_converse_like_route, _explicit_async_invoke_route and _explicit_openai_route. This also stops invoke/ from substring-matching async_invoke/. Route precedence and order are unchanged, and a note on the segment invariant is added to the helper docstring * test(bedrock): cover explicit route prefix segment matching Exercises all eight migrated _explicit_*_route helpers (converse, converse_like, invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each matches its token as a leading path segment and rejects the token glued to a preceding segment, so reverting any method to the old substring check fails the suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete improvement of the segment-boundary migration * test(proxy): assert negative spend is allowed (one-time grant use-case) Negative spend is intentionally permitted so admins can grant extra allowance for the current budget period only, without raising the recurring budget ceiling. Cover it explicitly in validate_finite_spend and via the /user/update invalidation test. * fix(google_genai): forward native generateContent top-level fields Google's native generateContent REST body carries safetySettings, toolConfig, cachedContent and labels at the top level as siblings of generationConfig. The proxy's :generateContent endpoint spread them into agenerate_content as loose kwargs and then dropped them, so callers had to wrap them in extra_body for them to take effect; safetySettings, for instance, was silently ignored The provider config now exposes the native top-level field names and setup_generate_content_call collects whichever are present, merging them into the outgoing request body through the existing extra_body merge so they reach Google verbatim. An explicit extra_body still wins on conflict. The sync generate_content_stream path now also forwards systemInstruction, matching the other three entry points Fixes #12671 Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK * fix(proxy): resolve env refs for DB-stored models * fix(proxy): restrict DB env ref resolution * fix(proxy): block team DB env ref resolution * fix(lint): resolve ANN401/UP045/C901 strict-gate violations - Replace Optional[X] with X | None (UP045) in 8 files - Replace Any return/param types with concrete types or object (ANN401) - Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix Users who pass a key already prefixed with "Bearer " get Authorization: Bearer. All other keys continue to use x-api-key, preserving backward compatibility with custom api_base endpoints that expect x-api-key rather than Authorization. Also consolidates get_auth_header to reuse _make_api_key_auth_header helper, eliminating the duplicated custom-endpoint routing logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base The backwards-compat change broke existing tests that verify the intentional Bearer-for-custom-base behavior (Fixes #30926). Restore original logic while keeping the _make_api_key_auth_header helper for code deduplication. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag Previously the auth-header switch from x-api-key to Authorization: Bearer applied unconditionally for non-sk-ant- keys on a custom api_base, silently breaking existing deployments that proxied to gateways expecting x-api-key. Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header, get_anthropic_headers, and get_auth_header. validate_environment reads it from litellm_params so callers can opt in per-model without any API surface change. Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981) DEL was the only Redis cache operation that skipped check_and_fix_namespace, so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM error on deployments with an ACL restricting DEL to the litellm:* pattern, and a silent no-op on all other deployments since the un-prefixed key was never stored. * style(anthropic): reformat common_utils.py with Black (--target-version py312) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve cache metadata and spend counters * style: apply ruff format to streaming_iterator.py * refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate Extract Anthropic message_start cursor reset into _reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter invalidation into _invalidate_user_spend_counter_if_changed, keeping both _calculate_usage_per_chunk and _update_single_user_helper under the max-complexity ceiling. Use builtin generics in the new signatures so no new UP006 violations are introduced. Behavior unchanged. --------- Co-authored-by: rupak-eng <rupakji99@gmail.com> Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com> Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com> Co-authored-by: Andrii Butko <booandrew23@gmail.com> Co-authored-by: Kent <kingdooo@gmail.com> Co-authored-by: kunal2002 <k.nayyar2002@gmail.com> Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com> Co-authored-by: jesco-absolut <team@srswti.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Matt Hill <mhill@dataminr.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Shivam Rawat
|
eb15fe667d |
refactor(passthrough): address multipart review feedback
Build form_data_dict in one pass with groupby instead of rescanning form_items per field name, and assert on the files list directly in the boundary regression test so repeated field names are not collapsed by dict(). Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Shivam Rawat
|
ef0785881a |
fix(passthrough): forward all multipart files with repeated field names
Passthrough multipart uploads used form.items() and a files dict, so only the last file under a repeated field name reached the upstream. Read multi_items() and send httpx a list of file tuples instead. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
ryan-crabbe-berri
|
f16af8853b
|
feat(mcp): opt-in least-privilege default for team key MCP access (#31380)
* feat(mcp): add require_key_mcp_access_defined to stop keys inheriting team MCP servers By default a virtual key that grants no MCP servers of its own inherits its team's full MCP server list. The new general_settings flag require_key_mcp_access_defined (default false) flips this so the team list acts purely as a ceiling: a key reaches only the servers it grants explicitly (or via an access group), and inherits none. This mirrors the existing require_end_user_mcp_access_defined setting. The default is unchanged, so existing deployments keep today's behavior until they opt in. The no-mcp-servers sentinel and key access-group grants are unaffected. * docs(mcp): note require_key_mcp_access_defined effect in resolver docstring |
||
|
yucheng-berri
|
9203488578
|
feat(spend): store litellm_call_id on spend logs for DB-to-trace correlation (#31344)
* feat(spend): store litellm_call_id on spend logs for DB-to-trace correlation Successful spend logs keyed request_id to the provider response id while tracing uses x-litellm-call-id, so a DB row could not be correlated with its trace; this only worked for failures, where request_id already fell back to the call id. Add a nullable litellm_call_id column to LiteLLM_SpendLogs, populate it in get_logging_payload, and surface it in the spend logs read endpoints so correlation works both directions for successful calls Fixes LIT-3868 * chore: sync schema.prisma copies from root * test(spend): cover cache-hit and missing-response-id paths for litellm_call_id Lock the intended behavior surfaced in review: on a cache hit request_id gets the uniqueness suffix while litellm_call_id stays the raw call id, and when the provider returns no id request_id falls back to the call id so both columns match. Both assertions fail when the populate line is reverted * test(spend): ignore litellm_call_id in spend logs payload comparisons get_logging_payload now always writes litellm_call_id, so the full-payload comparisons in test_spend_management_endpoints.py saw an unexpected key and failed. litellm_call_id is a per-request runtime uuid like request_id, which is already ignored, so add it to ignored_keys * test(logging): ignore litellm_call_id in gcs pubsub spend logs comparison The gcs pubsub spend logs payload comparison flags any key present in the actual payload but absent from the golden snapshot. get_logging_payload now always emits litellm_call_id, a per-request runtime uuid like request_id which is already ignored, so add it to ignored_keys * refactor(spend): store litellm_call_id in spend log metadata, drop column Switch DB-to-trace correlation off a dedicated column and onto the existing metadata JSON, avoiding a schema migration entirely. litellm_call_id is now written into spend log metadata (already selected and re-hydrated on the read paths) instead of a new LiteLLM_SpendLogs column, so the three schema.prisma copies and the migration are reverted and the read SELECTs go back to their original form. Correlation is queryable via metadata->>'litellm_call_id' Trade-off: an unindexed JSON lookup rather than an indexed column; acceptable for this use case and removes all migration risk * refactor(spend): thread litellm_call_id into _get_spend_logs_metadata Set litellm_call_id beside the other computed metadata values inside _get_spend_logs_metadata rather than mutating clean_metadata back in the caller, matching how applied_guardrails, cost_breakdown and the rest are threaded. No behavior change; the value still comes from kwargs with a litellm_params fallback --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
71ee1a852a
|
fix(proxy/client): redact api key from key/info client error messages (#31342)
* fix(proxy/client): redact api key from key/info client error messages
The keys management client builds GET /key/info?key=<key> and lets the
requests HTTPError propagate. str(HTTPError) renders the failing request URL
verbatim ("... for url: .../key/info?key=sk-..."), so any caller that logs the
exception leaks the full key; the 401 branch leaked the same way through
UnauthorizedError(str(orig_exception))
Redact both branches with the existing redact_secrets helper so the
secret-bearing query param is scrubbed to ?REDACTED while the status code,
reason, and response object are preserved. Server-side responses already mask
the key, so this closes the remaining client-side surface
* fix: preserve key info unauthorized response
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
|
||
|
Mateo Wang
|
6db55e0aa5
|
feat(mcp): add mcp_xff_num_trusted_hops to harden X-Forwarded-For client IP resolution (#31257)
* feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution MCP per-server IP access control reads the client IP from X-Forwarded-For and trusts the leftmost entry. Behind an append-style proxy or load balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy, Cloudflare), a client can prepend an arbitrary value to the header, so the leftmost entry is attacker-controllable even when the direct peer is a trusted proxy. An attacker can therefore spoof an internal IP and reach servers marked available_on_public_internet=false. This adds an optional mcp_xff_num_trusted_hops general setting modelled on Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries from the right of the chain (where N is the number of trusted appending proxies in front of the gateway) instead of the leftmost value, so any entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges, which still validates the direct peer, and only takes effect once that check passes; without a validated direct peer the gateway keeps failing closed, so hop counting cannot be abused by a direct-to-pod attacker. The chain must contain at least N valid entries or resolution fails closed. Default is unset, preserving existing behaviour. * chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops * fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum A 0 or negative value is silently treated as disabled, which could leave an operator believing they enabled append-style X-Forwarded-For hardening while client IP resolution stays on the spoofable leftmost value. Emit a warning, consistent with how the module already surfaces invalid CIDR config, so the misconfiguration is visible in logs. * fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time Add a ge=1 bound to the ConfigGeneralSettings field so the update_config_general_settings path rejects 0 and negative values with a clear validation error instead of accepting them, and self-documents the valid range. The runtime warning stays as defense-in-depth for raw-dict config that bypasses model validation. * style(mcp): black-format ip_address_utils.py * fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1) previously made _resolve_num_trusted_hops return None, which the caller treated identically to "unset" and silently fell back to the legacy leftmost X-Forwarded-For value. An operator who set the value to harden client IP resolution but typo'd it would get weaker security than before, with no fail-closed signal. Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid, _HopCount) so the three states are distinct: unset keeps the legacy path, a valid count drives hop-counting, and an invalid value fails closed (returns "") instead of reverting to the spoofable leftmost address. The caller matches on the union exhaustively. Add a parametrized regression test asserting get_mcp_client_ip returns "" for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry, and update the resolver unit tests for the new return type. |