Input didn't wrap its function component in React.forwardRef, so the ref
ConversationList passes for rename auto-focus/select silently never attached
under React 18 (function components need forwardRef to receive a ref; that
requirement is dropped in React 19, but this app is on 18.3.1).
MCPEnhancedStreamingIterator only auto-executed one round of MCP tool calls.
When a model retried a tool (e.g. after an error) in its follow-up turn, that
second tool call was streamed but never executed, and the response ended with
no final text. Route follow-up calls back through the same completion-check
phase as the initial response, so further tool-call rounds are handled the
same way, capped at MAX_MCP_TOOL_CALL_ROUNDS to avoid an unbounded loop.
Establishes a real design.md/AGENTS.md for the chat UI (tokens,
component patterns, decision trees) after several rounds of hand-rolled
Tailwind shipping invisible or broken states, then audits every
component in the directory against it: raw <button>s replaced with
shadcn Button throughout, spinners replaced with Skeleton for list/table
loading states, dark-mode contrast bugs fixed (MCPAppsPanel cards were
bg-background instead of bg-card, identical to the page background in
dark mode), Badge variants and status colors aligned with the documented
semantics, and the sidebar's active-nav-item styling switched to the
purpose-built sidebar-* tokens instead of the generic accent/secondary
tokens that collapse to the same value in this theme.
Also: disables model comparison mode and multi-select in favor of a
single active model, moves the model picker from a standalone top bar
into the composer, removes the sidebar collapse toggle and the
non-functional "Search chats" entry, and renames the conversation list's
"Today" group to "Recents".
Fixes a real scroll bug: the model picker's dropdown list was
unscrollable because its container used max-height instead of an
explicit height, which doesn't count as a definite size for the
percentage-height Radix ScrollArea viewport to resolve against — so the
viewport silently expanded to full content height instead of clipping,
and scroll events fell through to the page behind it. Same latent bug
fixed in the sidebar's conversation list.
"AI Gateway" in the topnav view switcher only called setMode(), which
is meaningful inside the dashboard SPA shell but a no-op on /chat,
which lives outside it (only "Chat" had a real navigation). Now
switching modes from the chat route does a real navigation back to
the dashboard root.
Also adds a persistent banner across all chat routes flagging it as a
pre-v0 feature not for production use, with a feedback link.
Moves Chats/Integrations/Credentials/API Keys/Usage from client-side
tab state to real nested routes (/chat, /chat/integrations,
/chat/credentials, /chat/api-keys, /chat/usage) so each is bookmarkable
and survives a hard reload. Extracts the chat sidebar into ChatShell
and shared state (MCP server selection, conversation history) into
ChatShellContext, both consumed via the new app/chat/layout.tsx.
Along the way: fixes conversation URLs pointing at the wrong path
(/ui/chat instead of /chat in dev, which 404'd after sending the first
message) by reusing the existing migratedHref helper instead of a
one-off uiConfig-based path; fixes the topnav view-switcher always
showing "AI Gateway" as selected even while on the chat route; and
cleans up several shadcn/tailwind styling bugs introduced by the antd
migration (boxed tab outline instead of underline, model-selector
dropdown overflowing its popover, sidebar nav labels centered instead
of left-aligned, duplicate logo, dead non-interactive controls).
* fix: prevent duplicate budget alert emails on concurrent threshold crossings
Budget alert emails were sent more than once for a single threshold crossing. The email dedup guard read the "already sent" marker, awaited the send, then wrote the marker, so concurrent requests crossing the same threshold within the send window all saw no marker and each sent. This affected the multi-threshold path (default_key_max_budget_alert_emails), the legacy single-threshold path (EMAIL_BUDGET_ALERT_MAX_SPEND_ALERT_PERCENTAGE), and the soft budget path, all in EmailBaseCallback.budget_alerts
All three branches now claim the send slot atomically before sending via async_increment_cache, which is atomic per event loop for the in-memory cache and across workers via Redis INCR; only the caller that observes a count of 1 sends. On send failure the marker is released with async_delete_cache so a transient failure does not suppress the alert for the full 24h TTL
* fix: harden budget alert claim release and skip-path event allocation
Addresses review feedback on the claim-before-send change. The claim release in each send-failure handler now logs the send error first and releases the claim best-effort through a shared helper, so a transient cache error during async_delete_cache cannot propagate out of the fire-and-forget budget_alerts task, drop the send-failure log, and leave the claim stuck for the full 24h TTL. In the multi-threshold branch the increment claim now runs before the WebhookEvent is built, so skipped concurrent crossings no longer construct and discard the event, matching the single-threshold and soft budget branches
The lint rule allows console.warn (allow: [warn, error]) but removeConsole
only excluded error, so approved console.warn calls were silently dropped
from production bundles. Add warn to the exclude list so the prod strip
and the lint allow-list agree; only console.log/debug/info are stripped
now, warn and error both survive (verified: warn 85 to 85, error 906 to
906, log 675 to 14).
Merging litellm_internal_staging dropped 2 console.log calls (the
currentUser logs removed in #32079), so the no-console budget max and
metric move from 486 to 484 to match the current count.
Introduce a gradual ratchet to remove raw console.* calls from the
dashboard, mirroring the existing no-explicit-any budget.
The no-console eslint rule is set to warn with allow: [warn, error] so
the 486 console.log/debug/info calls are tracked without force-deleting
the legitimate console.error/warn error reporting in catch blocks. The
count is grandfathered via eslint-budgets.json (max 486, target 0) and
eslint-metrics.json, so any newly added console.log fails the budget
check and follow-up PRs grind the max down toward zero.
Independently, next.config strips console output from production builds
via SWC removeConsole (exclude: [error]), gated on NODE_ENV=production so
dev keeps full console output. This gives an immediate prod-hygiene net
regardless of how long the source cleanup takes. Verified against a real
production build: app-code console.log dropped from 675 to 14 in the
bundle (remainder is node_modules, which the transform leaves alone),
console.warn app calls stripped, console.error preserved 906 to 906.
The ui_unit_tests CircleCI job logged ~45k lines for a single run, most of
it React act() warnings, antd deprecation notices and component stack traces
emitted as console output by passing tests, which buried real failures.
Set silent: "passed-only" (Vitest 3.2+) gated on process.env.CI so console
output from passing tests is suppressed while a failing test still prints its
logs and full stack trace. Also drop two stray console.log calls in
UsagePageView that dumped the whole currentUser object on every render in
production, not just tests.
Verified by running the suite the way CI does
(CI=true npm run test -- --run --pool forks --poolOptions.forks.maxForks=6):
45,075 lines before, 981 after, all 4075 tests still passing. A throwaway
failing test confirms its console.log and assertion diff remain visible.
* chore: clarify the linear ticket instruction in pr template
* fix: make it more concise
* Update CLAUDE.md
lol
* chore: tell claude not to search for it if it doesn't have it
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(anthropic): keep context_management working when drop_params is enabled
drop_params (proxy-wide or per-request) silently disabled the in-gateway
context_management polyfill on the /v1/messages -> chat completions adapter
path, even though context_management is a LiteLLM-supported param (native on
Anthropic, polyfilled elsewhere). Gate the polyfill on an explicit
additional_drop_params: ["context_management"] opt-out instead, which also
makes that escape hatch actually work on the adapter path.
* test(anthropic): cover sync adapter polyfill gate for global drop_params and additional_drop_params
The ui_unit_tests job runs vitest with maxForks=8 on an 8-vCPU xlarge
container, leaving no headroom for the main vitest process that services
worker RPCs. Under full CPU saturation the coordinator misses the
onTaskUpdate ack, vitest raises "Timeout calling onTaskUpdate" as an
unhandled error, and the job exits 1 even though every test passes.
Lower maxForks to 6 so the coordinator, jsdom, and OS keep two cores, and
raise teardownTimeout to 60s for extra slack on heavy runs.
* feat(mcp): add all-proxy-mcpservers sentinel to grant every MCP server
Teams can now be scoped to the all-proxy-mcpservers sentinel so they gain
access to every MCP server on the proxy without listing each id. The
sentinel expands to the live registry at request time, so a server added
later is picked up with no change to the team's stored permission. The team
ceiling that validates a key's MCP scope expands the sentinel too, so a key
can be scoped to any server (including one registered after the team) and
still pass subset validation
Expose the option in the team create and edit forms via a new exclusive
"All Proxy MCP Servers" choice in MCPServerSelector, mirroring the existing
"No MCP Servers" sentinel
* Update litellm/proxy/management_helpers/object_permission_utils.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mcp): honor all-proxy-mcpservers only on the team path, never per-key
The sentinel was expanded inside the shared expand_permission_list, which
also feeds the key, org, end_user and agent resolvers. A key whose stored
object_permission ever held all-proxy-mcpservers (a stale write, a
configured default, or a bug) would silently resolve to every MCP server at
runtime, and a teamless key had nothing to cap it, so all servers got
injected. Only write-time validation stripping the value stood between that
value and a full grant
Move the expansion out of expand_permission_list and into
_get_allowed_mcp_servers_for_team so the sentinel is honored only where it is
settable (a team). Anywhere else it now passes through as an inert literal
that matches no registered server and is denied downstream. Reserved-id
protection already blocks a real server from taking that id
* fix(mcp): require proxy admin to grant a team the all-proxy MCP sentinel
Granting a team every MCP server on the proxy is a proxy-wide authorization
decision, but team create/update let any caller who can manage a team set
object_permission.mcp_servers, with no ceiling check. Org admins reach
/team/update by default (org_admin_allowed_routes) and _verify_team_access
also admits team admins, so a non-proxy-admin could set all-proxy-mcpservers
and self-grant their team access to every MCP server on the proxy, including
servers never assigned to that team
Gate the grant in new_team and update_team: a non-proxy-admin cannot add the
all-proxy-mcpservers sentinel. The check is scoped to newly adding it, so a
team a proxy admin already scoped to all-proxy can still be edited by a team
admin without being forced to strip the sentinel. The UI only offers the
"All Proxy MCP Servers" option to proxy admins in the team create and edit
forms
* fix(ui): render friendly all-proxy MCP label for non-admins editing an all-proxy team
A team scoped to the all-proxy-mcpservers sentinel could be opened in the team
edit form by a team admin or org admin (canEditTeam admits them), but the
"All Proxy MCP Servers" option in MCPServerSelector was rendered only behind the
proxy-admin-gated allowAllProxyMcpServers flag. For a non-proxy-admin the stored
sentinel was hydrated into the selected value with no matching Select.Option, so
antd showed the raw all-proxy-mcpservers literal as a chip, and adding another
server could persist a mixed [all-proxy-mcpservers, <id>] value.
Render the option whenever the sentinel is present in the value, not only when
the caller may grant it, and drive the real-option disabling off presence too so
the selection stays exclusive. A non-proxy-admin now sees the friendly label
read-only and cannot build a mixed state; only a proxy admin can newly add it,
which the backend already enforces.
Adds regression tests: the selector shows the friendly option (not the raw
literal) when the sentinel is stored but the grant flag is off, plus exclusive
emit and disabled-real-options coverage, and MCPServerPermissions renders the
green "All" state instead of the raw sentinel string.
* fix(ui): drop redundant "All servers" hint from the all-proxy MCP chip
antd renders a Select option's children inside the selected tag, so the
all-proxy option showed both "All Proxy MCP Servers" and the green "All servers"
type-hint in the chip, which say the same thing. Collapse the option to a single
green "All Proxy MCP Servers" label so the dropdown row and the chip read cleanly
without the duplication.
* fix(ui): color the all-proxy MCP label blue to match server chips
Use the same blue (#1890ff) as regular MCP server entries for the
"All Proxy MCP Servers" option/chip instead of green.
* fix(ui): make the all-proxy MCP permissions display blue, not green
Match the blue used by the selector chip and regular server entries so the
"All Proxy MCP Servers" badge and row in MCPServerPermissions are consistent
across the team/key/org detail views. The red "Blocked" state for
no-mcp-servers is unchanged.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mcp): persist DCR client_id so interactive OAuth token refresh works
Interactive authorization_code MCP servers register an OAuth client via Dynamic
Client Registration (RFC 7591) during the authorize flow, but the minted
client_id and the discovered token_url were returned to the caller and never
written to the server row. The autonomous refresh_token grant reads client_id,
client_secret and token_url off the server, so an expired access token could not
be refreshed; the user was bounced back to re-authorize and tools/list returned
zero tools
Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when
the registration returns them) and the discovered token_url onto the server row,
reusing the encrypt_credentials write that client_credentials and token exchange
already use, then refresh the in-memory registry so the value is live at refresh
time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same
fields, so egress needs no change
* fix: reuse persisted MCP DCR clients
* fix(ui): persist DCR client_id from on-create MCP OAuth "Authorize & Fetch"
The interactive "Authorize & Fetch" flow on the create form registers an OAuth
client (RFC 7591) against a temporary server that has no DB row, then creates the
real server afterward. useMcpOAuthFlow captured the DCR client_id and client_secret
but passed only the token to onTokenReceived, so the create request dropped the
client identity and the created server could not refresh its access token; its row
had credentials={} and the refresh_token grant 401d at the upstream token endpoint
Forward the registered client to onTokenReceived and write client_id (and
client_secret when present) into the create form credentials, so the create request
carries them and the backend persists them through its existing encrypt_credentials
path. token_url is omitted because it is re-discovered on load (RFC 9728 then 8414);
token_endpoint_auth_method is unused because this flow only ever registers as
client_secret_post or none, never client_secret_basic
* fix(ui): prevent stale MCP OAuth credentials
* fix(ui): reset MCP OAuth authorization state
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
When AzureSentinelLogger is resolved from the string callback name
"azure_sentinel", it is constructed with no arguments, so audit_stream_name is
always None and resolved_audit_stream_name fell back to the standard
resolved_stream_name. Audit logs then ingested into the access-log DCR stream
whose schema is built from StandardLoggingPayload, so Azure Monitor Logs
Ingestion silently dropped the audit-specific columns and audit rows arrived
effectively empty.
Add an AZURE_SENTINEL_AUDIT_STREAM_NAME env var fallback in __init__, mirroring
the AZURE_SENTINEL_STREAM_NAME idiom already used for the standard stream, so
audit logs can target a separate DCR stream without a custom callbacks file.
* fix(mcp): persist DCR client_id so interactive OAuth token refresh works
Interactive authorization_code MCP servers register an OAuth client via Dynamic
Client Registration (RFC 7591) during the authorize flow, but the minted
client_id and the discovered token_url were returned to the caller and never
written to the server row. The autonomous refresh_token grant reads client_id,
client_secret and token_url off the server, so an expired access token could not
be refreshed; the user was bounced back to re-authorize and tools/list returned
zero tools
Persist the DCR client_id (plus client_secret and token_endpoint_auth_method when
the registration returns them) and the discovered token_url onto the server row,
reusing the encrypt_credentials write that client_credentials and token exchange
already use, then refresh the in-memory registry so the value is live at refresh
time. Both the v1 refresher and the v2 AuthorizationCodeRefresher read those same
fields, so egress needs no change
* fix: reuse persisted MCP DCR clients
* fix: reuse persisted MCP DCR clients
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
PR #31635 added a per-server token_endpoint_auth_method (client_secret_basic
or client_secret_post) for upstream OAuth token endpoints, but it could only be
set by editing the stored credentials JSON. This surfaces it in the dashboard as
an optional selector directly under the Token URL field, in both the create form
(OAuthFormFields, M2M and interactive flows) and the edit form. The field binds
to credentials.token_endpoint_auth_method, which the backend already reads; the
value is sent only when chosen, so leaving it blank keeps the existing setting
and preserves the client_secret_post default.
* feat(tinyfish): make search provider permissive, attribute errors
Reshapes the TinyFish search provider so LiteLLM mirrors the TinyFish
Search API surface instead of maintaining a parallel cherry-pick.
Request side:
- Drop misleading request TypedDict
- Stop sending max_results on wire (TinyFish ignores it); clamp to [1,10]
client-side via self-threaded state
- Guard non-numeric max_results from bare ValueError
- Auto-JSON-encode dict params; lowercase bool serialization for ux-labs
Response side:
- Drop both Pydantic response models; parse directly into SearchResponse
so per-result extras flow through via extra="allow"
- Default missing title/url/snippet to "" instead of failing the call
- Read top-level parameter_warnings and re-fire as verbose_logger.warning
(pre-wired for upcoming TinyFish-side rollout; no-op today)
Error handling:
- Attributed _wrap_error helper at 3 call sites in transform_search_response
("TinyFish Search: <msg>. See https://docs.tinyfish.ai/search-api for
details.")
- Dispatch non-2xx responses through _wrap_error (fixes pre-existing bug
where 4xx/5xx silently returned empty SearchResponse)
- Wrap json.JSONDecodeError on 200 bodies
- Wrap pydantic.ValidationError for envelope-shape mismatches
Bug fix worth flagging: 4xx/5xx responses now raise an attributed
BaseLLMException instead of silently returning SearchResponse(results=[]).
Follow-up to #30634.
* fix(tinyfish): apply ruff format; guard OverflowError in max_results clamp
- Run ruff format on the touched files (CI lint job rejected the prior
commit's formatting).
- Add OverflowError to the except clause in the max_results clamp so
callers passing math.inf (or other non-finite floats) get the same
warn-and-ignore behavior as other malformed values. Greptile spotted
this in the first-pass review.
- Add test_max_results_infinity_float_warns_and_skips covering the
inf case.
* fix(tinyfish): apply --line-length 88 ruff format to match CI
CI uses 'ruff format --check --line-length 88'; my prior format pass
used the default line length, leaving several lines unwrapped. No
behavior change — purely whitespace.
* fix(tinyfish): reduce transform_search_response complexity; sort imports
CI's ruff strict-rule budget rejected the prior commit with:
- C901: transform_search_response complexity 16 > 10 (cap exceeded by 1)
- I001: import sort violation (cap exceeded by 1)
Extract two module-level helpers from transform_search_response to drop
its cyclomatic complexity:
- _default_missing_result_fields: in-place title/url/snippet defaulting
- _emit_parameter_warnings: defensive parameter_warnings reader
Auto-fix the import sort via ruff --fix.
No behavior change; the 59 existing tests still pass.
* test(tinyfish): cover defensive branches in _default_missing_result_fields
Codecov flagged 97.61% patch coverage (2 lines missing). The uncovered
lines were the non-dict raw_json and non-dict per-result item early-exits
in _default_missing_result_fields. Add two unit tests on the helper
directly to bring patch coverage to 100%.
* chore(tinyfish): apply ruff format to fix lint after staging merge
---------
Co-authored-by: Chenlu Ji <jichenlulu@gmail.com>
* fix(azure_ai): preserve content, tables, and keyValuePairs in doc-intelligence /v1/ocr
Azure Document Intelligence analyzeResult.content, .tables, and
.keyValuePairs were dropped when normalizing to the Mistral OCR schema.
They are now passed through verbatim as top-level response fields, and
the duplicated sync/async response parsing is consolidated into one
pydantic-validated helper.
Also adds the Azure DI features query param (list[str] or
comma-separated string, e.g. features=keyValuePairs) which Azure
requires for keyValuePairs extraction.
* test(azure_ai): replace fastapi jsonable_encoder with model_dump in ocr unit tests
The bulk-update entrypoints `/key/bulk_update` and `/team/key/bulk_update`
route through `_process_single_key_update`, not through
`_validate_update_key_data`, so the `permissions` gate LIT-4092 wired
into the single-key path never fires on bulk. Currently safe by
construction: `BulkUpdateKeyRequestItem` doesn't declare `permissions`
(Pydantic silently drops it), and `KeyUpdateFields` uses
`model_config = ConfigDict(extra="forbid")` (Pydantic 422s at parse
time). Neither structural barrier is enforced by tests on the field
itself; a future widening of either allowlist to include `permissions`
would reopen the class silently.
This wires `_check_permissions_caller_permission` into
`_process_single_key_update` right after `_validate_max_budget`, before
`prepare_key_update_data`. Zero behavior change today for any caller
routing through the current bulk request models; a defense-in-depth
gate for the class.
Tests:
- test_process_single_key_update_non_admin_permissions_rejected
- test_process_single_key_update_non_admin_permissions_explicit_empty_rejected
Both mutation-killed against removing the gate. Full mapped test file
(341 tests) green.
`_check_allowed_routes_caller_permission` previously keyed its
admin-only rule on truthiness. The refactor adds an `allowed_routes_was_provided`
keyword param that raw-body call sites populate from
`"allowed_routes" in data.model_fields_set`, so a caller that omits
the field (default flows through) is distinct from one that sends
any explicit value.
Four raw-body call sites now pass `allowed_routes_was_provided=...`:
`_common_key_generation_helper`, `generate_service_account_key_fn`,
`_validate_update_key_data`, and `regenerate_key_fn`.
Two derived-value call sites keep the pre-fix shape: the
post-`handle_key_type` recheck at `_common_key_generation_helper`
and the mirror in `regenerate_key_fn`. Both pass values produced by
`handle_key_type` (not by the request body), so `allowed_routes_was_provided` stays
False and the `allow_safe_presets=True` carve-out continues to accept
the `llm_api_routes` / `info_routes` presets.
In `regenerate_key_fn` the gate runs before the `premium_user`
license check, matching the LIT-4092 ordering.
`test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate`
pins the ordering; it fails on a swap of the two gates.
Tests in `tests/test_litellm/proxy/management_endpoints/test_key_management_endpoints.py`
under `TestAllowedRoutesCallerPermission`:
- test_non_admin_generate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_empty_allowed_routes_rejected
- test_non_admin_update_key_explicit_null_allowed_routes_rejected
- test_non_admin_regenerate_key_explicit_empty_allowed_routes_rejected
- test_non_admin_regenerate_key_allowed_routes_rejected_before_enterprise_gate
- test_helper_accepts_derived_safe_preset_for_non_admin
- test_helper_rejects_derived_unsafe_preset_for_non_admin
- test_helper_rejects_when_provided_and_none_without_typeerror
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Three helper-level tests pin the derived-value branch and the
load-bearing None guard; each is mutation-killed against a targeted
change to the frozenset or the guard. Full mapped test file (347 tests)
green.
`NewUserRequest` and `UpdateUserRequest` inherit `permissions` from
`GenerateRequestBase`. `/user/new` passes the field into
`generate_key_helper_fn` which persists it on the auto-created key,
so an org admin who lands on `/user/new` (the route allowlist accepts
org_admin callers when the request body names an org where they hold
that membership) can mint a key with proxy-wide capabilities such as
`get_spend_routes`.
This wires the existing `_check_permissions_caller_permission` helper
into `new_user` and `_update_single_user_helper`. The helper's presence
check keys on `data.model_fields_set`, so an omitted field flows
through untouched and an explicit `{}` / `null` from a non-admin is
rejected 403 the same as any other value.
`_update_single_user_helper` is shared by `/user/update` and
`/user/bulk_update`, so both paths inherit the gate.
Tests in
`tests/test_litellm/proxy/management_endpoints/test_internal_user_endpoints.py`:
- test_new_user_non_admin_permissions_non_empty_rejected
- test_new_user_non_admin_permissions_explicit_empty_rejected
- test_new_user_non_admin_omits_permissions_succeeds (control)
- test_new_user_admin_can_set_permissions (control)
- test_update_single_user_non_admin_permissions_rejected
- test_update_single_user_non_admin_permissions_explicit_empty_rejected
The four attack-vector tests fail on the pre-fix HEAD and pass on this
commit. Full mapped test file (79 tests) green.
_enforce_key_and_fallback_model_access and can_key_call_resolved_model
both unconditionally skipped key-level model checks whenever
all-team-models was present, without verifying the key actually belongs
to a team. PR #29746 fixed the listing path (get_key_models) but these
two call-path checks were left untouched, letting teamless keys call
any model via /chat/completions while seeing an empty model list.
Add team_id is not None guard to both bypass conditions so teamless
keys fall through to can_key_call_model, which already correctly
rejects unresolvable all-team-models sentinels
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): bound per-request budget metric emission with a timeout (#31632)
* fix(prometheus): bound per-request budget metric emission with a timeout
Wrap the per-request budget-metric gather in asyncio.wait_for so a slow Redis or DB lookup cannot consume the whole LoggingWorker watchdog and get the success-logging event cancelled. On timeout the emission is skipped in isolation; budget gauges are still refreshed by the periodic cron. The timeout is configurable via PROMETHEUS_BUDGET_METRICS_PER_REQUEST_TIMEOUT and defaults to 5.0 seconds, falling back to the default on an invalid value instead of raising
* fix(prometheus): reject non-finite and non-positive budget-metrics timeout env
float() accepts 0, negatives, nan and inf, which bypass the fallback: a value <= 0 makes asyncio.wait_for time out immediately and skip every per-request emission, and inf reintroduces the unbounded wait the timeout was meant to bound. Validate the parsed value is finite and greater than zero before using it, otherwise fall back to the default
* fix: report the blocked LLM response's real token usage (#31217)
When a guardrail blocks a post-call response, the synthetic violation response
reported hard-coded zero usage, discarding the token usage the upstream call
had already consumed.
Fix the root cause rather than re-counting tokens:
- Add an optional `original_response` field to ModifyResponseException.
- The unified guardrail's post-call success hook attaches the blocked LLM
response to the exception.
- The /v1/messages and OpenAI-format (/v1/chat/completions, /v1/completions)
block handlers report `original_response.usage` directly. Pre-call blocks
never invoked the LLM, so usage is zero.
Mock-based tests cover the helper (returns original usage / zero), the success
hook attaching original_response, and the endpoint reporting it end-to-end.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(guardrails): buffer + cleanly terminate streamed responses on block (#31389)
Streaming moderation improvements for the unified guardrail post-call
streaming iterator hook:
- streaming_buffer_until_moderated: withhold all chunks until end-of-stream
moderation passes, then release the original response (clean) or only the
block message (blocked) -- the original content is never delivered on a
block. Snapshot chunks with a shallow list() copy (end-of-stream builds a
separate assembled response; chunks aren't mutated in place).
- Clean Anthropic SSE on block: synthesize a well-formed termination sequence
instead of a bare data: {"error": ...} blob that truncates the stream.
Provider-specific synthesis lives in AnthropicMessagesHandler via
build_block_sse_chunks (format-agnostic routing stays in the hook).
- Mid-stream blocks continue the in-progress message (close open content
block, append block message, terminate) rather than emitting a second
message_start, which clients reject. Standalone envelope only when no chunks
were sent (buffered path).
- ModifyResponseException imported under TYPE_CHECKING + locally at runtime to
avoid a module-level cyclic import.
Adds regression tests for buffering (content withheld on block) and mid-stream
continuation (single message_start).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: report real usage on streaming blocks, disable buffered mode for content-rewriting guardrails
- _standalone_block_chunks and _block_continuation_chunks now read real
token usage from ModifyResponseException.original_response instead of
hardcoding zero, matching the non-streaming _blocked_response_usage path.
Shared helper moved to guardrail_translation/utils.py.
- streaming_buffer_until_moderated is now forced off when the guardrail has
mask_response_content=True, since buffered replay releases the withheld
original chunks verbatim -- unsafe for a guardrail that rewrites content
(e.g. PII masking).
- Fix inverted streaming-flag precedence comment.
* style: ruff format after greploop fixes
* fix: handle Anthropic streaming guardrail blocks
* fix(responses): check terminal event type for streaming guardrail end-of-stream detection
_check_streaming_has_ended assumed responses_so_far held ModelResponse
objects with .choices, but for the Responses API the accumulated chunks
are raw SSE event dicts, causing an AttributeError on every call
* fix: preserve Anthropic blocked stream usage
---------
Co-authored-by: FERNANDO IZAR <fizar@me.com>
Co-authored-by: Joseph Barker <156112794+seph-barker@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* perf(lint): skip and cache base gate passes, parallelize make lint, skip redundant prisma generate
make pre-commit paid for a full second basedpyright pass over a merge-base
worktree on every run even when no rule was over its ceiling, re-generated an
unchanged Prisma client, and ran seven independent checks sequentially. The
basedpyright and ruff strict gates now skip the base pass when head is within
every limit (the same early-out type_discipline_gate already had), the
basedpyright base counts are cached under the git common dir keyed by
merge-base commit, pyrightconfig.json, and uv.lock, prisma generate only runs
when the schema or prisma version changed, and make lint fans its checks out
through a parallel sub-make after a single setup phase
* fix(lint): keep the base-cache scratch file out of the prune glob
The tmp+rename scratch in store_counts was named basedpyright-base-<hash>.json.tmp,
which the stale-entry prune glob (basedpyright-base-*) also matches, so a concurrent
lint run from another worktree sharing the same git common dir could unlink it between
write_text and replace and crash the gate with FileNotFoundError. The scratch is now
dot-prefixed so the glob can never see it, pid-suffixed so concurrent writers of the
same entry never share a scratch, and the prune glob is restricted to committed
*.json entries
* chore(e2e): untrack gateway config and document e2e test location
Stop tracking tests/e2e/gateway/litellm-config.yml so the local proxy config stays on the machine
Add a note to CLAUDE.md that new e2e tests belong in tests/e2e/ and must follow that directory's conventions
* chore(e2e): add self-contained docker compose stack for local runs
Ship a docker-compose.yml that starts the proxy with a throwaway Postgres and Redis and inlines the proxy config with example models, so contributors can bring up a local gateway with nothing but a .env. Update CONTRIBUTING.md to match the inline-config flow
* chore(e2e): drop the second gemini deployment; one key is enough locally
* docs(e2e): make pre-commit steps ordered and require flagging internally found issues
* feat(ui): shadcn migration foundation: Tailwind v4, shadcn init, antd cascade fix
Upgrade the dashboard from Tailwind v3 to v4 with CSS-first config: the
official upgrade codemod renamed utilities across 151 files, and
tailwind.config.js (plus the dead tailwind.config.ts) is replaced by
@theme tokens, @source globs, and @plugin directives in globals.css. The
Tremor safelist becomes @source inline patterns and the legacy tremor
theme tokens carry over verbatim. ui_colors.json was build-time only and
fed the dying Tremor palette, so its brand values are inlined and the
file removed; runtime theming replaces that path next.
shadcn is initialized with a hand-authored components.json (rsc,
cssVariables, baseColor gray) pointing utils at the existing
lib/cva.config.ts, which now exports cn (cva beta cx + twMerge) instead
of adding class-variance-authority as a second variant library. The two
ad-hoc cn helpers fold into it. Button lands as the canary primitive,
adapted to cva beta and React 18 forwardRef, with tests covering the
variant, twMerge, asChild, and ref seams. --radius is 0.5rem so the
shadcn radius scale reproduces Tailwind defaults and legacy rounded-*
classes render unchanged.
antd v5 emits unlayered CSS-in-JS that would beat every layered v4
utility, so AntdGlobalProvider now wraps the app in StyleProvider layer
and ConfigProvider cssVar, and globals.css declares
@layer theme, base, antd, components, utilities. antd wins over
preflight but yields to utilities, which is what lets migrated shadcn
pages coexist with legacy antd pages. Preflight stays global with the
three v3 behaviors pinned (default border color, button cursor,
placeholder color).
* fix(ui): restore tremor opacity tints removed by tailwind v4
Tailwind v4 removed the *-opacity-* utilities, but the precompiled
@tremor/react dist still composes them with shade-500 palette classes
(bg-opacity-10 over bg-<color>-500 etc.), so Badge, BadgeDelta, Callout,
light Icon and Button, BarList, and ProgressBar lost their tints and
rendered solid 500-shade fills. Adversarial review caught it; the
original smoke pages only exercised antd Tags.
tremor-v3-compat.css restores exactly the pairs tremor emits: for each
of the 22 safelisted colors, bg-opacity-{10,20,40}, hover/group-hover
bg-opacity-{20,30}, and ring-opacity-{20,40} against the -500 shade,
via color-mix into the utilities layer. Tremor's colorPalette maps both
background and iconRing to 500, so the -500 pairing covers every
composition in the dist; dark: variants are inert until dark mode ships.
The shim dies with @tremor/react at the end of the migration.
The upgrade codemod also missed two hand-rolled modal scrims using
bg-black bg-opacity-{30,50} (solid black under v4); now bg-black/30 and
bg-black/50. Removed the docker/build_admin_ui.sh copy of
enterprise_colors.json into the deleted ui_colors.json; that build-time
rebrand path is retired and its runtime replacement lands with the
theming phase.
* fix(ui): pair ring-opacity-40 with shade 300 in tremor compat shim
Tremor's colorPalette maps ring to shade 300, and the only consumer of
ring-opacity-40 (Icon variant outlined) composes it with that shade,
so the shade-500 rows were dead and outlined icon rings would render
at full opacity. Latent today (no dashboard usage of the outlined
variant); caught by adversarial review. ring-opacity-20 stays at 500
(iconRing), matching Badge and BadgeDelta.
* fix(bedrock): map guardrailConfig to InvokeModel guardrail headers
The InvokeModel API takes the guardrail identifier, version and trace as
X-Amzn-Bedrock-* request headers, unlike Converse which takes them in the
request body. The invoke transformer never set these headers, so
guardrailConfig was silently dropped (or leaked into the request body)
and Bedrock guardrails never ran on invoke-route models. Pop
guardrailConfig in AmazonInvokeConfig.validate_environment, validate it,
and set the headers before SigV4 signing; explicitly passed headers keep
winning over guardrailConfig so existing workarounds are unaffected
* fix(bedrock): reject guardrailConfig missing guardrailIdentifier
A guardrailConfig without guardrailIdentifier (e.g. an empty dict) would
validate, produce no guardrail headers, and let the request proceed with
guardrails silently not applied; that silent skip is the exact failure
mode this fix exists to remove, so fail fast with a 400 instead
* feat(tencent): add Tencent TokenHub as a provider
Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a
new provider: TencentChatConfig routes /v1/chat/completions and gates the
thinking/reasoning_effort params behind supports_reasoning, and
TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API.
Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider
endpoint support metadata.
* test(tencent): add unit tests for Tencent TokenHub provider
Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig
(messages API) across transformation, param mapping, URL building, and header
validation, plus get_optional_params routing. Tests mock supports_reasoning to
stay independent of remote model cost data.
* fix(tencent): correct max_output_tokens and reuse parent messages env validation
Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent
Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden
Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling
* fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix
* fix(tencent): register tencent in models_by_provider
The provider was added to the LlmProviders enum and cost map but not to the
models_by_provider lookup, so test_models_by_provider (which asserts every
litellm_provider present in the cost map is registered) failed once the tencent
models were loaded. Add the tencent_models set, populate it from the cost map,
and expose it under the tencent key, mirroring deepseek.
* fix(tencent): import generic_cost_per_token from its canonical module
Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils
instead of the top-level litellm.cost_calculator dispatcher, which imports the
tencent cost module at load time. Removing the back-reference avoids the circular
import and matches how deepseek and the other providers source the helper.
---------
Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
A 401 while listing tools (a missing or expired per-user OAuth token, or an
upstream 401 for any auth_type) was swallowed to an empty tool list, so a
single-server client got a 200 with no tools and no WWW-Authenticate challenge
instead of a 401 it could re-authenticate against. Only oauth pass-through and
delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the
missing-token case for all of them, masked it.
The surface-vs-absorb decision now keys on the route, not the auth_type. An
upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError
regardless of auth_type, and the per-user OAuth challenge raised during client
creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is
converted to the same type in _get_tools_from_server. The challenge is scoped
to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a
re-auth signal and degrades to an empty list like any other non-auth error, and
the stdio-allowlist 403 (no challenge header) stays absorbed. The existing
routing then does the right thing: single-server routes turn the error into a
401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty
list so one unauthenticated server does not fail the whole listing.
On the UI tools page, an OBO (per-user authorization_code) server now shows the
Authorize gate when the list call returns 401, not only when no credential row
exists. The backend already refreshes a still-refreshable token on the list
call, so a 401 means there is no valid token and none could be minted (expired
with no usable refresh token), which is exactly when the user must reauthorize.
* test(e2e): add vertex_ai passthrough spend-log coverage
Port the de-flake of the SDK-based vertex spend test (#31689) into the
tests/e2e/llm_translation harness. The vertexai SDK intermittently ignored the
proxy api_endpoint override and billed Vertex directly, so the request never
reached LiteLLM and no spend was logged; driving native generateContent over the
shared transport always reaches the proxy, which the harness already guarantees.
The vertex deployment is added at runtime through /model/new with
use_in_pass_through rather than declared in the gateway config, and deleted on
teardown. That registers the deployment's service account for the /vertex_ai
route, so the passthrough call sends only its litellm virtual key in
x-litellm-api-key and no upstream bearer, and the proxy mints the Vertex token
itself. The credential is the one the proxy already holds, read from the same
VERTEXAI_CREDENTIALS/VERTEXAI_PROJECT env; the test never mints a token.
Asserts both that the forward succeeds and that a costed SpendLogs row lands
(vertex_ai provider, a gemini model, spend > 0, call_type pass_through_endpoint),
correlated by the x-litellm-call-id header.
* Update tests/e2e/llm_translation/test_vertex_passthrough_e2e.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/e2e/llm_translation/test_vertex_passthrough_e2e.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(bedrock): honor ttl for tool_config cache injection points
Pass cache_control_injection_points control.ttl through to Bedrock
toolConfig cachePoint blocks, matching message/system cache behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex
is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a
manual update for every new Claude release (it already silently missed
Sonnet 5 and Fable 5). Replace it with a lookup against
cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json,
which AWS docs confirm tracks the same 1h-TTL-capable model set.
Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried
that pricing field (their own regional variants didn't have it), which
would have made the JSON-driven check wrongly grant them 1h TTL support.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests
test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and
test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id
(...-20250514-v1:0) that never shipped. This passed under the old regex-based
is_claude_4_5_on_bedrock, which matched on substring alone, but fails now
that it looks up cache_creation_input_token_cost_above_1hr in
litellm.model_cost, since the fake id has no pricing entry.
Also force the bundled local cost map in both tests so ttl eligibility reads
this branch's pricing data instead of the network-fetched main copy, which
lacks the fix until merge.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): restore cache and tool config compatibility
* fix(bedrock): preserve Sonnet 5 parallel tool config
* fix(bedrock): decouple parallel tool support from cache ttl
* refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns
Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and
bedrock_converse_supports_strict_tool_schemas (dead code) with a
supports_parallel_tool_use_config key in model_prices_and_context_window.json,
matching how is_claude_4_5_on_bedrock already reads
cache_creation_input_token_cost_above_1hr from the pricing JSON.
New models pick up parallel tool use support automatically when their
pricing entry ships with the key set, with no code change required
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test
anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in
model_prices_and_context_window.json, so
bedrock_converse_supports_parallel_tool_use_config returned False and the
test died with KeyError on additionalModelRequestFields. Use
jp.anthropic.claude-opus-4-7, a real entry that carries
supports_parallel_tool_use_config without 1h-TTL cache pricing, which is
exactly the decoupling this test exists to cover
* test(utils): allow supports_parallel_tool_use_config in pricing schema
The misc unit test job validates model_prices_and_context_window.json
against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects
unknown keys. Add the supports_parallel_tool_use_config key this PR
introduced so test_aaamodel_prices_and_context_window_json_is_valid
passes again
* fix(bedrock): preserve ttl for regional claude models
* fix(bedrock): fall back to base model entry when regional pricing lacks capability fields
Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit
cache_creation_input_token_cost_above_1hr shadowed the base entry that has it,
so is_claude_4_5_on_bedrock returned False and requested cache ttl values were
dropped for those deployments. Both capability lookups now consult the full
model id and the region-stripped base entry, matching the coverage of the old
name-pattern list. Also restores ToolBlock keyword construction for the
tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every
supported Python version
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
