mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-06 02:48:13 +00:00
2982 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
a5b75e8bab
|
fix(ui): keep team Organization optional for proxy admins in single-org setups (#30861)
The Create Team form auto-selected, disabled, and required the Organization field whenever exactly one organization existed, regardless of role. For a proxy admin the organization is optional, so single-org setups could not create a standalone team even though the field is presented as optional. Gate the single-org preselect, the disabled state, and the restrictive help text on the org-admin role so they apply only to org admins, who must scope a team to their organization. Proxy admins now keep an optional, clearable, empty organization field regardless of how many organizations exist, matching the multi-org behavior. The /team/new endpoint already accepts a null organization, so this was a UI-only restriction. |
||
|
Mateo Wang
|
0d08a57dc4
|
fix(ui): clarify OpenAI-compatible provider dropdown labels (chat vs legacy completions) (#31046)
* fix(ui): clarify OpenAI-compatible provider dropdown labels (chat vs legacy completions) * style: change labeling to be clearer |
||
|
ryan-crabbe-berri
|
ee5b2a367d
|
fix(ui): label request logs column "Key Alias" to match filter (#31037)
The request logs table column displayed "Key Name" while its accessor (metadata.user_api_key_alias) and the corresponding filter both use the "Key Alias" label; this aligns the column header with that naming. |
||
|
ryan-crabbe-berri
|
26da56fbb6
|
feat(ui): add Amazon Bedrock Mantle to the Add Model provider dropdown (#31034)
The Add Model provider dropdown is driven by provider_create_fields.json (served at /public/providers/fields), and Bedrock Mantle had no entry, so it could not be selected even though the backend provider, its models, and the UI enum/logo mappings already existed. Add a bedrock_mantle entry exposing the credential fields the provider actually honors: an optional bearer api_key for BYOK, the AWS SigV4 chain, a region, and an api_base override. Selecting it now populates the bedrock_mantle models from the cost map via the existing getProviderModels filter. Also resolve the provider logo when getProviderLogoAndName is given the enum key (e.g. BedrockMantle) rather than the slug, which the dropdown passes; previously only slugs that lowercase-matched their key (like bedrock) resolved a logo. |
||
|
ryan-crabbe-berri
|
19a29e0579
|
feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel (#31029)
* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel
A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.
* refactor(ui): centralize no-mcp-servers sentinel in a shared constant
The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.
* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes
Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.
|
||
|
Yassin Kortam
|
e70f7e2d7a
|
fix(ui): resolve user_id to email in Spend Per User usage chart (#30992)
The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and default_user_id) instead of emails. /user/daily/activity passed entity_metadata_field=None, so every user entity in the breakdown carried empty metadata; the chart could only fall back to the user_id. The frontend resolved labels from a separately paginated user list, so any spender not on a loaded page showed as a UUID. Resolve the email/alias for the user_ids actually on the page (mirroring how api key metadata is already resolved) and attach it to the entity metadata, so the chart labels each spender with their email and falls back to the UUID only when no email is on file. get_daily_activity gains an optional resolve_entity_metadata hook so the user endpoint can do this page-scoped lookup without loading the whole user table. Resolves LIT-3889 |
||
|
Yassin Kortam
|
bc5638b38e
|
fix(ui): stop per-model usage export from duplicating user spend across models (#30980)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
The "day-by-day by user and model" usage export overcounted spend by repeating each user-day total once per model. generateDailyWithModelsData iterated day.breakdown.models crossed with the entity's own api_key_breakdown and added the api key's full daily spend to every model, leaving the per-model object (modelData) unused and never matching the api key to the model. A user who called N models got N identical rows, so the column summed to N times the real spend; one reported export totaled $167,953 against a true $21,353 (7.87x). Attribute each model only the spend of the keys that actually used it by intersecting the entity's api keys with modelData.api_key_breakdown. This mirrors the already-correct pattern in activity_metrics.tsx, so per-model rows now sum back to the user-day total and models a user never called no longer appear. The existing tests passed only because their mocks omitted models[*].api_key_breakdown and never asserted numeric values. The mocks now match the real /user/daily/activity payload, and new tests assert that per-model spend sums to the user-day total and that uncalled models are omitted; both fail on the old code. Resolves LIT-3866 |
||
|
tin-berri
|
963816c00e
|
fix(mcp): stop exposing MCP server URLs on the AI Hub and public hub API (#30902)
The AI Hub MCP Hub listed each MCP server's upstream URL in a table column and in the server Details modal, on both the authenticated dashboard and the public hub. The unauthenticated GET /public/mcp_hub endpoint also returned the url field via MCPPublicServer, so the upstream address was readable by any client even with the column gone. These surfaces are for end users discovering available servers, so the gateway-internal endpoint should not be exposed there. Drop the URL column from both MCP Hub tables and the URL field from both detail modals, and remove url from MCPPublicServer so /public/mcp_hub no longer serialises it; schema.d.ts is updated to match. The public hub MCPServerData interface no longer declares url since the response omits it. Admin surfaces that configure the endpoint (the MCP server management page, the submissions review tab, the make-public form) and the authenticated /v1/mcp/server endpoint are untouched. publicMCPHubColumns is lifted to a module-level export so both hub column sets get a mutation-killing regression test, public_model_hub.test.tsx covers the details modal hiding the url, and test_public_endpoints.py asserts /public/mcp_hub never returns url even when the server has one. |
||
|
Krrish Dholakia
|
accbd7e587
|
feat: litellm plugin architecture v2 (#30688)
* feat: plugin architecture — toggle between AI Gateway and external plugins
Adds a generic plugin system so any external service can register with
litellm and appear as a mode in the UI alongside the AI Gateway.
Backend (litellm/proxy/plugin_routes.py — new):
- GET /api/plugins: returns registered plugins from config; returns
plugin_key only to authenticated requests
- ANY /plugin-proxy/{name}/{path}: reverse proxies API calls to plugin
Config:
general_settings:
plugins:
- name: my-plugin
display_name: My Plugin
url: https://my-plugin.example.com
plugin_key: sk-... # plugin auth key, passed to iframe
UI:
- PluginModeContext.tsx: fetches /api/plugins, persists mode to localStorage
- leftnav.tsx: mode switcher dropdown at top of sidebar; plugin mode shows
plugin-specific nav items
- layout.tsx: renders iframe to plugin URL in plugin mode; passes plugin_key
as ?token= for auto sign-in
Plugin contract: expose GET /api/plugin-manifest returning
{ name, display_name, nav_items[], capabilities[] }. No litellm changes
needed to add new plugins — config only.
Reference implementation: LiteLLM-Labs/litellm-agent-control-plane
* feat: add Plugins tab to Admin Settings UI
Allows admins to add/edit/delete plugin registrations directly in the
litellm UI under Admin Settings > Plugins, instead of editing config.yaml.
Uses existing /config/field/update API to persist to general_settings.plugins.
Each plugin entry has: name (identifier), display_name, url, plugin_key.
* fix(ci): black, prettier, eslint, async-client violations
- Black: format plugin_routes.py and proxy_server.py
- Prettier: format PluginModeContext.tsx and PluginSettings.tsx
- ESLint: replace raw fetch() with createApiClient in PluginModeContext
- ESLint: use lazy useState initializer to read localStorage instead of
calling setModeState inside useEffect (react-hooks/set-state-in-effect)
- code-quality: replace httpx.AsyncClient per-request with
get_async_httpx_client() shared client (avoids +500ms overhead)
* fix(ci): schema.d.ts regen, Black proxy_server.py, ApiClientConfig fix
- Regenerate schema.d.ts for new /api/plugins routes
- Re-run Black 26.3.1 on proxy_server.py (matches CI version)
- Fix PluginModeContext: createApiClient requires getBaseUrl field
* fix: security hardening + CI fixes
Security (Greptile 1/5 → addressing all 3 findings):
- plugin_routes.py: add Depends(user_api_key_auth) to both /api/plugins
and /plugin-proxy/{name}/{path} — was an unauthenticated open relay
- plugin_routes.py: /api/plugins now returns plugin_key only to callers
with a valid litellm token (enforced by user_api_key_auth), not just
any header presence
- layout.tsx: replace ?token= URL param with postMessage(targetOrigin)
— token no longer exposed in browser history / logs / Referer headers
CI:
- backend/routes/allowlist.py: add /api/plugins and /plugin-proxy/ to
fix test_gateway_plus_backend_covers_full_app
- schema.d.ts: regenerated with enterprise routes included
- Black + Prettier formatting
* fix: regenerate schema.d.ts with enterprise routes included
Install litellm-enterprise workspace member before gen:api so audit and
other enterprise routes appear in the generated types, matching what CI
produces with uv sync --extra proxy.
* fix: exclude plugin routes from OpenAPI schema, restore upstream schema.d.ts
Both /api/plugins and /plugin-proxy/ are internal infrastructure routes,
not part of the public litellm API surface. Marking include_in_schema=False
prevents Python-version-dependent schema diffs from breaking the schema
sync check across different environments.
* fix: schema.d.ts - passing schema base + exact plugin route types from openapi-typescript
Use the CI-correct schema from a recently passing branch as base, then
inject plugin route entries (paths + operations) generated by
openapi-typescript from the plugin routes' OpenAPI spec. This avoids
Python-version-dependent formatting differences that made local gen:api
produce incorrect output.
* fix: schema.d.ts - insert plugin ops at correct route registration position
Plugin operations belong after delete_memory_v1_memory__key__delete
(memory_router is included immediately before plugin_router in proxy_server.py),
not after list_organization which is alphabetically but not registration-order.
* fix: schema.d.ts - correct op positions from hunk analysis
list_plugins_api_plugins_get goes after event_logging_batch op (hunk 1: line 33583).
plugin_proxy ops go after create_policy_policies_post (hunk 2: line 44634).
Previous location after delete_memory_v1_memory__key__delete was wrong.
* fix: schema.d.ts - proxy ops go before create_policy (after otel_spans)
* fix(security): restrict plugin_key to proxy_admin role only
Veria finding: plugin_key was returned to any authenticated caller.
Now only proxy_admin users receive plugin credentials in /api/plugins
response — regular internal users see plugin name/url but not the key.
* fix: update schema.d.ts docstring for list_plugins
* fix: clear plugin registry on config reload (Greptile medium)
register_plugins_from_config now replaces the registry instead of
merging, so plugins removed from config are unreachable immediately
without requiring a process restart.
* fix(security): encrypted token exchange for plugin iframe — no raw litellm credential exposure
The dashboard was sending the user's litellm bearer token to the plugin
iframe via postMessage, allowing a compromised plugin to act as that user.
Fix:
- GET /api/plugins/auth-token: proxy encrypts caller token with Fernet
keyed from LITELLM_SALT_KEY, returns ciphertext only
- UI postMessages the ciphertext (not raw token) to the iframe
- Plugin decrypts server-side with same LITELLM_SALT_KEY via POST /api/plugin-auth
- Raw litellm credential never leaves the proxy in plaintext
Additional hardening already in place:
- /plugin-proxy/* strips Authorization header, injects plugin_key instead
- plugin_key only returned to proxy_admin role via /api/plugins
- Plugin registry cleared (not merged) on config reload
Adds docs/plugin_architecture.md with plugin integration guide.
* fix(code-quality): use get_async_httpx_client in plugin_proxy
* fix: add /api/plugins/auth-token to schema.d.ts
* fix: use apiClient for auth-token fetch, copy correct layout.tsx and PluginModeContext
- Replace raw fetch() with createApiClient (fixes no-restricted-syntax ESLint rule)
- Copy correct layout.tsx with encrypted token + postMessage approach
- Copy correct PluginModeContext.tsx with accessToken prop injection
- Update schema.d.ts with auth-token path and operation entries
* fix: add plugin_auth_token operation to schema.d.ts
* fix(security): strip cookie/set-cookie + fix compressed response headers
Veria High: cookie header was forwarded to plugin backends allowing
capture of litellm JWT session cookies. Strip cookie on requests.
Strip set-cookie from responses so plugins cannot overwrite litellm
session cookies.
Greptile P1: httpx decompresses responses but resp.headers still
contained Content-Encoding/Transfer-Encoding/Content-Length from the
wire. Forwarding these caused double-decompression and length errors.
Now filtered via _RESPONSE_STRIP before returning to the browser.
* fix: update plugin_key help text — no more ?token= reference
* fix(security): disable follow_redirects to prevent SSRF
follow_redirects=True allowed a plugin backend to return a 3xx to an
internal URL, causing the proxy to fetch that internal service and relay
the response. Disabled: clients handle their own redirects.
* fix: forward user identity headers to plugin to address confused deputy
Plugins receive X-LiteLLM-User-Id and X-LiteLLM-User-Role so they can
enforce their own per-user access control before acting on requests that
arrive with the shared plugin_key credential.
* fix(security): restrict /plugin-proxy/* to proxy_admin role
Closes the confused deputy gap: regular users could invoke any plugin
endpoint using the shared plugin_key as a bearer credential. Now only
proxy_admin callers can use the plugin proxy route.
Plugin UIs communicate with the plugin service directly via the iframe
(using the encrypted token exchange); this proxy route is for
administrative/server-to-server access only.
* fix: update schema.d.ts for admin-only proxy route docstring
* fix(bug): use PassThroughEndpoint instead of None for get_async_httpx_client
get_async_httpx_client(llm_provider=None) raises TypeError — the function
concatenates the provider string and None is not a str. Use
httpxSpecialProvider.PassThroughEndpoint, the enum value used by other
internal proxy pass-through routes.
* fix(security): add 30s TTL to encrypted plugin auth tokens
Veria medium: encrypted tokens had no expiry, allowing indefinite replay.
Fernet embeds a timestamp; decrypt_token now passes ttl=30 so tokens
older than 30 seconds are rejected even with a valid HMAC.
Plugin's /api/plugin-auth must call litellm within 30s of the iframe
receiving the postMessage — normal browser behavior, tight enough to
close the replay window.
* feat(ui): topnav plugin switcher, embed plugins at their root
Builds on the plugin architecture already on this branch (encrypted-token
postMessage handshake, /api/plugins, PluginSettings) and removes the parts of the
embed that assumed a specific plugin's shape.
The mode switcher moves out of the sidebar into the topnav and lists AI Gateway
plus each registered plugin by its display_name. Selecting a plugin hides
litellm's sidebar entirely and renders the plugin full-bleed at its root url; the
plugin draws its own navigation inside the iframe. This drops the hardcoded
"Agent Control Plane" label and the hardcoded Sessions/Agents/Routines/... nav
groups (agentControlPlaneMenuGroups / acpPagePaths) that only matched the agent
platform and 404'd for a plugin that serves only / (e.g. the chat UI). The
encrypted-token postMessage flow is unchanged.
Note: embedding at root means a plugin must route internally from /; plugins that
previously relied on the /sessions entrypoint should redirect from their root.
* fix(security): audience-scoped identity claim replaces litellm token
Veria: shared LITELLM_SALT_KEY with plugins + encrypting user bearer token
created delegation/impersonation risk.
Architecture change:
- /api/plugins/auth-token now issues a plugin-scoped identity CLAIM
{user_id, user_role, plugin, exp} encrypted with HMAC(LITELLM_SALT_KEY, plugin_name)
- Each plugin holds only its own HMAC-derived key; cannot forge claims for
other plugins or recover LITELLM_SALT_KEY
- Claim contains NO litellm bearer token — compromised plugin learns caller
identity only, cannot act as that user against the proxy
- 30s TTL enforced in both Fernet header and explicit exp field
- LAP /api/plugin-auth verifies claim, returns its own master key to browser
(LAP key never exposed without valid claim)
* fix(plugins): allow registering plugins from the admin UI
Adding a plugin in the UI POSTs general_settings.plugins to /config/field/update,
which rejected it with "Invalid field=plugins passed in." because `plugins` was
not a field on ConfigGeneralSettings. Add a typed PluginConfig model and a
`plugins` field so the update validates and persists.
The in-memory plugin registry only refreshed at startup, so a plugin added via
the UI did not appear in /api/plugins (the view switcher) until a restart. Refresh
the registry from the new general_settings whenever the plugins field is updated.
While here, type the registry as dict[str, PluginConfig] instead of raw dicts so
list_plugins and plugin_proxy access typed attributes.
Fix the Plugin Key field copy: it is optional and only used to authenticate
litellm's server-side reverse proxy to a plugin's own backend
(/plugin-proxy/<name>/*). It is not involved in iframe auth, which forwards the
user's litellm token. Plugins that use the forwarded token leave it blank.
* fix: regenerate schema.d.ts with PluginConfig type and updated auth-token endpoint
* fix: use CI-compatible schema base for plugin entries
* fix(plugins): load DB-persisted plugins on startup
Plugins added through the admin UI are saved to DB general_settings, but the
registry only initialised from the YAML config at boot, so UI-added plugins
disappeared from the view switcher after a restart (the Plugins table still
listed them since it reads the DB directly). Refresh the registry from the DB
general_settings when it is merged in at startup.
* fix: add PluginConfig schema, plugins field, fix list_plugins return type
* fix: correct PluginConfig and plugins field positions in schema
* fix: correct plugins field position in schema (after pass_through_endpoints)
* fix: update PluginConfig.plugin_key description to match _types.py source
* fix: move plugins field after pass_through_request_timeout (correct alphabetical position)
* fix: redact plugin_key in config/field/info response
Veria medium: proxy_admin_viewer could read plugin_key via
GET /config/field/info?field_name=plugins. Now plugin_key is
replaced with *** in the response regardless of caller role.
The credential is only usable server-side.
* fix(security): correct plugin docs salt-key guidance, drop iframe clipboard-read
Address the two open Veria findings on the plugin architecture.
The plugin docs told external services to decrypt the iframe auth payload
with the proxy's LITELLM_SALT_KEY directly. That is both insecure and wrong:
the running code derives a per-plugin key as HMAC-SHA256(LITELLM_SALT_KEY,
plugin_name) and ships only a short-lived identity claim with no litellm
bearer token. Sharing the master salt would let a compromised plugin decrypt
any litellm secret recovered from a dump or backup. Rewrite the doc to match
the implementation: the proxy computes the per-plugin key once and provisions
it as a dedicated secret, the plugin validates the claim's audience and 30s
TTL, and LITELLM_SALT_KEY never leaves the proxy. Also refresh the now-stale
module and UI comments that still described the old shared-key token flow.
Drop clipboard-read from the plugin iframe's allow attribute so an untrusted
plugin can no longer read the user's clipboard; clipboard-write is retained.
* fix(ci): modernize PluginConfig typing, refresh budget baselines via merge
* fix(plugins): close iframe auth race and empty-plugins mode fallback
Address the two open Greptile behavioral findings.
The iframe auth handshake only posted the encrypted claim on the iframe's
`load` event. When the auth-token fetch resolved after the iframe had already
loaded, that listener never fired again and the plugin never received the
claim. Send the claim immediately as well as on subsequent loads so both
orderings are covered.
The plugin mode fallback guarded on a non-empty plugins list, so removing all
plugins left a user stranded on a stale mode with a blank iframe instead of
returning to the AI Gateway. Track a loaded flag and fall back to ai-gateway
once plugins have loaded whenever the stored mode is no longer registered,
including the empty-list case.
Add a PluginModeContext regression test covering the empty-list fallback and
the still-registered path.
* chore: re-trigger CI (GH Actions missed the prior head; re-run flaky live-API suites)
* fix(plugins): scope iframe auth claim to the active plugin
The iframe auth-token fetch omitted plugin_name, so the proxy always issued a
claim encrypted under the default plugin's per-plugin key. For any other active
plugin the iframe received a claim it could not decrypt and sign-in silently
broke, and because the cached claim was posted to whichever plugin was mounted,
a compromised iframe could replay the default plugin's claim. The active
plugin's name was also missing from the fetch effect's dependencies, so
switching plugins never refreshed the claim.
Request the claim with the active plugin's name, re-fetch when the active
plugin changes, and only deliver a claim while it still matches the mounted
plugin so one plugin's claim is never replayed to another.
* fix(plugins): never overwrite a stored plugin_key with its redaction placeholder
/config/field/info redacts every plugin_key to "***", so an admin editing a
plugin in the settings UI posted that placeholder straight back and the update
handler persisted "***" as the real credential, permanently destroying the key.
Preserve the stored credential on update: a blank or redacted plugin_key now
sources the existing key from the saved config, only a real value replaces it,
and a placeholder with no stored key is dropped rather than written. The edit
modal also starts the key field blank so an untouched save keeps the current
key, with the field labelled accordingly.
* fix(security): sandbox proxied plugin responses on the dashboard origin
The /plugin-proxy reverse proxy returned the plugin's body and content-type on
the litellm dashboard origin, so a compromised plugin could serve an HTML/JS
document that a proxy_admin navigates to and have it execute with the admin's
session against same-origin management APIs.
Force every proxied response inert: set Content-Security-Policy: sandbox (opaque
origin, scripts disabled) and X-Content-Type-Options: nosniff, applied after the
plugin's own headers so they cannot be overridden. The header construction moves
to a pure helper with a unit test covering the sandbox enforcement and the
existing wire/cookie header stripping.
* fix(plugins): recover to ai-gateway when the plugins fetch fails
The loaded flag was only set on a successful /api/plugins response, so when the
fetch failed a user with a plugin mode stored in localStorage stayed on the
blank plugin placeholder with no switcher to escape. Mark loaded in a finally
so the stored mode still falls back to ai-gateway on failure, and add a
regression test for the failed-fetch path.
* fix(security): never return plugin_key from /api/plugins
The plugin list endpoint returned the plaintext plugin_key to proxy_admin
callers, and the dashboard fetches /api/plugins on every load into React state,
so the credential was exposed to DevTools, memory snapshots, and any same-origin
script. The browser never uses the key; the proxy injects it server-side from
the registry and admin key management runs through the redacted
/config/field/info path. Drop plugin_key from the response for every caller and
update the regression test to assert it is never returned.
* chore(ui): regenerate schema.d.ts for updated list_plugins docstring
* fix(security): strip every litellm auth header before forwarding to plugins
The plugin reverse proxy only removed Authorization and x-api-key, but
user_api_key_auth also authenticates a caller via API-Key, x-goog-api-key,
Ocp-Apim-Subscription-Key, x-litellm-api-key, and any configured custom key
header. A malicious plugin could lure a proxy_admin into calling
/plugin-proxy/... with the litellm key in one of those headers; the request
authenticated locally and then forwarded the same key to the plugin, letting it
impersonate the admin.
Add a canonical SpecialHeaders.litellm_credential_header_names() that the auth
header enum is the single source for, and strip that whole set plus the live
general_settings.litellm_key_header_name from every forwarded request. New auth
headers added to SpecialHeaders are now stripped automatically. Regression tests
cover each credential header, the custom configured header, and the canonical
list's contents.
|
||
|
Mateo Wang
|
15aa40b36e
|
test(ui): isolate OldTeams delete-warning tests from leaked mock (#30871)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
The deprecated OldTeams component takes only accessToken, userID, userRole and premiumUser; it ignores the teams prop these tests passed and instead populates its table from the mocked teamListCall. The delete-warning block never set teamListCall, and vi.clearAllMocks clears call history but not implementations, so the table rendered the "Legacy Team" (keys.length 2) left behind by the previous block's last test. Both delete tests therefore ran against that leaked team: the keys-present case passed only because the leaked count happened to be 2, and the no-keys case rendered the same warning it asserted should be absent, so it failed. Seed the team through the channel the component actually reads (teamListCall) and drop the props it never consumes, so each test renders exactly the team it declares. The keys-present case now uses a distinctive count so it can no longer pass on a coincidental leak |
||
|
yuneng-jiang
|
ea17236a1e
|
fix(ui): warn that team models are deleted in the delete-team modal (#29990)
The delete-team confirmation modal warned that a team's keys would be deleted but said nothing about models. #29977 made team deletion also delete the team's BYOK models, so the modal copy was understating what gets removed. The warning banner now mentions models alongside keys, and the always-shown confirmation message does too so a team that has models but no keys (the banner only renders when keys exist) still gets warned. |
||
|
yuneng-jiang
|
60dc8420ed
|
fix(ui): repoint dead usage guide link to cost tracking docs (#30859)
The "View Usage Guide" button on the legacy Usage page (shown when DISABLE_EXPENSIVE_DB_QUERIES is set, i.e. SpendLogs has 1M+ rows) linked to docs/proxy/spending_monitoring, which was removed from the docs and now returns 404. Point it at docs/proxy/cost_tracking, which is live. Fixes LIT-2724 |
||
|
ryan-crabbe-berri
|
e4a53f50de
|
chore: remove in-product survey and Claude Code feedback nudges (#30773)
Delete the in-product survey and Claude Code feedback prompts end to end. Frontend: remove the src/components/survey/ module, the index page's nudge state/effects/handlers, the getInProductNudgesCall helper, and the orphaned "Disable UI nudges" toggle in the admin UI Settings page; prune the stale eslint-suppressions entries. Backend: remove the now-dead /in_product_nudges route, the InProductNudgeResponse type, and the disable_ui_nudges UI setting (Field + allowlist). Nothing read it for logic and the UISettings model is extra="allow", so existing stored configs are unaffected (the value is just no longer surfaced). schema.d.ts is regenerated and the two tests covering the removed route/setting are dropped. |
||
|
Sameer Kankute
|
4c25b7a13d
|
chore: litellm oss staging (#30745)
* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708) OpenAI GPT-5 models require max_completion_tokens >= 16. Health checks were using 5 (proxy/health_check.py) and 10 (health_check_helpers.py), causing failures on GPT-5 models. Fixes #23836 * fix: increase health check max_tokens from 5 to 16 (#23836) (#26610) GPT-5 models enforce a minimum of 16 for max_output_tokens. The current default of 5 still causes health checks to fail for these models. Bump the non-wildcard default to 16 — the smallest value that satisfies all known provider minimums while keeping health checks lightweight. Also tightens the wildcard test assertion from a weak disjunctive check to strict key-absence. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696) * fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema. * fix: remove async keyword from test. * fix: make Bedrock Mantle Responses routing data-driven per model (#30700) * Make Bedrock Mantle Responses routing data-driven per model Route Bedrock Mantle models to the native Responses API based on each model's price-map capability signal instead of a hardcoded model-name heuristic, and derive the OpenAI-compatible base path segment per model. Responses dispatch now selects the native config when the model advertises responses support (/v1/responses in supported_endpoints, or mode=responses), both overridable via register_model and proxy model_info. This enables native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing chat-completions emulation. Capability is per-model, so gpt-oss-120b routes natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss substring. The wire path is a separate concern, driven by the existing use_openai_responses_path flag rather than a model-name match: gpt-5.x and gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat config now derives its base from the same flag, fixing gemma-4 chat-completions requests that previously went to /v1 instead of /openai/v1. Cost maps: add supported_endpoints to the gpt-oss entries (responses for the non-safeguard variants, chat-only for safeguard) and supported_endpoints + use_openai_responses_path to all three gemma-4 entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: move capability helper into bedrock_mantle package Move the Responses capability check out of utils.py into litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses, alongside its companion wire-path helper mantle_base_segment. Both are now pure functions of (model, model_cost): the price-map mode/supported_endpoints read replaces the get_model_info call, so the rules are unit-testable without patching global state and the Bedrock Mantle package is self-contained. Use str | None instead of Optional[str] on the new signatures to satisfy the ruff UP045 strict-rule gate. Add direct unit tests for both helpers. Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b now legitimately supports Responses, so it can no longer be the "None after restore" vehicle; use the chat-only safeguard variant, which isolates the register/restore effect from the model's own capability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653) The tiered cost calculator resolved a tier's per-token cost with `tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or` short-circuits on any falsy value, a tier that legitimately prices a component at 0.0 (e.g. a free-cache-read tier with cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated as missing and silently billed at the full fallback rate (input_cost_per_token / output_cost_per_token). The flat-pricing path in the same module already handles this correctly with an `is None` guard. Resolve tier costs through a small helper that mirrors it, so 0.0 is honored at both the in-range and overflow sites. No shipped model currently has a 0.0 tier cost, so this is a latent defect; the fix makes the tiered path consistent with the flat path and prevents over-charging the first time such a tier appears. Adds unit tests covering the in-range and overflow paths, and drops an unused import flagged by ruff in the touched test file. * feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507) * fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618) In the messages->chat/completions bridge, translate_anthropic_tools_to_openai merged every non-mapped tool key into the function parameters dict. The Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object' -> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type). Exclude 'type' from the passthrough. Fixes #30557. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * feat(bedrock): support file content retrieval for batch output files (#30595) Implements transform_file_content_request and transform_file_content_response in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch files. The request transform resolves the file id (direct s3:// URI or base64 unified id) to its S3 object, validates bucket and key prefix against the server-configured bucket, and SigV4-signs an S3 GetObject using the same credential and region resolution as the existing upload path. The credential and region params are validated into a typed model at the boundary, so the only untyped values left are the botocore signing primitives. Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries s3_bucket_name (previously dropped when building deployment credentials) and the managed-files hook passes the deployment credential snapshot when routing afile_content, so unified-id content retrieval works with per-model bucket config instead of only the AWS_S3_BUCKET_NAME env var. Preserves managed-file access control: the proxy file-content endpoint now rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the owner/team check that only runs for unified ids and let a caller read another tenant's batch output by its object key. Managed outputs are reachable only through their unified file id. The afile_content "not found" error now reports the caller's unified id rather than the resolved internal S3 URI. Fixes #16186, #15563 * fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646) * fix(oci): map Cohere tool array/object params to lowercase builtins OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare "List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema arrays. MLflow {{trace}} judges trip this: their tools (get_root_span, get_span) take an attributes_to_fetch array. The lowercase builtins list/dict are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but both are lowercased for consistency). Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest). Adds a unit regression on the transformed parameterDefinitions plus a gated integration test exercising an array-param tool end to end. * fix(oci): make Cohere agentic tool-calling continuation work Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges drive once a tool has been executed and its result is fed back. Request side: litellm pulled the last user message into the top-level `message` and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that ("cannot specify message if the last entry in chat history contains tool results"), and an empty message alone is rejected too ("message must be at least 1 token long or tool results must be specified"). OCI carries the current turn's results in a dedicated top-level `toolResults` field. The Cohere transform now sends an empty message, keeps the user turn in chatHistory, and puts the results in `toolResults`, matching the langchain-oracle reference. Tool results are no longer represented as chatHistory entries. Response side: tool-grounded answers come back with citations carrying `documentIds` (camelCase) and no `document_ids`, which made the required `CohereCitation.document_ids` field fail validation and sink the whole response parse. Those citations are never surfaced, so the field (and CohereSearchQuery's generation_id) is now optional. Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest), single and multi-round tool loops. Adds unit regressions on the transformed request shape and on citation parsing, plus gated integration tests for the continuation. * feat: integrate Repelloai Argus guardrail (#30673) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * refactor(guardrails/repello): clean up typing and remove lint-any workarounds - Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout - Use dict[str, object] instead of bare dict in all signatures - Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly - Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel - Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks - Use TypeAdapter.validate_json() instead of response.json() + manual dict construction - Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any - Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check - Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType - Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel] * fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning - Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate - Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks * refactor: modifications for lint check * feat: add Pinstripes as an OpenAI-compatible provider (#30567) * feat: add Pinstripes as an OpenAI-compatible provider Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.) with per-token pricing and no subscriptions. Changes: - `litellm/llms/openai_like/providers.json`: register pinstripes with base_url, api_key_env, and max_completion_tokens→max_tokens mapping - `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders - `litellm/constants.py`: add to openai_compatible_providers and openai_compatible_endpoints lists - `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect provider when api_base is "https://pinstripes.io/v1" - `provider_endpoints_support.json`: document supported endpoints - `tests/`: 7 unit tests covering provider registration, resolution, URL auto-detection, api_base override, and Router config Usage: import litellm response = litellm.completion( model="pinstripes/ps/glm-4.5-air", messages=[{"role": "user", "content": "Hello"}], api_key=os.environ["PINSTRIPES_API_KEY"], ) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): resolve Greptile P1 review comments - Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works - Set responses: false in provider_endpoints_support.json — not actually wired up - Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): add api_base_env and correct responses capability - Add api_base_env: PINSTRIPES_API_BASE to providers.json - Set responses: false in provider_endpoints_support.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): wire up Responses API — add supported_endpoints Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so JSONProviderRegistry.supports_responses_api returns true correctly, matching what provider_endpoints_support.json advertises. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(pinstripes): enable embeddings endpoint Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings. Add /v1/embeddings to supported_endpoints and set embeddings: true. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json Matches the file's existing convention. Flagged by Greptile review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): set a2a: false — A2A protocol not implemented All comparable JSON-configured providers (tensormesh, parasail, empiriolabs, libertai, neosantara) have a2a: false. Pinstripes does not implement the Google A2A protocol, so this should be false to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(rag): attach existing OpenAI file ids (#30628) * fix(rag): attach existing OpenAI file ids * chore: use modern typing in rag ingest fix * chore: retrigger ci * fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341) cache_control_injection_points was only consumed by the chat/completions prompt-management hook; on the native Anthropic /v1/messages path it was forwarded unused, so deployment-level cache injection was silently dropped (cache_creation_input_tokens stayed 0 for Anthropic-native clients). Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject cache_control at block level for system / tools / message locations (the only forms /v1/messages accepts), wire it into the native anthropic_messages handler, and pop the param so it does not leak upstream as an unknown field. A {location: message, role: system} config is redirected to the top-level system prompt so the same YAML works on both endpoints. Injection respects Anthropic's 4-block cache_control limit shared across system, tools, and messages: client-supplied markers count toward the cap and are never overwritten, a slot is reserved per Bedrock tool_config point, and injection stops once the budget is exhausted. Locations this path cannot represent (tool_config) are forwarded downstream instead of being silently consumed, mirroring get_chat_completion_prompt's remaining_points pass-through. Built on litellm_internal_staging. Refs BerriAI/litellm#30293 * fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522) * fix(proxy): release budget reservation on cancel when no chunk was delivered The pre-call budget reservation increments the cross-pod spend counter by a request's worst-case cost, then reconciles it on success (cost callback) or error (failure hook). A client disconnect or timeout cancels the request and surfaces as CancelledError / GeneratorExit, which neither path catches, so the reservation leaks. Under a retry storm the leaked holds accumulate, pin the counter above real spend, and return spurious 429 "Budget has been exceeded" to keys whose spend is far below budget; the counter only recovers when its TTL lapses, so the failure is intermittent and self-healing. Release the reservation in async_streaming_data_generator (which the Anthropic and Google SSE generators delegate to) on the (CancelledError, GeneratorExit) path, alongside the existing max_parallel_requests release. release_budget_ reservation_on_cancel runs under asyncio.shield so it completes despite the in-progress cancellation, is guarded by the reservation's finalized flag, and swallows a failing release so it cannot replace the in-flight cancellation. The refund is gated on whether a chunk reached the client. The flag is set immediately before the yield, after the slow-path hook await: an async generator suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk sees it True (keep the hold), while a cancellation during the slow-path await leaves it False (refund, nothing sent). A non-streaming cancellation delivers nothing and a completed non-streaming response is reconciled by the success callback, so neither needs a release here. Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): reconcile a cancelled reservation to input cost, not zero A streaming request cancelled before the first chunk previously reconciled its reservation to zero and finalized it. But by the time the generator is consuming the response the provider call was already dispatched, so the input tokens were billed even though no chunk reached the client, and the success/failure cost callbacks are skipped on cancellation. Refunding to zero let a caller send an expensive request and abort pre-token to dodge the input charge. Compute the request's input-token cost at reservation time and reconcile the cancelled reservation to it instead of zero. The worst-case output portion of the reservation is still released (so a legitimate mid-flight cancellation no longer pins the counter and 429s the key), while the input the provider already processed is charged. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(caching): encode object name in GCS cache GET path (#30378) GCS cache reads always missed when gcs_path was set. The GET methods interpolated the object name directly into the URL path, while the GCS JSON API requires it to be URL-encoded (a "/" must be sent as %2F). With gcs_path configured the object name is "<prefix>/<sha256>", so the raw slash produced a malformed object path and GCS returned 404. httpx does not raise on 4xx, so the status_code == 200 check fell through and get/async_get returned None, silently missing on every read. Without gcs_path the key has no slash, which is why this went unnoticed. Wrap the object name with urllib.parse.quote(..., safe="") in get_cache and async_get_cache. Apply the same encoding to the name= query parameter in set_cache and async_set_cache so the key written matches the key read back. Adds regression tests asserting the GET path and SET query are encoded (%2F) when gcs_path is set, for both sync and async paths; these fail on the unpatched code. Fixes #30377 * chore: add soniox stt-async-v5 model (#30672) * fix(proxy): include model group aliases in v1 model info (#30626) * Include model group aliases in v1 model info * Fix model info alias implementation * removed extra blank line * chore: rerun CI * fix(lint): remove redundant noqa directive in proxy_cli.py * fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme * Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme" This reverts commit |
||
|
Sameer Kankute
|
e33e2917c6
|
chore: litellm oss 170626 (#30637)
* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (#30089) * fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets, /realtime/calls and their /v1 + /openai/v1 variants) to LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as LLM API routes. Without this, non-admin virtual keys received 401 'Only proxy admin can be used to generate, delete, update info for new keys/users/teams' when calling these endpoints. Fixes #29923 * fix(proxy): validate session.model for realtime routes in model-access check The GA Realtime WebRTC HTTP routes resolve the effective model from the nested session.model (falling back to the top-level model), but the auth layer's get_model_from_request() only extracted the top-level model. A model-restricted virtual key could therefore place a disallowed model in session.model, leave the top-level model unset, and skip can_key_call_model() entirely - obtaining an ephemeral token for a model it is not allowed to use. Extract session.model for the realtime client_secrets/calls routes so the model-access check runs against the model the request will actually use. Legitimate callers are unaffected; their permitted model still validates. Relates to https://github.com/BerriAI/litellm/issues/29923 * fix(proxy): classify realtime transcription_sessions routes as LLM API routes Add the GA Realtime WebRTC transcription_sessions HTTP routes to openai_routes so is_llm_api_route() returns True for them, matching the client_secrets and calls routes already fixed. These endpoints are registered with user_api_key_auth in realtime_endpoints/endpoints.py, so without this a non-admin virtual key calling POST /v1/realtime/transcription_sessions would hit the admin-only 401 branch. Extends the regression test parametrization accordingly. --------- Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com> * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (#30272) * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models * fix(proxy): degrade /v1/models gracefully when model-group lookup fails --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: sort tiered token-cost thresholds numerically (#30375) * fix: sort tiered token-cost thresholds numerically _get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a lexicographic sort, so for tiers whose thresholds have different digit lengths (e.g. 90k vs 128k) a request crossing both was billed at the lower tier that sorted first. Sort by the parsed numeric threshold instead, so the highest tier the request actually crosses is applied. * refactor: reuse _parse_above_token_threshold for inline threshold parse --------- Co-authored-by: Eric (GabiDevFamily) <271972409+santino18727-debug@users.noreply.github.com> * fix(openai): preserve cache_control for openai-compatible custom endpoints (#30387) * fix(openai): preserve cache_control for openai-compatible custom endpoints * fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation * fix(proxy): drain all daily-spend batches per flush cycle (#30281) (#30505) * fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (#30545) * fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers * test(types): add regression test for internal rate-limit fields in all_litellm_params * fix(init): add bool type annotation to suppress_debug_info (#30531) Module-level `suppress_debug_info = False` had no annotation, so strict type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to `True` (as done in proxy_server.py and router.py) then fails with an invalid-assignment error. Annotate it as `bool` to match every other flag in this module. * fix: coalesce null aggregates in update_metrics for no-spend keys (#29945) * feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (#30006) * feat(team_endpoints): Add query parameter key_limit to /team/info * feat(team_endpoints): update schema.d.ts to include the new query parameter * feat(team_endpoints): add tests for limitting key count in /team/info response * feat(team_endpoints): Apply suggestions from greptile * Set greater-than constraint on key-limit * Fix type * fix(router): release aiohttp connection when stream iteration ends abnormally (#30271) * fix(router): release aiohttp connection when stream iteration ends abnormally A streaming response that terminates with a mid-stream read timeout, a task cancellation (client disconnect), or GeneratorExit never closed the underlying aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body EOF, so each abnormally terminated stream permanently leaked one slot from the shared TCPConnector pool. During a backend traffic spike the pool drains; once exhausted every subsequent request to that host waits for a slot, times out and surfaces as a 408, indefinitely, even after the backend recovers. Only a proxy restart cleared the in-memory sessions, which matched the reported symptom of a router stuck returning 408 for a healthy vLLM backend. Close the response in a finally clause when iteration ends. On a fully read response the connection was already released at EOF and close() is a no-op, so keep-alive reuse for normal requests is unchanged. Fixes #30192 * test(aiohttp): cover GeneratorExit path with a mock instead of a live socket The previous slot-release test started a real aiohttp TCP server, which can flake in offline CI and does not exercise this fix's code path directly. Replace it with a dependency-injected mock that closes the stream generator (GeneratorExit) and asserts the response is closed, covering the third abnormal-exit path the finally block handles * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273) * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery * refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils * fix(proxy): make model_list request param optional for direct callers * feat(dashscope): add Responses API support (#30286) * feat(dashscope): add Responses API support DashScope's OpenAI-compatible endpoint serves /responses, so register a DashScopeResponsesAPIConfig that routes dashscope/* responses calls to {api_base}/responses without rewriting the upstream model id, instead of falling back to the chat-completions -> responses emulation pipeline. Closes #29780 * feat(dashscope): mark responses API as not supporting native websocket Matches the hosted_vllm/perplexity/openrouter responses configs, which all override supports_native_websocket() to False since the OpenAI-compatible endpoint has no native wss:// responses transport. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): preserve error_message on ProxyException failures (#30381) * fix(spend-logs): preserve error_message on ProxyException failures `StandardLoggingPayloadSetup.get_error_information` used `str(original_exception)` to populate the human-readable error message stored in `spend_logs.metadata.error_information.error_message`. `ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in its constructor but does NOT call `super().__init__(message)` and does NOT define `__str__`. As a result, `str(ProxyException(...))` returns the empty string, and every auth/budget/quota rejection was landing in spend_logs with `error_message=""` despite a fully populated traceback. Operator impact: dashboard "LLM Failure" rows became untriageable — the only way to tell a 401 from a 429 was to manually unpack the traceback JSON via psql. Burst failure patterns (e.g. a UI session polling with a stale token) produced 20-30 indistinguishable `error_code=401` rows per second. Fix: prefer the `.message` attribute (set by ProxyException and every litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback is retained for non-litellm exception types, preserving prior behavior. Test plan: - 2 new unit tests in tests/test_litellm/litellm_core_utils/ test_litellm_logging.py: * test_get_error_information_prefers_message_attribute_over_str * test_get_error_information_falls_back_to_str_when_no_message_attr - Existing test_get_error_information_error_code_priority still passes - End-to-end verified: bad-key 401 now stores full "Authentication Error, Invalid proxy server token passed..." message in spend_logs.metadata.error_information.error_message * fix(spend-logs): preserve explicit empty .message + drop dead reference Greptile P2 on #30381. The truthiness check `if message_attr:` silently skipped an explicit empty-string `.message` and fell through to `str(original_exception)`. For ProxyException-shaped objects both produce empty, so the bug was latent; for other exception types it would inject a different string into error_information.error_message and corrupt the signal. Use `is not None` so an empty string survives verbatim. Also drop the stale `See e2e/cases/11.` comment reference — that path does not exist anywhere in the repo and confuses future readers. Regression test added: an exception with `.message=""` and a non-empty `super().__init__()` arg must yield error_message == "". * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (#30382) * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response The non-streaming /v1/messages response carries a LiteLLM-injected usage.total_tokens = input_tokens + output_tokens that is not part of the Anthropic API spec. This caused three problems: 1. Shape divergence with streaming on the same endpoint. message_delta.usage in the SSE path never carries total_tokens. Clients parsing both paths get two different schemas from one endpoint. 2. Shape divergence with upstream. Direct calls to https://api.anthropic.com/v1/messages return no total_tokens field, so clients using the official Anthropic SDK couldn't rely on it, and clients that did rely on the LiteLLM-injected one broke when bypassing the proxy. 3. Numerical misuse. total = input + output undercounts when cache_read_input_tokens and cache_creation_input_tokens are non-zero, because cache tokens are reported in their own fields. A 100k-token cached prompt with 1 non-cache input token + 200 output tokens reports total_tokens = 201, off by ~99.8% from any reasonable definition of "total." Fix: add _strip_total_tokens_from_anthropic_response in litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the success path of anthropic_response right before returning. Only mutates dict-shaped responses; streaming (which already lacks the field) is left untouched. spend_logs / Prometheus continue to compute total_tokens internally for billing — this fix only strips the field from the wire response. Scope: only the Anthropic passthrough endpoint /v1/messages. The OpenAI-shape /v1/chat/completions is unaffected. * fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage Two P1 greptile threads on #30382: P1 — **Backwards-incompatible removal without a feature flag** Stripping `usage.total_tokens` unconditionally breaks any client currently reading the LiteLLM-shaped non-streaming /v1/messages response. Per the codebase's policy (mirrors #30418), gate behind a new flag. - `litellm.strip_anthropic_total_tokens: bool = False` (default — backward-compat: clients keep seeing total_tokens). - Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`. - Docstring: planned to flip to True in a future major release; opt in early. P1 — **Silent no-op if `result` is a Pydantic model** `base_process_llm_request` may return a Pydantic-style object whose `.usage` is a plain dict (the most common shape — e.g. objects wrapping raw upstream JSON). The original `isinstance(response, dict)` guard skipped strip on those, so `total_tokens` would still hit the wire. Helper now also reads `getattr(response, "usage", None)` and strips when that's a dict. Strongly-typed Pydantic `Usage` sub-models with required `total_tokens` fields are still skipped — those impose type constraints the helper doesn't try to subvert. Tests: - `test_strips_total_tokens_on_pydantic_model_with_dict_usage` - `test_flag_defaults_off` 8/8 pass locally. * fix(anthropic): drop env var for strip flag (docs CI) Mirrors #30418's pattern (`expose_router_debug_in_errors: bool = True`, no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var introduced in the prior commit was flagged by `tests/documentation_tests/test_env_keys.py` because the documentation file `docs/my-website/docs/proxy/config_settings.md` lives in `BerriAI/litellm-docs` (separate repo) and registering a new env key requires a parallel docs PR — a friction we avoid here by exposing the flag only as a Python attribute + `litellm_settings` config key, both of which load through the existing proxy config plumbing without needing the env-var registry to be updated. No semantic change: default still False, behavior identical when set via `litellm.strip_anthropic_total_tokens = True` or `litellm_settings.strip_anthropic_total_tokens: true` in config.yaml. Verified locally: env scan no longer surfaces the key; 8/8 tests pass. * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (#30413) * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 * test: resolve model prices JSON relative to test file for pip installs * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (#30417) * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit signal from upstream inside an HTTP 500/503 envelope, with the real code only surfaced in the JSON body: {"error":{"message":"...high demand...","type":"upstream_error", "param":"","code":429}} Previously LiteLLM only looked at the HTTP status and mapped this to InternalServerError, which Router treats as non-retryable for many configs — so users got hard 500s instead of fallback/retry. Now the Gemini/Vertex exception mapper parses error.code from the body and routes code 429 to RateLimitError before falling through to the HTTP-status branches. Other body codes fall through unchanged. Tests cover: - new-api gateway's `code:429` payload now maps to RateLimitError - Genuine 500-body responses stay InternalServerError - Non-JSON body strings fall through to status-code mapping unchanged * fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes Addresses greptile P1/P2 + @Sameerlite's review on #30417. The new elif branch was firing for any HTTP status, so a gateway response of HTTP 400 with body {"error":{"code":429,...}} would be incorrectly promoted to RateLimitError (retryable) instead of falling through to BadRequestError. Same trap for 401 -> AuthenticationError. Scoped the body-code 429 check to `500 <= status_code < 600` — covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx envelope) without inviting the 4xx misclassification. Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401), and the existing fall-through cases, asserting each maps to the exception type that matches the HTTP status code. 50/50 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (#30418) * feat(router)!: redact internal model_group/fallback names from exception messages The Router was unconditionally appending internal config names onto exception.message: - "Received Model Group=..." - "Available Model Group Fallbacks=..." - "No fallback model group found... Fallbacks={...}" - "context_window_fallbacks={...}" - Deployment-timeout messages including model_group - Fallback failure detail listing fallback chain ProxyException forwards .message verbatim to clients, so gateways were leaking their model_name / fallback wiring in every failed call. Fix: gate all five mutation sites on a new `litellm.expose_router_debug_in_errors` flag (default False). Set to True to restore upstream debug behavior for local debugging. Why: matches the redaction posture this codebase already has for upstream model identifiers (cf. _litellm_returned_model_name) and removes the last common error-path leak of internal model_group names. Breaking change marker (!): if anything parses "Received Model Group=" out of client error messages, flip the flag on or migrate to the x-litellm-* response headers instead. Tests: 7 cases covering each of the 5 redaction sites + the flag-on inverse path, plus a "default off" sanity check. * test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate Addresses Greptile / codecov feedback on #30418: patch coverage was 55.6% with 4 lines uncovered in litellm/router.py. The existing tests exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found), and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3 were declared in the PR description as covered by "site 5 also fires" but the gate body lines for each (the `e.message +=` inside the `if litellm.expose_router_debug_in_errors:` branch) only execute when the flag is on AND the specific exception path is taken, which neither existing test triggered. Added 4 new tests (default + flag-on × 2 sites): - test_default_does_not_leak_deployment_timeout_debug - test_flag_on_leaks_deployment_timeout_debug - test_default_does_not_leak_content_policy_fallback_hint - test_flag_on_leaks_content_policy_fallback_hint Trigger details: - Site 1 (litellm.Timeout in _acompletion) is reached via the Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on `acompletion(...)`. Cannot embed a Timeout instance in model_list because Router.__init__ deep-copies it and Timeout.__reduce__ does not preserve the required positional args. - Site 3 (ContentPolicyViolationError without content_policy_fallbacks set, in async_function_with_fallbacks_common_utils) is reached by passing a `mock_response=litellm.ContentPolicyViolationError(...)` instance via the call-site kwarg — same deepcopy-avoidance reason. 11/11 tests pass locally. Patch coverage on litellm/router.py for this PR's diff should now be 100%. * chore(router): flip expose_router_debug_in_errors default to True Addresses @Sameerlite's review on #30418 — maintain backward compat on the wire. Redact becomes opt-in via setting the flag to False; the historical behavior (leak internal model_group / fallback wiring through exception messages) is preserved as the default. - litellm/__init__.py: default flipped to True, docstring rewritten with deprecation note pointing at a future flip to False (redact by default) in a major release. - tests/test_litellm/test_router_exception_redaction.py: fixture resets to True (was False); the "off" tests now explicitly set False; the "default_leaks_*" tests rely on the fixture default. test_flag_defaults_off -> test_flag_defaults_on. - No router.py change needed; the gate keys off the same flag, only the default changes. - PR title no longer needs the breaking-change `!` marker — no client sees a behavior change at default settings. 11/11 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(guardrails): integrate Repelloai Argus guardrail (#30465) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486) * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its result is carried in `provider_specific_fields.web_search_results` — PRs #17746 / #17798 restore it for callers that round-trip provider_specific_fields. A generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on replay and instead sends back an assistant `tool_call` + a `tool` message both keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use` (with no following *_tool_result) plus a user `tool_result` for the same id — both invalid, so the next turn 400s: messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks: srvtoolu_... Each `tool_result` block must have a corresponding `tool_use` block in the previous message. This is the commonly-reported vertex_ai symptom where Gemini works but Claude 400s on the 2nd turn of a web-search chat. Fix (litellm/litellm_core_utils/prompt_templates/factory.py): - convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching *_tool_result is available to pair with it; otherwise skip it (a bare server_tool_use is itself rejected). - anthropic_messages_pt: drop a replayed `tool`/`function` message whose tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client result; a user tool_result for it is invalid). The existing reconstruction path (provider_specific_fields present, e.g. the litellm SDK) is unchanged, as is regular client tool_use/tool_result. Tests (tests/llm_translation/test_prompt_factory.py): - update test_convert_to_anthropic_tool_invoke_server_tool -> test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped - add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool Follow-up to #17746 / #17798; addresses the generic-client (no provider_specific_fields) case of #17737. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite The regression tests added in tests/llm_translation/test_prompt_factory.py aren't run by the coverage CI job (it runs tests/test_litellm), so the new factory.py branches showed as uncovered (codecov patch coverage). Add equivalent focused tests in the unit suite so both new branches are exercised there: - convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no matching *_tool_result is available. - anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic OpenAI client replays. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the server_tool_use + result valid-pair path in unit suite Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke emitting server_tool_use followed by its web_search_tool_result when the matching result is present (the litellm-SDK round-trip path). Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(anthropic): flatten srvtoolu_ tool-message guard to a negated if Addresses the Greptile style nit: replace the if-pass/else with a single negated `if not (...)` guard around the tool_result append. Behavior unchanged. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(proxy): require premium only when enabling premium metadata fields (#30285) (#30506) Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (#30488) * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR #18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate. Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions). Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate). Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced. * style(perplexity): drop redundant type annotation on else branch to satisfy mypy mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file. * fix(perplexity): update integration test expected costs for non-double-billed math Three tests in test_perplexity_integration.py asserted the old buggy expectation that reasoning_tokens are billed in addition to the full completion_tokens count. After the fix in cost_per_token, reasoning_tokens are billed at the reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the standard output rate, matching OpenAI/Perplexity convention (PR #18607). Updates: test_end_to_end_cost_calculation_with_transformation, test_main_cost_calculator_integration, test_high_volume_cost_calculation. The high-volume sanity threshold drops to 0.25 to reflect the corrected total. * fix(ui): use dynamic proxy base URL in MCP usage examples (#30487) Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the MCP server usage example and copy-to-clipboard snippet so the generated configuration works for non-local deployments. Fixes #30466 * feat: add missing UK PII entity types to Presidio guardrail (#30537) * feat: add missing UK PII entity types to Presidio guardrail Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection. * test: remove fragile hardcoded entity count test Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions. * style: apply Black formatting to match CI requirements * fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357) Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0 Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry Fixes #30346 * feat(mcp): make MCP gateway name and description configurable via env vars (#30473) * feat(mcp): make MCP gateway name and description configurable via env vars * Rename function _restore_env to _apply_env * docs(mcp): document import-time capture of env-backed identity constants Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module reload to observe env changes after import. Generated with AI assistance Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Yevhen Luhovtsov <yevhen.luhovtsov@intapp.com> Co-authored-by: Claude <noreply@anthropic.com> * fix(mcp): preserve native tools in semantic filter hook (#26650) * fix(mcp): preserve native tools in semantic filter hook The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP + native) to filter_tools(), which only knows MCP-registered tool names. Native tools silently failed the name match in _get_tools_by_names() and were dropped from the request. Fix: partition tools into native and MCP-registered before filtering. Run the semantic filter only on MCP tools, then merge native tools back unconditionally. Changes: - Robust _is_mcp_tool() using shape-based detection for OpenAI-format dicts, safe regardless of future _extract_tool_info changes - Single-pass partition loop (no double _is_mcp_tool calls) - Preserve native tools in MCP expansion path (mixed requests) - Track MCP expansion to prevent expanded tools bypassing filtering - filter_stats reports MCP-only counts for accurate metrics - Extracted _emit_filter_metadata() helper - Skip spurious filter headers for all-native tool requests Closes #26212 * remove stale docstring note referencing tools_expanded_from_mcp * fix: handle Responses API name collision and preserve tool ordering - Classify Responses API tools ({type: 'function', name: '...'}) as native to prevent name collisions with MCP canonical names - Preserve original request tool ordering using id()-based merge instead of naive native+mcp concatenation - Add 2 regression tests: name collision and ordering preservation * style: apply black formatting * fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge * lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention) * ci: retrigger checks after rebase onto litellm_internal_staging * feat(fireworks): sync Fireworks AI model registry with current platform catalog (#30616) Adds 12 new Fireworks serverless models and updates 3 existing entries in model_prices_and_context_window.json and its bundled backup to match the current Fireworks platform model list. New direct models: glm-5p2, qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6, deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast, kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and gpt-oss-20b now carry correct output token caps, cache-read pricing, and explicit capability flags max_tokens is set equal to max_output_tokens (not the full context window) for models whose generation cap is below their context window. This avoids the shared input+output budget path in get_modified_max_tokens, which would otherwise let callers request output sizes the model cannot produce. The same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b entries that had max_tokens equal to the full context window Short-form aliases (fireworks_ai/<model>) are added for every direct accounts/fireworks/models/ entry so cost attribution works for callers using bare model names. Router endpoints get short-form aliases too, and transform_request now routes bare names ending in -fast to the accounts/fireworks/routers/ path instead of defaulting every bare name to models/. This keeps the kimi-k2p6-fast router from being misrouted to the nonexistent models/kimi-k2p6-fast endpoint kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its replacement. Context windows for deepseek-v4 and kimi models use the power-of-two values (1048576 and 262144) published on the Fireworks model pages, matching the convention already used by existing entries Two regression tests in test_utils.py assert the exact per-token costs, token limits, capability flags, and short-form-to-long-form equality for all 15 models against both the main and backup cost maps. Two routing tests in test_fireworks_ai_chat_transformation.py verify bare -fast names route to routers/ and bare direct-model names route to models/ * fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443) * feat(anthropic): hoist leading in-array system to top-level (helper) * test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control * test(anthropic): mid-conversation system normalization cases * feat: add supports_mid_conversation_system flag to Claude Opus 4.8 Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost map and the bundled package backup, since the runtime helper and tests read the backup in local/offline mode. Pin the mid-system passthrough regression test to the local cost map via the existing local_model_cost_map fixture so it reads the branch-local flag rather than the network-fetched main copy. * fix(bedrock): normalize in-array system in /v1/messages handler (#29698) Wire normalize_system_messages_for_anthropic into anthropic_messages_handler so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform / Converse-bridge) hoist leading in-array system entries (and demote mid-conversation ones on models lacking supports_mid_conversation_system) into the top-level system field. The normalized messages/system are written back into the local_vars snapshot the base_llm branch reads from, otherwise the Invoke/Mantle fix would silently no-op. Also fix the helper to resolve supports_mid_conversation_system through the prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw _supports_factory could not see the flag once get_llm_provider left the invoke/ prefix on the model id, which would have wrongly demoted mid-conversation system on a Bedrock invoke opus-4-8 path. * fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param * fix(types): widen system param to Union[str, List] for hoisted system blocks * refactor(bedrock): drop dead local_vars messages writeback * fix(bedrock/converse): translate in-array system in anthropic->openai adapter (#29698) * fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty * fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches * fix(types): register supports_mid_conversation_system in model-info schema The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid) rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8 cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a typed, first-class capability flag alongside its peers (supports_output_config, etc.). --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (#28885) * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload By default the agentcore provider flattens the last message to a text-only {"prompt": "..."} payload via convert_content_list_to_str, silently dropping OpenAI multimodal blocks (image_url, file, input_audio, ...). This adds an opt-in `forward_multimodal_content` litellm param. When truthy and the last message's content is a list containing a non-text block, the original OpenAI content list is forwarded verbatim under a new "content" field so an attachment-aware AgentCore agent can read it. Default off keeps the payload byte-identical to the legacy {"prompt": "..."} shape — existing agents are unaffected. The flag is read from optional_params (where other AgentCore params land) with a litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...). AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint parses arbitrary JSON up to 100 MB (per https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html), so this is a purely upstream change; no AgentCore-side schema is asserted. * fix(bedrock/agentcore): shallow-copy forwarded multimodal content list Address review feedback (Sameerlite): payload["content"] = last_content aliased the caller's mutable messages[-1]["content"] list. Harmless today because the payload is JSON-serialized immediately, but a latent footgun if a future caller mutates the returned payload before serialization. Forward list(last_content) so the payload owns its own list. Block dicts stay shared on purpose — a deep copy would clone potentially large base64 media on the request hot path, and the flagged risk was the shared list, not the blocks. Update the passthrough tests to assert equality + distinct identity, and add a regression test that mutating the payload list can't leak back into the original message content. * Revert "fix(mcp): preserve native tools in semantic filter hook (#26650)" This reverts commit |
||
|
Sameer Kankute
|
1ccc1e5b23
|
chore: litellm oss staging160626 (#30527)
* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit |
||
|
ryan-crabbe-berri
|
5c41aabab0
|
feat(ui): cut the teams page over to the /ui/teams path route (#30343)
* feat(ui): cut the teams page over to the /ui/teams path route OldTeams renders from app/(dashboard)/teams/page.tsx via useAuthorized. The component already fetched its own paginated, filtered team list through v2TeamListCall; setTeams was only a round-trip back into the shell's lifted state, so it becomes internal useState. The organizations prop was redundant with the useOrganizations hook the component already calls, and searchParams was never read, so both are dropped. The shell keeps its own teams state and fetch for the still-coupled api-keys and models arms. Tests no longer inject teams through a prop; they mock teamListCall to drive what the component renders. * test(ui): drop unnecessary as-any casts on teamListCall mocks teamListCall returns Promise<any>, so mockResolvedValue already accepts the payload untyped. The casts pushed the repo-wide no-explicit-any lint budget over its ceiling in CI. * test(ui): drop redundant as-any casts from team mock fixtures The mocked teamListCall resolves an any-typed payload, so the inner team fixtures no longer need casts to carry a null organization_id, a keys_count field, or partial key objects. |
||
|
ryan-crabbe-berri
|
e6423d3b74
|
fix(ui): stop Virtual Keys page from infinite render loop (#30397)
The keys-filter effect called setFilteredKeys with a fresh array on every [keys, filters] change, and the caller passed keys?.keys || [] which mints a new array identity each render when the list is empty or loading. That unstable reference re-fired the effect every render, looping until React's max update depth. The effect now bails out when the filtered result is unchanged (matching the sibling teams/orgs effects), and the caller memoizes the array it passes in. |
||
|
ryan-crabbe-berri
|
685ec00afd
|
feat(ui): cut the organizations page over to the /ui/organizations path route (#30336)
* feat(ui): cut the organizations page over to the /ui/organizations path route
OrganizationsTable now owns its data through React Query instead of lifted
shell state: useOrganizations (extended with optional org_id/org_alias
filters) replaces the organizations/setOrganizations prop pair, and a new
useUserModels hook replaces the userModels prop. Create and delete
invalidate the organizations list queries rather than refetching into a
parent setter. The dead currentOrg and guardrailsList props are removed.
The shell keeps its own fetchOrganizations call because the teams and
api-keys arms still read the lifted organizations state; the userModels
state had no remaining readers and is deleted.
* fix(ui): seed organization detail initialData from any cached list, not just the unfiltered one
useOrganization's initialData only read organizationKeys.list({}), so on a
session that only ever fetched a filtered organization list the detail view
fell back to a loading state and a redundant info call. Scan every cached
list variant via the lists() prefix instead, with regression tests covering
the filtered-cache hit and the no-cache fallthrough.
|
||
|
ryan-crabbe-berri
|
1fab5c4ca6
|
chore(ui): remove dead UI components unreferenced by any page (#30340)
These components are not reachable from any page: WebRTCTester, the agents agent_card/agent_card_grid pair (the agents route renders a Table directly), and view_logs ErrorViewer/RequestResponsePanel (logs uses inline equivalents in columns.tsx). Each was imported only by its own test, so the tests go too. Also drops the now-stale agent_card_grid vi.mock in agents.test.tsx and the WebRTCTester eslint-suppressions entry. |
||
|
Yassin Kortam
|
ec9353cb69
|
feat(guardrails): add Cisco AI Defense integration (#28249) (#30338) | ||
|
ryan-crabbe-berri
|
0dc203bd65
|
feat(ui): cut the users page over to the /ui/users path route (#30334)
ViewUserDashboard renders from app/(dashboard)/users/page.tsx via useAuthorized and useTeams instead of the legacy ?page=users switch arm. The keys and setKeys props were declared but never destructured by the component, so they are removed from the interface rather than wired into the wrapper. |
||
|
ryan-crabbe-berri
|
8b9a90e0fc
|
feat(ui): migrate agents and router-settings to path routes (#30323)
* feat(ui): cut agents and router-settings over to path routes Both pages depended on a slice of the legacy shell's lifted state, now replaced with React Query hooks in their route wrappers: agents pulls teams from useTeams, and router-settings feeds the Fallbacks model dropdown from useAllProxyModels. The shell's modelData copy only populated after visiting the Models page in the same session, so the dropdown was empty on a fresh load of router-settings; the hook fixes that as a side effect of the cutover. * refactor(ui): delete the dead modelData prop chain AddFallbacks fetches its own model list when its modal opens and never reads the models prop, so the whole shell modelData -> GeneralSettings -> Fallbacks -> AddFallbacks chain fed a prop nobody consumed; RouterSettings declared it without using it at all. Remove the chain and the router-settings wrapper's useAllProxyModels adaptation that was feeding it. Also corrects this PR's earlier claim: the Fallbacks dropdown was never broken by the empty shell state, because the component self-fetches. |
||
|
ryan-crabbe-berri
|
e5a3083c2e
|
refactor(ui): remove unreachable /chat page (#30178)
The /ui/chat route is not linked from anywhere: no sidebar entry, no redirect, and no backend reference. It is only reachable by typing the URL by hand. Delete the route (src/app/chat) and its components (src/components/chat), which nothing else imports, and drop the deleted files' entries from the eslint suppressions baseline. |
||
|
ryan-crabbe-berri
|
40301820e7
|
feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes (#30267)
* feat(ui): cut caching, cost-tracking, transform-request, ui-theme, and logs over to path routes Completes the simple-leaf portion of the page-by-page App Router migration. All five legacy switch arms passed only identity props (accessToken/userRole/userID, plus token/premiumUser for caching and logs), all of which useAuthorized() provides, so each route wrapper is a thin useAuthorized() + render. MIGRATED_PAGES routes the sidebar and redirects the legacy ?page= URLs; the e2e fixture picks all five up in the migration smoke and sidebar specs automatically. * refactor(ui): colocate caching, cost-tracking, transform-request, and ui-theme components Each had the legacy switch as its only importer. caching takes its whole closure (cache_dashboard, cache_health, cache_settings, response_time_indicator); CostTrackingSettings moves as the cost-tracking components folder; the transform-request and ui-theme single-file panels move under their routes. view_logs stays at src/components: six other pages (guardrails monitor, tool policies, pass-through, MCP toolsets, usage) import it. Suppressions re-keyed. * chore: retrigger ci e2e_ui_testing failed on three specs unrelated to this PR's pages (team-info tabs, MCP create form) and local_testing_part1 on test_batch_completions; all pass on the pre-merge commit and none touch files in this diff. |
||
|
ryan-crabbe-berri
|
2893f9b67b
|
feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes (#30263)
* feat(ui): cut policies, guardrails, prompts, tool-policies, and skills over to path routes Continues the page-by-page App Router migration. All five legacy switch arms passed only accessToken/userRole, so each route wrapper is a thin useAuthorized() + render. skills keeps a claude-code-plugins alias in MIGRATED_PAGES because the old switch matched both page ids, mirroring the api_ref/api-reference precedent. * refactor(ui): colocate the prompts panel under its route The new route wrapper was its only importer, so the 32-file folder moves wholesale into (dashboard)/prompts/components; tree-escaping relative imports (networking, molecules, common_components) become @/components aliases and the suppressions baseline is re-keyed. policies, guardrails, claude_code_plugins, and ToolPoliciesView stay at src/components: each has consumers on other pages (playground selectors, AI Hub, public model hub), so their shared/page splits go in the colocation follow-up. * fix(ui): move the PromptsPanel file along with its folder @/components/prompts resolved to the prompts.tsx FILE next to the prompts/ folder, not the folder itself; the colocation moved only the folder, so the wrapper's ./components import and the panel's ./prompts/* imports both broke and next build failed. Move the panel in as components/index.tsx and fix its now-escaping relative imports. Caught by next build; tsc --noEmit missed it because incremental mode reused a stale tsbuildinfo. * test(ui): lock skills alias resolution in legacyKeyForPathname Both skills and claude-code-plugins map to the skills segment, and sidebar highlighting depends on first-match-wins returning the sidebar key; assert it so a future reorder of MIGRATED_PAGES cannot silently break highlighting. Mirrors the api_ref/api-reference assertion. Flagged by Greptile. |
||
|
Sameer Kankute
|
079c136742
|
chore(oss): litellm oss staging 120626 (#30292)
* feat(bedrock): add bedrock mantle gemma 4 models (#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from #25776 Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on #30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (#30200) (#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. #27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes #30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes #27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes #27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes #27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes #27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes #29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR #29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes #28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request #29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to #30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Sameer Kankute
|
e8e5b47fc2
|
feat(passthrough): add configurable pass-through request timeouts (#30266)
* feat(passthrough): add configurable pass-through request timeouts Allow operators to set general_settings.pass_through_request_timeout and per-endpoint timeout values, and apply them to native HTTP passthrough routes and SDK passthrough paths such as Bedrock /converse. * fix(passthrough): address CI lint and regenerate dashboard API types * refactor(passthrough): extract timeout utils to proxy-free module, fix router_timeout drop - Move resolve_llm_passthrough_timeout + resolve_pass_through_request_timeout to litellm/passthrough/timeout_utils.py (no fastapi/proxy imports at module scope) - router.py and passthrough/main.py now import from timeout_utils directly, avoiding the fastapi transitive import in pure SDK contexts - pass_through_endpoints.py re-imports from timeout_utils for backward compat - resolve_llm_passthrough_timeout now accepts router_timeout so Router(timeout=X) is respected for passthrough calls instead of being silently dropped - Use _get_httpx_client (cached) instead of bare HTTPHandler(...) in sync path to avoid creating an unclosed client per call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(router): use _explicit_timeout for passthrough to not shadow general_settings self.timeout defaults to litellm.request_timeout (6000s) when the user doesn't pass timeout= to Router(). Using it as router_timeout caused general_settings.pass_through_request_timeout to be silently ignored. Only pass router_timeout when the user explicitly set Router(timeout=X). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(timeout_utils): avoid fastapi transitive import by using sys.modules resolve_pass_through_request_timeout previously did a lazy `from litellm.proxy.proxy_server import general_settings` which loads the proxy module (and transitively fastapi) even in pure SDK contexts. Replace with a sys.modules lookup: if the proxy module is already loaded (i.e. we're inside the proxy), read general_settings from it; otherwise skip and fall back to the 600s default. No import is triggered. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): remove unused imports from pass_through_endpoints.py DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS and resolve_llm_passthrough_timeout are not used in this file; only resolve_pass_through_request_timeout is. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pass_through_endpoints): re-export DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS Tests import this constant directly from pass_through_endpoints.py; re-add it to the import from timeout_utils for backward compatibility. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pass_through_endpoints): re-export resolve_llm_passthrough_timeout for backward compat Tests import both DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS and resolve_llm_passthrough_timeout from pass_through_endpoints.py; use noqa comments to suppress the unused-import lint warning on re-exports. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
ryan-crabbe-berri
|
2d576b5695
|
feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes (#30261)
* feat(ui): cut mcp-servers, search-tools, tag-management, vector-stores, and memory over to path routes Continues the page-by-page App Router migration. All five legacy switch arms passed only accessToken/userRole/userID, so each route wrapper is a thin useAuthorized() + render. MIGRATED_PAGES routes the sidebar and redirects the legacy ?page= URLs; the e2e fixture picks all five up in the migration smoke and sidebar specs automatically. * refactor(ui): colocate MemoryView under the memory route The legacy switch was its only importer, so the three files move wholesale into (dashboard)/memory/components. mcp_tools, SearchTools, tag_management, and vector_store_management stay at src/components for now: each has importers on other pages (teams, playground, guardrails), so their colocation needs a shared/page split as a follow-up. |
||
|
ryan-crabbe-berri
|
3ad385a8a4
|
feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes (#30236)
* feat(ui): cut budgets, workflows, and guardrails-monitor over to path routes Continues the page-by-page App Router migration (#30185, #30226). All three legacy switch arms passed only accessToken, so each route wrapper is a thin useAuthorized() + render. MIGRATED_PAGES routes the sidebar and redirects the legacy ?page= URLs; the e2e fixture picks all three up in the migration smoke and sidebar specs automatically. * refactor(ui): colocate budgets, workflows, and guardrails-monitor components budgets and workflow_runs were imported only by the legacy switch, so they move wholesale into their route folders; the budgetItem type hoists into the shared useBudgets hook, which owns the API response shape, so the hooks layer no longer imports from a page folder. GuardrailsMonitor keeps LogViewer, mockData, and MetricCard at the shared src/components home because ToolDetail and ToolPolicies import them; the rest moves. eslint suppressions are re-keyed accordingly. * fix(ui): restore MetricCard test-utils path and merge duplicate import MetricCard.test.tsx got the moved-tree depth rewrite before being moved back to src/components/GuardrailsMonitor, leaving a five-level path that escapes the project root; the suite failed at import. Also merge the two imports from useBudgets in budget_panel.tsx. Both flagged by Greptile. |
||
|
ryan-crabbe-berri
|
a2c916fb45
|
feat(ui): migrate projects and access-groups to path routes (#30226)
* feat(ui): cut projects and access-groups over to path routes Same recipe as playground (#30185): MIGRATED_PAGES entries route the sidebar and redirect the legacy ?page= URLs, the switch arms are deleted, and the e2e fixture grows two entries. Both components were already zero-prop and self-fetching via React Query hooks, so the route wrappers are trivial. * refactor(ui): move Projects and AccessGroups components into their route folders Both folders were imported only by the legacy switch, so they colocate wholesale under (dashboard)/{projects,access-groups}/components. Their React Query hooks stay in the shared (dashboard)/hooks layer. eslint suppressions are re-keyed to the new paths. * test(ui): enable enable_projects_ui in e2e global setup The projects migration smoke clicks the Projects sidebar link, which only renders when the enterprise-gated enable_projects_ui setting is on; the seeded e2e database starts with it off, so the locator timed out in both e2e_ui_testing jobs. CI already launches the proxy with LITELLM_LICENSE for premium UI coverage, so flip the setting in globalSetup via the same /update/ui_settings call the admin UI toggle makes, failing loudly if the PATCH is rejected. * test(ui): use Playwright request context instead of raw fetch in global setup The frontend lint bans raw fetch() outside src/lib/http/; the e2e convention for proxy API calls is Playwright's APIRequestContext, as in routerSettings.spec.ts. |
||
|
ryan-crabbe-berri
|
530c0b2326
|
feat(ui): migrate playground to path routing and colocate its files (#30185)
* feat(ui): cut playground over to the /ui/playground path route
Follows the api-reference recipe: the sidebar and deep links route
llm-playground to the path route, ?page=llm-playground redirects, and
the legacy switch arm is deleted. The route's page.tsx was already the
real implementation, so no view extraction was needed.
* refactor(ui): move playground-owned files into its route folder
Per the (dashboard) README convention, page-owned code lives in the
page's folder: chat_ui/compareUI/complianceUI components, the chat
hooks, and the playground-only llm_calls helpers move under
(dashboard)/playground/. Modules with non-playground consumers (chat
message primitives; fetch_models, chat_completion, responses_api) stay
at their lowest common ancestor in src/components/{chat_ui,llm_calls}
because legacy pages still import them. eslint-suppressions entries are
re-keyed to the new paths so the grandfathered baseline still applies.
* test(ui): teach sidebar e2e spec about migrated path routes
The sidebar spec asserted ?page=<key> for every item, which the
playground cutover correctly broke: the sidebar now links to
/ui/playground and the legacy URL redirects there. Drive the expected
URL from the migration fixture (now a page-id -> segment map) so
future cutovers only add a fixture entry. Also wrap one import line
in AgentBuilderView.tsx that the move left unformatted; the changed-
files prettier check flagged it.
|
||
|
ryan-crabbe-berri
|
4def6916da
|
refactor(ui): consolidate dashboard to one shell in the (dashboard) layout (#30166)
* refactor(ui): consolidate dashboard to one shell in the (dashboard) layout Moves the legacy ?page= switch page into the (dashboard) route group and hoists Navbar, sidebar, ThemeProvider, and DebugWarningBanner into the shared layout with real props, deleting the degraded duplicate shell that wrapped migrated routes. The active page key now derives from the URL at render time, so navigating between legacy and migrated pages no longer remounts the shell. useProxySettings becomes a React Query hook taking accessToken, shared by the navbar, the AdminPanel arm, and migrated pages; this replaces the lifted proxySettings state and the Navbar setProxySettings prop drilling. The invitation onboarding flow (?invitation_id=) keeps rendering without chrome via a layout escape hatch. Dead dark mode state and the no-op antd ConfigProvider are removed. * fix(ui): include accessToken in useProxySettings query key The queryFn closes over accessToken, so the key must include it for the cache to be honest about its inputs. Settings are instance-global today, which made the omission harmless, but a token change while mounted would have served the cached entry without refetching. * test(ui): point CreateKeyPage test at the moved page The page moved into the (dashboard) route group and no longer renders the navbar (the layout owns chrome now), so the valid-token test asserts the default page content (UserDashboard stub) instead. |
||
|
Sameer Kankute
|
3b40ac987f
|
Litellm oss 090626 (#30021)
* fix(mcp): report scoped server name during initialize (#29865) * fix mcp scoped server name * Update litellm/proxy/_experimental/mcp_server/mcp_context.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(mcp): cover scoped server name in the SSE initialize handler --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): show all session logs in the drawer, not just the first 50 (#29795) * fix(ui): show newest session logs first * test(ui): keep session log pagination coverage * fix(ui): show all session logs in the drawer, not just the first page The session detail drawer fetched session logs via sessionSpendLogsCall without page/page_size, so it only ever received the backend default of one page (50 rows). Sessions with more than 50 calls had the rest unreachable in the UI (#29153). sessionSpendLogsCall now takes page/page_size, and the drawer fetches the first page, reads total_pages, then fetches the remaining pages and accumulates them before the existing client-side sort. This keeps the single continuous list (and the selected-log lookup and keyboard navigation, which all assume the full session) correct. Fetching is bounded by a page cap, and the sidebar shows a "showing most recent N" note if a session exceeds it. The rows are lightweight metadata (the endpoint excludes messages/response), so the full set is small; request/response bodies are still loaded per log on demand. * fix(ui): default session drawer to most recent log, newest first Open a session with its most recent log selected, and order the sidebar newest-first to match the all-sessions logs overview. MCP calls stay grouped last. The latest log by time is computed explicitly, since the MCP grouping means it is not always the first row. * Apply fetching pages in batches suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): derive session total from accumulated rows when backend omits it Compute the session total after all pages are fetched, falling back to the accumulated row count rather than the first page's. Guards the truncation note against a backend response that omits total but spans multiple pages. --------- Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): handle Mistral multipart passthrough (#29927) * fix(proxy): handle Mistral multipart passthrough * chore: satisfy passthrough ci formatting * test(proxy): cover Mistral passthrough in CI shard * fix(vertex_ai): use REP host for context caching on eu/us multi-region endpoints (#29573) Context caching built the cachedContents URL as https://{location}-aiplatform.googleapis.com, which is an invalid host for the eu/us multi-region endpoints and returns 404. The inference path already resolves these to the REP host (https://aiplatform.{geo}.rep.googleapis.com) via get_vertex_base_url(); reuse that helper in _get_token_and_url_context_caching so caching uses the same host as inference. Adds tests covering the eu/us multi-region cachedContents URLs (v1 and v1beta1). Fixes #29571 * Support per-model encrypted content affinity config (#29760) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: propagate upstream status code in proxy API exception handler (#29402) * fix: propagate upstream status code in proxy API exception handler When Google GenAI / Vertex returns a 404 for deprecated or missing models via streamGenerateContent, the exception was falling through to a generic handler that defaulted to 500. Now provider exceptions carrying a valid HTTP status_code correctly propagate it through to the ProxyException. * fix: apply black formatting to common_request_processing.py * fix: tighten status code range to 400-599 and deduplicate ProxyException raise * fix(tests): use valid vertex_location in context caching tests Replace "test_location" (contains underscore) with "us-central1" so tests pass the regex validation added in get_vertex_base_url(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sdk): add xAI OAuth provider (#29866) * Add xAI OAuth provider * Update oauth.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix xAI OAuth CI failures * Add xAI OAuth coverage tests * Move xAI OAuth coverage tests to core utils * Address xAI OAuth review comments * Prevent xAI OAuth api_base token exfiltration * Treat blank xAI OAuth api keys as absent * Wrap invalid xAI OAuth JSON responses * Use xAI OAuth behind explicit flag --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy) #27734 allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update (#27751) * fix(proxy): allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update Fixes #27734 Sending null for budget_duration, team_member_budget, team_member_budget_duration, team_member_rpm_limit, or team_member_tpm_limit via /key/update or /team/update returned 200 OK but silently ignored the null value. The fields remained unchanged in the database. Root causes: - /key/update: prepare_key_update_data() popped budget_duration from the update dict but never re-added it (or budget_reset_at) when the value was None. - /team/update: _set_budget_reset_at() only acted when budget_duration was non-None, leaving a stale budget_reset_at in the DB. - /team/update: team_member_* null values bypassed the budget table update entirely because should_create_budget() requires at least one non-None field. * test(proxy): cover no-budget-row path in clear_team_member_budget_fields * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes (#30028) * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes When output_parse_pii=true on the Anthropic native path (anthropic/claude-*), response chunks arrive as raw bytes in SSE format. _stream_pii_unmasking was yielding those bytes unchanged, so <PERSON_1> tokens were never replaced with the original values before reaching the caller. Add _unmask_sse_bytes_chunk to parse each data: line, find content_block_delta / text_delta events, and apply _unmask_pii_text before re-encoding. Wire it into _stream_pii_unmasking so bytes chunks are unmasked when pii_tokens exist. * fix(presidio): handle CRLF line endings and non-ASCII PII in SSE unmask Strip trailing \r before the [DONE] guard so CRLF-terminated SSE chunks don't bypass it and silently swallow a JSONDecodeError. Add ensure_ascii=False to json.dumps so non-ASCII replacement values like accented names are preserved as UTF-8 on the wire rather than being \uXXXX-escaped. Add regression tests for both cases. * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) (#29925) * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) Bedrock Mantle serves the Responses API on two upstream paths: - gpt frontier models (gpt-5.5 / gpt-5.4) on /openai/v1/responses - every other Responses-capable model (e.g. gpt-oss) on the standard /v1/responses BedrockMantleResponsesAPIConfig gains a `use_openai_path` flag; the provider gate in utils.py picks the path per model: openai.gpt-* (non gpt-oss) -> /openai/v1/responses; any model declared mode=responses (price-map entry or user model_info) -> /v1/responses; everything else returns None and keeps the existing chat-completions emulation. Adds gpt-5.5 / gpt-5.4 price-map entries, registry wiring, and the routing-matrix tests. * feat(bedrock_mantle): data-driven frontier routing via use_openai_responses_path Addresses the Greptile review point that frontier detection should be a price-map field rather than a hardcoded name match. The gate now routes a model to /openai/v1/responses when its price-map entry declares use_openai_responses_path, so a frontier model whose name does not follow the openai.gpt- convention can be onboarded by JSON alone. The name-convention check is kept as a fallback that needs no price-map entry, which preserves zero-change routing for a future gpt-6 before its entry loads. gpt-5.5 / gpt-5.4 get the flag in both price maps. Adds tests for the data-driven flag path and for the flag presence on the gpt-5.x entries; both branches are mutation-tested. * test(model_prices): allow use_openai_responses_path in price-map schema The model_prices_and_context_window.json schema validator (test_aaamodel_prices_and_context_window_json_is_valid) enforces additionalProperties: false, so the new use_openai_responses_path flag on the gpt-5.5 / gpt-5.4 entries failed validation. Add it to the schema as a boolean, alongside the other supports_* / capability flags. * Add Tensormesh serverless models to the model cost map (#30037) * Add Tensormesh serverless models to the model cost map * Flag reasoning support on the Tensormesh models that expose thinking mode * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update (#30001) * fix(proxy): reconcile stale key spend counter after budget reset * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update * fix(proxy): remove read-time stale counter reconciliation to prevent budget bypass * revert: undo unrelated formatting changes in enterprise directory * test(proxy): add unit test for key spend update invalidating counter * test(proxy): fix mocked update_data and hash token expectations in unit test * fix(proxy): use Responses-API transformer in pass-through cost tracking (#29728) The `elif is_responses:` branch of `openai_passthrough_handler` was calling the chat-completions `transform_response` on a Responses API payload. The chat-completions transformer expects `choices: [...]` in the raw response; the Responses API uses `output: [...]` and `usage.input_tokens` / `usage.output_tokens` (not `prompt_tokens` / `completion_tokens`). The result was a KeyError 'choices' deep inside `convert_to_model_response_object`, swallowed by the surrounding `except Exception` in the handler, and the SpendLogs row was written by the fallback path with zeroed-out tokens, spend, and model. This bug silently undercounts cost for every successful pass-through call to either OpenAI's `/v1/responses` or Azure's `/openai/v1/responses` (deployments configured for the Responses API). Reproduced 2026-06-04 against a real Azure OpenAI Responses API deployment proxied through LiteLLM v1.88.0. Fix: use the dedicated `OpenAIResponsesAPIConfig.transform_response_api_response` for the Responses branch. This transformer already exists in LiteLLM (`litellm/llms/openai/responses/transformation.py`) and knows the Responses-API on-the-wire shape. `litellm.completion_cost` already handles `ResponsesAPIResponse` natively with `call_type="responses"`, so no downstream changes are needed. Tests: test_responses_api_uses_responses_transformer_not_chat_completions NEW. Real regression test — exercises the openai_passthrough_handler with a real-shaped Responses payload (no `choices`, has `output` and Responses-API `usage` keys) and NO mocked `get_provider_config`. Pre-fix: raises KeyError 'choices' inside the chat-completions transformer (the bug). Post-fix: returns a ResponsesAPIResponse, completion_cost is called with call_type="responses" and a ResponsesAPIResponse instance (asserted). Verified to fail on un-fixed handler + pass on fixed handler before commit. test_responses_api_cost_tracking UPDATED. Old test mocked `get_provider_config` (no longer called in the responses branch post-fix). Now mocks the Responses transformer directly (`OpenAIResponsesAPIConfig.transform_response_api_response`) to test the downstream cost-calc contract. Out of scope for this PR (separate followup): - Recognizing *.cognitiveservices.azure.com (the newer Azure OpenAI hostname) in the is_openai_*_route checks. Separate PR. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(skills): execute DB skills by matching the litellm_skill_ tool name prefix (#30116) Skill IDs are generated as litellm_skill_<uuid> and the model-facing tool name is the sanitized skill ID, but the post-call execution gates in SkillsInjectionHook only ran tools whose name starts with "skill_", so DB skills were silently returned to the client as raw tool calls. Fixes #28122. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): synthesize content_block_start when Responses stream omits output_item.added (#30115) * fix(team): reserve team budget raises for proxy admins on /team/update (#30030) The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974) * test(ui): add a data-driven App Router migration E2E smoke Add a growing Playwright smoke for migrated pages: for each segment it deep-links to the path route, asserts the URL and that the dashboard shell rendered, then clicks off to a legacy page and asserts navigation still works. Driven by e2e_tests/fixtures/migratedPages.ts, so adding a page is one line. Runs in two situations against the same proxy: the default mount (npm run e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root). globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage state is valid under a prefix. Seeded with api-reference; append the rest as their migrations merge. * test(ui): support headed slow-motion + watch pauses in the migration smoke Honor SLOWMO in the server-root-path config (the default config already did), and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state. Both are no-ops by default, so CI behavior is unchanged. * test(ui): make the migration smoke a sidebar-click user journey Rework the smoke from deep-linking to a real navigation journey: start at the landing page, click the migrated page in the sidebar (expanding submenus for nested items), assert the path route rendered, reload it (the check a wrong server_root_path breaks), bounce to a legacy page and back, and — once two pages are migrated — navigate directly between two migrated pages. Verifies via URL + shell render, driven by the same fixture list. * test(ui): address review on the migration smoke Escape ROOT and segment before interpolating them into RegExp URL matchers so a future segment containing regex metacharacters can't silently widen the match. Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead of silently re-running the default mount and passing without exercising the prefix. * test(ui): drop unused watch helper and fix stale smoke README * test(ui): run the migration smoke under a server root path in CI * test(ui): harden + instrument the server-root-path proxy reboot in CI * test(ui): run the server-root-path migration smoke as its own CI job Replace the in-place proxy reboot in e2e_ui_testing with a dedicated e2e_ui_testing_server_root_path job that boots the proxy once with SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the config gets its own job rather than killing and relaunching the live proxy. The reboot was failing deterministically: after pkill -9 and relaunch the prefixed proxy never came back up on :4000 (connection refused), so the smoke never ran. The readiness step that was supposed to surface the cause could never reach its boot-log tail because CircleCI runs steps under bash -eo pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's exit 7. Booting the proxy as the job's own background step lets any boot crash land in that step's log instead of being swallowed. The default e2e_ui_testing job is unchanged aside from dropping the reboot, prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at the root mount there via the default Playwright config. * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests * chore(ui): remove dead App Router route stubs under (dashboard) (#30045) models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them * fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000) * fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss * fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation * docs(security): require a reproduction video for vulnerability reports (#30048) (#30063) With AI models capable of automated vulnerability discovery now publicly available, we expect a large increase in report volume, much of it unverified. Requiring a video of the exploit running against a live instance raises the bar for submissions and keeps triage focused on reproducible issues. Reports without a video will be closed and reopened if one is added later. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> * feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. * chore(ui): remove dead dashboard files and unused dependencies (#30047) * chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit |
||
|
ryan-crabbe-berri
|
9e0d92c129
|
chore(ui): remove dead dashboard files and unused dependencies (#30047)
* chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. |
||
|
ryan-crabbe-berri
|
248176112e
|
feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796)
* feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. |
||
|
tin-berri
|
d8fe091938
|
fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000)
* fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss |
||
|
tin-berri
|
51ba6e39cd
|
fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path (#29960)
* fix(ui): load MCP tool configuration tools via the OBO/passthrough-aware GET path * fix(mcp): admin-only include_disabled_tools so the settings UI shows toggled-off tools * fix(ui): repopulate MCP server edit form when server data loads after mount (OAuth return) * fix(ui): persist MCP OAuth token on save and return to the Settings tab after authorize * fix(ui): scope MCP OAuth callback to the initiating form so create and edit flows don't cross-talk * fix(ui): derive OAuth-return Settings tab via lazy state init instead of setState-in-effect * Fix MCP OAuth edit token handling --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
tin-berri
|
1528f43d4c
|
fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page (#29867)
* fix(ui): let non-creator users OAuth into OBO-mode MCP servers from the Tools page * fix(ui): clear OBO Tools-tab one-shot on navigate-back and gate on credential-status errors |
||
|
ryan-crabbe-berri
|
1afc41cb29
|
fix(ui): unify migrated-route URLs and migrate the API Reference page (#29953)
* fix(ui): unify migrated-route URLs and cut the API Reference page over to path routing Route all migrated-page navigation through one /ui-prefixed, serverRootPath-aware builder in migratedPages.ts (migratedHref/legacyPageHref/legacyKeyForPathname), replacing the three divergent base-URL constructions that lived in the dashboard layout's withBase, leftnav, and the page.tsx redirect. The previous migratedHref read NEXT_PUBLIC_BASE_URL, which no build sets, so it produced URLs without the /ui prefix the app is served under; every other internal link hardcodes /ui and this now matches that convention. Remove the sidebar's own pushState navigation so the parent (legacy root page or dashboard layout) is the single owner of navigation, fixing the double-navigate that fired when moving between a path route and a legacy ?page= route. Cut API Reference over to its path route: add api_ref -> api-reference to MIGRATED_PAGES and delete its arm from the legacy switch. Visiting /ui/?page=api_ref redirects to /ui/api-reference, the sidebar links to and highlights it, and navigating away returns to the legacy switch. * fix(ui): address review on migrated-page routing Keep the legacy hyphenated ?page=api-reference form working by mapping it to the api-reference route alongside api_ref; the old switch matched both, so a bookmark using the hyphen would otherwise fall through to the Usage default. Add legacyKeyForPathname coverage: a migrated path (with and without trailing slash) resolves to the api_ref sidebar key rather than the alias, a non-migrated path returns null, and a non-root serverRootPath prefix is stripped before matching. * fix(ui): populate serverRootPath from getUiConfig so migrated nav links keep the root path getUiConfig updated proxyBaseUrl but never called updateServerRootPath, so the module-level serverRootPath stayed at its "/" default. Under a custom server_root_path the unified migratedHref/legacyPageHref builders then dropped the prefix and the sidebar produced /ui/api-reference (404) instead of /<root>/ui/api-reference. Adds the missing updateServerRootPath call plus a regression test asserting getUiConfig sets serverRootPath and that migratedHref carries the prefix |
||
|
yuneng-jiang
|
728f057c5e
|
fix(ui): label default key type as "Full Access" on key edit page (#29870)
The key edit page showed the default key type (no allowed_routes restriction) as "Default", while the key creation form already labels the same value "Full Access". Align the edit page to the create form so the two surfaces agree on both the label and its description. |
||
|
yuneng-jiang
|
47b383dbbf
|
fix(ui): keep create guardrail modal open on outside click (#29871)
The create guardrail modal used antd's default maskClosable, so clicking
outside it dismissed the modal and reset every field the user had entered.
Setting maskClosable={false} keeps the modal open; it now closes only via
the explicit close button or Cancel, matching the other form modals in the
dashboard
|
||
|
yuneng-jiang
|
26fe26a5c0
|
fix(ui/model-hub): render provider icons on the public model hub (#29958)
The provider logo base path was relative ("../ui/assets/logos/"). With
trailingSlash enabled, the public model hub is served at /ui/model_hub_table/,
so the browser resolved the base to /ui/ui/assets/logos/ (a doubled /ui/), which
404s every icon. The authenticated hub renders inside the single-level /ui/ SPA
route where the relative path resolves correctly, so only the public hub broke.
Make the base root-absolute so it resolves at any route depth.
|
||
|
ryan-crabbe-berri
|
ff6cea4833
|
refactor(ui): single source of truth for migrated-page routing (#29949)
Consolidate the three hand-synced copies of the migrated-pages map (LEGACY_REDIRECTS in app/page.tsx, MIGRATED_PAGES in the dashboard layout, and MIGRATED_PAGES in leftnav) into one shared module, src/utils/migratedPages.ts, which also owns the migratedHref helper. Delete the unused, incomplete Sidebar2 prototype. No runtime behavior change: the map is still empty and Sidebar2 had no importers, so this is pure deduplication ahead of the per-page App Router migration. Follow-up work will unify the remaining base-URL builders (layout's withBase and page.tsx's redirect) onto migratedHref. |
||
|
yuneng-jiang
|
3448bf79f8
|
fix(ui): default guardrails page to first tab for admins, not submitted (#29872)
The Guardrails page hardcoded defaultActiveKey="submitted", so admins landed on the "Submitted Guardrails" tab (the last of their four tabs) instead of the primary view. The original intent was for non-admins, whose only tab is Submitted Guardrails, to default there; admins should open on their first tab. Make the default role-aware: admins default to the first tab (Guardrail Garden), non-admins keep Submitted Guardrails. |
||
|
Mateo Wang
|
13924fa1d6
|
feat: standardize rate limit errors with category, rate_limit_type, model, and llm_provider fields (#27687)
* feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError
LiteLLM previously surfaced rate-limit conditions through several unrelated
error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException).
This commit adds the data model needed to consolidate them under a single
class:
* RateLimitErrorCategory enum exposing four categorical values
(vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit,
litellm_batch_rate_limit) so callers can switch on the rate-limit source.
* New optional fields on RateLimitError:
- category (defaults to vendor_rate_limit, preserving today's behavior for
every existing call site in exception_mapping_utils);
- headers (preserves retry-after / rate_limit_type / reset_at across the
proxy boundary instead of dropping them on the floor);
- detail (mirrors FastAPI HTTPException.detail so the same instance can be
serialized through both paths).
litellm.RateLimitErrorCategory is re-exported at the package root to match
the existing exception-export pattern.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException
Adds a single proxy-side error class that subclasses BOTH
litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative
multiple inheritance.
Why both bases:
* Subclassing RateLimitError lets user code catch every rate-limit source
with one 'except RateLimitError' and switch on the new .category field.
* Subclassing HTTPException keeps every existing FastAPI plumbing path (the
isinstance(e, HTTPException) branches in proxy_server.py route handlers,
FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException))
working without modification, and preserves retry-after / rate_limit_type /
reset_at headers on the wire.
The class declaration order is (HTTPException, RateLimitError) so the MRO
puts HTTPException's no-super-call __init__ ahead of openai's cooperative
__init__ chain — preventing openai.APIError.super().__init__(message) from
landing in HTTPException.__init__(status_code=message).
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters
Replaces three bare HTTPException(status_code=429, ...) call sites with
ProxyRateLimitError, which is both a RateLimitError (catchable by category)
and an HTTPException (preserves existing FastAPI serialization). Drops the
now-unused HTTPException import in the iteration / per-session limiters.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters
Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
parallel-request limiters (key/team/user/model/customer rate limits) with
ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type
annotation accordingly.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters
Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
dynamic rate limiters (project-level TPM/RPM allocation, model-saturation
checks, priority-based limits, fail-closed guards) with ProxyRateLimitError.
The v3 limiter still imports HTTPException for an unrelated bare 'except
HTTPException:' branch.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter
Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error
with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT
so users can distinguish batch-level throttling (which counts requests/tokens
across an uploaded batch input file before submission) from the generic
key/team/user RPM/TPM limiter.
The HTTPException import is retained because the same module raises
HTTPException for unrelated 403/IO error paths.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): pin down unified rate-limit error contract
Adds a dedicated test module covering the new RateLimitErrorCategory enum,
RateLimitError.category default + override behavior, ProxyRateLimitError's
dual nature (RateLimitError + HTTPException), and a parametrized regression
guard that asserts every proxy hook module imports the unified class.
The regression guard catches the failure mode the refactor is designed to
prevent: someone re-introducing a bare HTTPException(status_code=429, ...)
in one of the hook modules instead of going through ProxyRateLimitError.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(logging): expose rate-limit category via StandardLoggingPayload
Adds an optional 'error_rate_limit_category' field to
StandardLoggingPayloadErrorInformation, populated from the unified
RateLimitError.category attribute (introduced in the previous commits on
this branch).
Why: the .category attribute is reachable off the raw exception today via
getattr(e, 'category', None), but the structured contract that downstream
custom callbacks / loggers / spend log writers consume is the
StandardLoggingPayload. Without this field, a user building custom
rate-limit metrics on top of callback data has to special-case the raw
exception object — which defeats the purpose of the StandardLoggingPayload
abstraction.
The field is None for non-rate-limit exceptions (so consumers can read it
unconditionally without isinstance checks) and is one of the
RateLimitErrorCategory string values otherwise.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): assert StandardLoggingPayload carries the category
Five tests covering: vendor default, explicit litellm_rate_limit and
litellm_batch_rate_limit values, None for non-rate-limit exceptions, and
None when no exception is provided. Pins down the contract that custom
callbacks can read 'error_information.error_rate_limit_category' off the
StandardLoggingPayload to drive custom rate-limit metrics without ever
reaching for the raw exception.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(types): silence mypy [misc] on intentional dual-base attr overlap
mypy emits two [misc] errors on the ProxyRateLimitError class line because
its two bases declare overlapping attributes with related-but-not-identical
annotations:
* status_code: int on starlette HTTPException vs. Literal[429] on openai's
RateLimitError (every openai status-error subclass narrows it the same
way and silences pyright with the same convention).
* headers: Mapping[str, str] | None on HTTPException vs. our Optional[
Dict[str, str]] (the proxy hooks always carry a stringified dict).
Both narrowings are intentional and enforced at construction time. Add a
type: ignore[misc] with an inline explanation rather than relax the
annotations on the parent or change the wire-format guarantees.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): add direct hook-invocation tests to lift patch coverage
Adds six end-to-end tests that drive each refactored hook past its
limit and assert the unified ProxyRateLimitError is raised with the
correct category and dual-base shape. Complements the
import-shape-only parametrized guard above by actually executing the
new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage
sees them as hit.
Hooks covered (one test each):
* parallel_request_limiter v1 — direct call to raise_rate_limit_error()
* parallel_request_limiter v3 — direct call to _handle_rate_limit_error
with a fabricated OVER_LIMIT response
* max_iterations_limiter — full async_pre_call_hook with mocked agent
registry, second call exceeds budget=1
* max_budget_limiter — async_pre_call_hook with mocked get_current_spend
* dynamic_rate_limiter v1 — async_pre_call_hook with mocked
check_available_usage forcing available_tpm == 0
* batch_rate_limiter — direct _raise_rate_limit_error call, asserts
category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the
generic LITELLM_RATE_LIMIT)
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: guard rate_limit_category extraction with isinstance check
* test(rate-limit): cover remaining hook raise sites for codecov
Adds five more direct hook-invocation tests so every PR-touched line
in the proxy hooks is exercised by tests in tests/test_litellm/, which
codecov measures:
* parallel_request_limiter v1 — check_key_in_limits inline raise
(the second raise site, separate from the raise_rate_limit_error
helper covered earlier)
* dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already
covered)
* dynamic_rate_limiter v3 — parametrized over all three raise sites:
model_saturation_check, priority_model, and the fail-closed
fallback for an unrecognized descriptor_key
* max_budget_per_session_limiter — full async_pre_call_hook with a
mocked agent registry and over-budget cached spend
All 42 tests in test_rate_limit_error_unification.py now pass and
together exercise every changed import + raise line across the eight
refactored proxy hooks.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: use computed error_message in ProxyRateLimitError detail
* fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn
The v1 ' raise_rate_limit_error' helper built an unused 'error_message'
variable and then assembled the actual ' detail' via an f-string that
interpolated 'additional_details' verbatim — producing
'Max parallel request limit reached None' when invoked without
arguments (flagged by code review).
Fix the helper to:
- use the constructed 'error_message' as the detail
- annotate the helper as NoReturn since it always raises
- drop the redundant 'raise'/'return' at the two call sites
Add two regression tests covering both the with- and without-
additional_details paths.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail
The v1 parallel_request_limiter's raise_rate_limit_error helper has a
long-standing bug: it computes a None-guarded 'error_message' string but
then ignores it and emits an f-string that interpolates the raw
'additional_details' arg. Callers that pass no argument get
'Max parallel request limit reached None' as the user-facing detail.
This commit:
* wires error_message into the detail kwarg so the None-guard actually
applies and operators see a clean message;
* changes the return-type annotation from ProxyRateLimitError to NoReturn
(the function always raises) so type-checkers know callers after this
invocation are unreachable.
Greptile P1 + P2 review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(types): demote TypedDict floating string to a # comment
A string literal placed after a field declaration in a TypedDict body is
not a per-field docstring — it's an orphaned string expression Python
discards. Tools like mypy / pyright that inspect TypedDict fields won't
surface that text either.
Move the documentation for error_rate_limit_category to a real comment
so the intent is visible to readers and type-checker tooling without
the misleading docstring framing.
Greptile P2 review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* security(exceptions): do not auto-copy vendor response headers to e.headers
A vendor 429 response can set arbitrary headers (Set-Cookie, CORS
overrides, …). Previously, when RateLimitError was constructed with only
a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to
a copy of response.headers. If a downstream proxy serializer ever
forwarded e.headers to the client, a malicious upstream could inject
browser-interpreted headers for the proxy origin.
Drop the fallback. Only headers passed explicitly via the headers= kwarg
make it onto self.headers (proxy hooks pass retry-after etc. — they
control what's surfaced). Vendor response headers stay reachable on
e.response.headers for callers that explicitly want them.
Today's proxy_server.py route handlers don't actually forward e.headers
on the wire (they construct ProxyException without passing headers), so
no current behavior changes — this is a defensive narrowing so the
fallback can never be turned into a vector when someone wires
e.headers through later.
Veria-AI security review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): regression guards for review-pass fixes
Pins down the three review-pass fixes:
* test_parallel_request_limiter_v1_helper_no_additional_details — calls
raise_rate_limit_error() with no args and asserts the detail does NOT
contain the literal string 'None'. Pre-fix, callers got 'Max parallel
request limit reached None'.
* test_rate_limit_error_does_not_auto_copy_response_headers — passes a
vendor httpx.Response with a Set-Cookie header to RateLimitError
WITHOUT an explicit headers= kwarg, asserts self.headers stays None
(no leak), then re-checks that an explicit headers= kwarg DOES
populate self.headers. Vendor headers remain reachable on
e.response.headers for callers that explicitly want them.
* The existing v1-helper test now also asserts the additional_details
string makes it through to the detail.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations)
trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by
the dimension that was exceeded, not just by who rate-limited (vendor vs.
litellm). Adds:
- RateLimitType str-enum exposed at `litellm.RateLimitType` with values
requests / tokens / concurrent_requests / budget / max_iterations.
- `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError;
None default so existing callers (vendor-429 path in exception_mapping_utils)
remain a no-op.
- StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom
callbacks can split rate-limit failures by cause without parsing free-text
error messages. Mirror to error_rate_limit_category extraction in
get_error_information(); single isinstance(RateLimitError) check covers both.
- map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels
("requests", "tokens", "max_parallel_requests") onto the public enum so
the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive
None on unknown values rather than silently picking a wrong dimension.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy/hooks): wire rate_limit_type onto every limiter raise site
Each refactored proxy hook now populates rate_limit_type with the dimension
that actually tripped the limit, so downstream consumers (custom callbacks,
prometheus exporters via the StandardLoggingPayload) can split key/team/user
rate-limit failures by cause:
- parallel_request_limiter (v1): detect dimension from current vs. limit in
the post-cache branch (concurrent_requests > tokens > requests, matches the
boolean condition order). Base case (current is None, one limit set to 0)
picks the most-specific zero. raise_rate_limit_error() helper accepts an
explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches
every existing internal call site, including the global-limit branch).
- parallel_request_limiter (v3): forward status["rate_limit_type"] through
map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS
for the public field while the raw v3 jargon stays on the HTTP header for
wire-format backward compat.
- dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass
data["model"] through so callbacks see the model that hit the limit
(addresses the secondary "provider missing" complaint in the original
Slack thread, partially — the model is what dashboards typically split on).
- dynamic_rate_limiter (v3): forward status["rate_limit_type"] via
map_v3_rate_limit_type() at every raise site (model_saturation_check,
priority_model, fail-closed unknown-descriptor guard). Also pass model.
- batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map
directly without going through the helper's None branch.
- max_budget_limiter, max_budget_per_session_limiter: BUDGET.
- max_iterations_limiter: MAX_ITERATIONS.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation
27 new tests across five new test classes:
- TestRateLimitType: enum exposed at litellm.RateLimitType, all five values
defined, RateLimitError default is None (vendor 429 path makes no claim
about which dimension), accepts both string and enum forms with
str-coercion guarantee for downstream JSON serializers.
- TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts
string or enum, doesn't break existing callers that pass nothing.
- TestMapV3RateLimitType: pins each v3-internal → public-enum mapping
(tokens, requests, max_parallel_requests → concurrent_requests, unknown
→ None) so a future v3 refactor can't silently swap dimensions.
- TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type
field reaches the structured payload for both ProxyRateLimitError and
plain RateLimitError, is None when unspecified, and is None for
non-rate-limit exceptions (symmetric with error_rate_limit_category).
- TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the
v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error
(both "tokens" and "max_parallel_requests" paths), and the batch
limiter (both tokens and requests paths) — coverage tools see the new
rate_limit_type= kwargs as exercised, not just the import shape.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): cover _coerce_message branches and v1 dimension detection
Drives the patch coverage on the new orthogonal RateLimitType wiring up
to (or close to) 100% on the touched files.
ProxyRateLimitError._coerce_message — was 22% covered, now 100%:
* nested {error: {message}} dict
* nested {message: {message}} dict (alt key)
* dict without 'error'/'message' keys → JSON dump fallback
* non-JSON-serializable dict value → str() fallback
* non-string non-mapping detail (int) → str() coercion
v1 parallel_request_limiter dimension detection — was 0% covered, now
exercised across 6 parametrized cases:
* check_key_in_limits else-branch: current at concurrent / TPM / RPM cap
→ asserts rate_limit_type is concurrent_requests / tokens / requests.
* check_key_in_limits base case (current is None): max_parallel_requests
/ tpm_limit / rpm_limit set to 0 → asserts the most-specific zero
attribution wins per the helper's order.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver
Introduces a small helper layer used by every proxy-side rate-limit
hook so that the 429 they raise carries a populated llm_provider /
model — instead of an empty exception.llm_provider that downstream
loggers (Prometheus failure metric, observability callbacks) read as
'no provider attribution'.
ProxyHTTPRateLimitError inherits from both fastapi.HTTPException
(so the proxy server still renders it as a 429) and
litellm.exceptions.RateLimitError (so isinstance checks and
PrometheusLogger._get_exception_class_name pick up llm_provider).
We deliberately don't call RateLimitError.__init__ — it constructs
an httpx.Response we don't need and would just add failure surface;
attribute parity is what downstream consumers care about.
resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider
defensively. Internal limiter hooks fire from async_pre_call_hook —
well before get_llm_provider runs anywhere else in the request
lifecycle — so we have to call it ourselves at raise time. If the
model is missing or unparseable (alias, router-only model) we fall
back to llm_provider='litellm_proxy' rather than letting a second
exception leak out and break the request path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on parallel-request 429s
Both v1 and v3 parallel-request limiters fired bare HTTPException(429)
from inside async_pre_call_hook. The downstream Prometheus failure
metric reads exception.llm_provider via _get_exception_class_name —
the empty value showed up as exception_class='HTTPException' and
left model_id='None' on the time series.
Threads requested_model through every raise site in:
* parallel_request_limiter.py:
- check_key_in_limits (the per-key/per-model/per-user/per-team/
per-customer over-limit path)
- raise_rate_limit_error (zero-limit + global_max_parallel_requests
paths) — now takes an optional requested_model kwarg
* parallel_request_limiter_v3.py:
- _handle_rate_limit_error (the OVER_LIMIT translator), called
from both the should_rate_limit pre-check and the TPM
reservation path
Resolved via resolve_llm_provider_for_rate_limit so unknown / missing
models silently fall back to llm_provider='litellm_proxy' instead of
breaking the request path with a second exception.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s
Same plumbing change as the parallel limiters, applied to both
dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3:
* v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve
data['model'] -> (model, llm_provider) once and pass it into both
raises.
* v3: All three raise sites in _check_rate_limits — the
model_saturation_check enforced raise, the priority_model
enforced raise, and the fail-closed unknown-descriptor branch —
now attribute the 429 to the actual provider.
Falls back to llm_provider='litellm_proxy' when the model can't be
resolved.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s
batch_rate_limiter._raise_rate_limit_error now takes a
requested_model kwarg threaded from data['model'] in
_check_and_increment_batch_counters. The batch-creation 429 is what
gets raised when the input file's tokens/requests count would push
the per-key TPM/RPM window over its limit.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on budget/iterations 429s
Final batch of internal raise sites — the user/session-budget and
max-iterations hooks. Same pattern: resolve data['model'] once at
raise time, attach to ProxyHTTPRateLimitError so Prometheus and
observability callbacks can attribute the 429.
Hooks updated:
* max_budget_limiter (per-user max_budget exceeded)
* max_iterations_limiter (per-session agent iteration cap)
* max_budget_per_session_limiter (per-session dollar cap)
All three fall back to llm_provider='litellm_proxy' when data['model']
is missing or unparseable. Drops the now-unused HTTPException import
from each module.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy/hooks): pin provider field on internal rate-limit 429s
Regression coverage for the 'provider field missing' bug across every
proxy-side rate-limit hook + the helper layer:
* ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError,
dict-detail stringification, None-provider normalization).
* resolve_llm_provider_for_rate_limit happy paths
(gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback
branches (None, '', unknown name) plus a 'get_llm_provider raises'
case that asserts we swallow the secondary exception.
* For each limiter (parallel v1/v3, dynamic v1/v3, batch,
max_budget, max_iterations, max_budget_per_session): assert the
raised exception is a RateLimitError carrying the resolved
model + llm_provider, and a sibling test that asserts the
fallback path returns 'litellm_proxy' without leaking a second
exception.
* Two PrometheusLogger._get_exception_class_name pins so the
Prometheus failure metric label flips from 'HTTPException' to
'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on
fallback) — that's what dashboards consume.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* perf(proxy/hooks): defer provider resolution to over-limit branches
* fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail
* Consolidate rate_limiter_utils imports in dynamic_rate_limiter
* fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError
ProxyHTTPRateLimitError inherits from RateLimitError but did not call
RateLimitError.__init__, so num_retries/max_retries were never set.
When Starlette's HTTPException lacks __str__, MRO falls through to
RateLimitError.__str__, which unconditionally reads these attributes
and raises AttributeError during logging/traceback formatting.
Initialize them to None defensively.
* fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError
HTTPException declares 'status_code: int' while openai.RateLimitError
(via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy
flags the multi-base override as [misc] in CI lint. The runtime semantics
are fine (we set self.status_code in __init__), so silence the
class-level annotation conflict with a targeted ignore.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: annotate batch limiter _raise_rate_limit_error as NoReturn
* feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to #27687) (#27706)
* feat(prometheus): add rate_limit_category and rate_limit_type labels
Adds two new labels to litellm_proxy_failed_requests_metric so dashboards
can split 429s by rate-limit source (vendor vs. litellm-internal) and by
the dimension that was exceeded (requests/tokens/concurrent_requests/
budget/max_iterations) without parsing free-text error messages.
Closes the Prometheus side of LIT-2718. The unified RateLimitError.category
and .rate_limit_type fields landed in PR #27687 but were only surfaced on
StandardLoggingPayload (custom-callback channel); this exposes them on
the metric label set as well.
Both labels are populated only when the underlying exception is a
litellm.RateLimitError; non-rate-limit failures keep them empty.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(prometheus): populate rate-limit labels + preserve exception_class back-compat
Two coupled changes in the Prometheus integration:
1. async_post_call_failure_hook now extracts the new RateLimitError
.category / .rate_limit_type fields (added in PR #27687) via a
_extract_rate_limit_labels helper and forwards them through
UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric.
Empty for non-rate-limit failures.
2. _get_exception_class_name special-cases ProxyRateLimitError and
keeps emitting 'HTTPException' for the exception_class label.
Without this shim, ProxyRateLimitError (which multi-inherits from
HTTPException + RateLimitError) would silently flip the label
from 'HTTPException' (the historical value for proxy-side 429s)
to 'ProxyRateLimitError', breaking existing dashboards / alerts
that key off exception_class='HTTPException'. Distinguishing
vendor vs. litellm 429s is now the job of the new
rate_limit_category label.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(prometheus): cover rate-limit labels and exception_class back-compat
Adds 19 tests across:
- enum / label-list registration
- _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError,
non-rate-limit and None inputs (incl. parametrized over every
RateLimitErrorCategory x RateLimitType combo)
- _get_exception_class_name back-compat: ProxyRateLimitError keeps the
legacy 'HTTPException' string while vendor RateLimitError keeps the
historical 'Provider.ClassName' format
- end-to-end through async_post_call_failure_hook with both
ProxyRateLimitError and vendor RateLimitError, asserting both new
labels populate and exception_class stays back-compat
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import
Address greptile feedback:
- async_post_call_failure_hook docstring: drop the stale labelnames listing
and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric
as the source of truth so the doc cannot drift from the actual labelset.
- _get_exception_class_name: guard the lazy ProxyRateLimitError import with
ImportError so router-side fallback callsites don't blow up in non-proxy
installs that don't have fastapi (a transitive dep of
proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when
fastapi is available.
Also fix the existing enterprise callback test that asserted the old
labelset on litellm_proxy_failed_requests_metric — it now expects the new
rate_limit_category / rate_limit_type labels populated for vendor 429s.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(bugbot): simplify rate-limit label coercion + guard None detail
- prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already
normalizes category/rate_limit_type to plain str, so the getattr(.value)
+ isinstance dance was dead code. Reduce to str(value) if not None.
- proxy_rate_limit_error.py _coerce_message: short-circuit None to ''
instead of falling through to str(None) = 'None', which produced the
literal message 'litellm.RateLimitError: None'.
* fix(rate-limit): surface unified category/type fields on BudgetExceededError
The most common budget cap (virtual-key max_budget enforcement in
auth_checks.py) raises litellm.BudgetExceededError, a bare Exception
subclass that bypassed the unified rate-limit error class introduced
by PR #27687. Custom callbacks reading
StandardLoggingPayload.error_information saw category=None and
rate_limit_type=None for these 429s, missing the most common budget
case (team / org / end-user budgets all hit the same code path).
Surface the fields off BudgetExceededError as plain attributes:
- category = RateLimitErrorCategory.LITELLM_RATE_LIMIT
- rate_limit_type = RateLimitType.BUDGET
- llm_provider = "" (or caller-supplied)
Switch get_error_information and _extract_rate_limit_labels from
isinstance(RateLimitError) gating to duck-typed attribute reads,
guarded by membership in the rate-limit enums so unrelated third-party
exceptions exposing a .category attribute can't leak garbage values
into the payload.
This is strictly additive: BudgetExceededError keeps its bare-Exception
base class, so `except BudgetExceededError:` handlers keep firing and
`except RateLimitError:` does not start catching budget errors.
* fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider
Two follow-ups uncovered during the second QA pass on PR #27687:
1. Guard third-party `.category` / `.rate_limit_type` attribute leakage.
The duck-typed read in `get_error_information` and
`_extract_rate_limit_labels` would forward any string attribute named
`category` / `rate_limit_type` on an unrelated third-party exception
into the StandardLoggingPayload and Prometheus labels — silently
mislabeling custom-callback payloads and blowing out Prometheus label
cardinality. Add `validate_rate_limit_category` /
`validate_rate_limit_type` helpers that gate on the documented enum
value sets; non-matching values are dropped to None.
2. Enrich BudgetExceededError.llm_provider from request_data.
Budget checks live in tenant-scoped helpers (key / team / org / tag /
end-user / project) that don't see the request model, so the
BudgetExceededError they raise carried llm_provider="" — leaving
custom-metrics consumers without provider attribution for the most
common 429 case. Resolve it once at the central
UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook
fires, so the StandardLoggingPayload the callback sees has the same
provider attribution as RPM/TPM 429s.
Regression tests pin both: 4 leakage tests + 4 enrichment tests. The
leakage tests would fail under the pre-validation version of either read
site; the enrichment tests would fail if the handler skipped the
resolver call.
* fix(rate-limit): resolve router model_name aliases to real provider (#27914)
* fix(rate-limit): resolve router model_name aliases to real provider
For nearly every real LiteLLM proxy deployment the request model is a
router model_name alias (e.g. 'tpm-locked' -> litellm_params.model:
openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about
router aliases — it raises 'LLMProviderNotProvidedError'. The resolver
then fell through to the defensive 'litellm_proxy' fallback, so the
'llm_provider' field this PR adds was effectively always
'litellm_proxy' in the field, defeating its purpose for the most common
proxy configuration.
Add a router-alias fallback step: when 'get_llm_provider' raises, scan
the active 'llm_router.model_list' for a deployment whose 'model_name'
matches the request model and resolve from its 'litellm_params.model'
instead. If multiple deployments share the same alias (load-balancing
case) the first one wins — every deployment under one alias should
agree on provider in any sensible config, and 'first' is deterministic
so the Prometheus label stays stable.
Defensive throughout: an uninitialized router, a malformed deployment,
a 'litellm_params.model' that itself fails 'get_llm_provider' — every
branch falls through to the existing 'litellm_proxy' fallback rather
than letting a secondary exception escape and mask the rate-limit
error we're trying to surface.
Tests:
- test_router_alias_resolves_to_underlying_provider: alias
'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai',
model='gpt-4o-mini'.
- test_router_alias_with_multiple_deployments_uses_first.
- test_router_alias_unknown_falls_back.
- test_router_alias_with_malformed_deployment_falls_back.
- Existing fallback test updated to also stub
'litellm.proxy.proxy_server.llm_router' so it exercises the
full 'no resolution anywhere' path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(rate-limit): harden router alias resolver + test isolation
- Wrap _resolve_provider_from_router_alias loop in top-level try/except so
a non-iterable model_list / unexpected deployment shape can't escape and
mask the 429 with a 500.
- Type-check litellm_params before .get() to handle non-dict truthy values.
- Patch llm_router=None in the parametrized fallback test so a router left
by another test in the session can't redirect the unknown-model path.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(bugbot): preserve "BudgetExceededError" Prometheus label
Adding llm_provider to BudgetExceededError (so callbacks get provider
attribution from StandardLoggingPayload) made the provider-prefix step in
_get_exception_class_name silently flip the label from "BudgetExceededError"
to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the
historical value.
Short-circuit BudgetExceededError in _get_exception_class_name the same way
ProxyRateLimitError already is. Provider/category attribution still lands on
the new rate_limit_category / rate_limit_type labels.
* test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks
The v3 rate limiter only emits 'requests', 'tokens', or
'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to
return None, leaving the expected RateLimitType.REQUESTS untested.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels
- dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit
above the TPM/RPM if/elif so the lookup runs once per request, matching
the pattern in dynamic_rate_limiter_v3.py.
- prometheus.py: gate the new rate_limit_category / rate_limit_type
labels on litellm_proxy_failed_requests_metric behind
litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the
existing prometheus_emit_stream_label opt-in. Preserves the metric's
pre-unification label set so existing dashboards / recording rules
keep matching after upgrade; operators can enable the new labels once
downstream consumers include them.
- Tests updated: default-off back-compat case, opt-in path enables the
flag before asserting label presence.
* fix: stabilize prometheus label sets and drop redundant model normalization
- Cache PrometheusLogger.get_labels_for_metric per metric_name so that
the label set used to construct counters at __init__ time stays in
sync with the label set used at increment time, even if module-level
toggles like prometheus_emit_rate_limit_labels or
prometheus_emit_stream_label are flipped at runtime. Without this,
toggling these flags after the logger was created would cause
ValueError from prometheus_client because the runtime labels would
not match the counter's declared labelnames.
- Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__
where model is already normalized one step earlier.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* perf(dynamic_rate_limiter): only resolve provider when rate limit hit
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(prometheus): clear cached metric labels after toggling rate-limit flag
The PrometheusLogger caches each metric's label set at construction
time so that labels used at counter.labels(...) time stay consistent
with the labels the metric was registered with. The enterprise
async_post_call_failure_hook test toggles
litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture
has already built the logger, so without invalidating the cache the
rate_limit_category / rate_limit_type labels never reach the mocked
counter and the assert_called_once_with check fails.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test: fix CI failures from prom label cache + flaky time-window assertion
PrometheusLogger.get_labels_for_metric now caches the per-metric label
set at first read so the labels passed to counter.labels(...) stay in
lock step with the labels the counter was registered with. This broke
two existing test patterns:
- test_prometheus_labels.py: tests bind the real method onto a
MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels
whose .get(...) returns a truthy Mock — treated as a populated cache
and returned as the label set, producing empty filtered labels and
KeyError on labels["requested_model"] / ["route"]. Seed real {}
containers for _cached_metric_labels and label_filters before binding.
- test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels:
the fixture builds the logger before the test monkeypatches
litellm.custom_prometheus_metadata_labels, so the cached label set
never picks up the new metadata labels. Clear the cache after the
monkeypatch (same pattern already used for the rate-limit toggle in
test_async_post_call_failure_hook).
UI: view_logs/index.test.tsx "Last Minute" window assertion is off by
one at the minute boundary. start_date is floored to the minute, so the
dropped sub-minute fraction can push the truncated-seconds diff up to
(minMinutes+1)*60 exactly when the click lands near a minute rollover.
Switch the upper bound to toBeLessThanOrEqual.
* feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans
PR #28909 introduced the typed v2 OTel engine that builds spans from
StandardLoggingPayload, with SpanError carrying error_type + message and
the genai mapper stamping error.type onto every failed LLM-call span.
This PR's earlier commits added error_rate_limit_category and
error_rate_limit_type to the same StandardLoggingPayload.error_information
the v2 engine reads — but neither field reached a span attribute, so v2
OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm,
RPM vs TPM vs concurrent vs budget vs max_iterations) even after the
custom-callback and prometheus surfaces gained that decomposition.
Three coupled changes:
1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY /
LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace
(no GenAI semconv equivalent exists for who-rate-limited /
which-dimension).
2. payloads.py: extend SpanError with rate_limit_category +
rate_limit_type, populated by _parse_error() from the same
error_information.error_rate_limit_* fields the custom-callback
channel and prometheus rate_limit_category / rate_limit_type labels
read. Single source of truth across all three observability surfaces.
3. mappers/genai.py: stamp the two attributes on the LLM-call span when
present. drop_none guarantees they stay absent (not 'None') for
non-rate-limit failures so trace consumers can read them
unconditionally.
Three regression tests in test_otel_v2_emitter.py pin: a vendor /
litellm-internal RateLimitError lands category=litellm_rate_limit +
rate_limit_type=requests on the span; a BudgetExceededError lands
rate_limit_type=budget; a non-rate-limit failure (BadRequestError)
keeps the rate_limit_* attributes absent. Mutation-tested against
reverting either the SpanError extension or the _parse_error read site
— both new tests fail under either mutation.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test: align prometheus user-budget + logs quick-select tests with merged code
The merge into this branch left two test patterns out of step with the code
they exercise.
test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in
flipped litellm.prometheus_user_budget_label_include_email_alias after the
fixture had already built the PrometheusLogger. get_labels_for_metric now
snapshots each metric's label set at construction time, so the runtime flip
no longer reached the cached labels. Enable the flag before constructing the
logger, matching how the proxy applies config at startup.
view_logs/index.test.tsx referenced uiSpendLogsCall and moment without
importing them, and the merged index.tsx now fetches through
useLogFilterLogic (the hook the file stubs out) rather than calling
uiSpendLogsCall directly. Add the imports and restore the real hook for the
Quick Select window assertions so the call is actually observed.
* refactor(otel/v2): drop rate-limit decomposition from the LLM-call span
Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are
rejected at the gate before any upstream call, so async_post_call_failure_hook
tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the
v2 OTel logger never opens an LLM-call span for them; the
litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes
were dead for exactly the cases they were meant to surface. The only failure
that does open an LLM-call span carrying a RateLimitError is a vendor 429, where
rate_limit_type is always None and the category just restates
error.type=RateLimitError.
The decomposition still reaches downstream consumers through
StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus
rate_limit_category / rate_limit_type labels, both unchanged.
Removes the SpanError fields, the _parse_error reads, the genai mapper
attributes, the semconv keys, and the three span tests that asserted a scenario
that never reaches the mapper in production.
* fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests
* refactor(prometheus): drop transitive fastapi import from _get_exception_class_name
Read the legacy exception_class label from a prometheus_exception_class_name
marker on ProxyRateLimitError instead of importing the proxy module, keeping
the integrations layer free of a transitive fastapi dependency.
* chore(ui): sync schema.d.ts with unified rate-limit error spec
The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429
response description, so the generated dashboard types were out of sync.
Regenerated via npm run gen:api (Check UI API Types Sync).
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
yuneng-jiang
|
7bfce053a9
|
fix(ui): make workflow runs page fill full width (#29868)
The Workflow Runs page rendered its table at roughly a quarter of the available width. Its root container is a flex child of the dashboard content row but set only padding, min-height and background, so with no width it shrank to the table's natural content size. Sibling pages (logs, memory) fill the area with a full-width root; mirror that by setting width 100% on the container. Fixes LIT-3636 |
||
|
ryan-crabbe-berri
|
f31d059aa3
|
feat(ui): add budget duration to edit team member form (#29717)
* feat(ui): add budget duration to edit team member form Editing a team member created a member budget with no duration, so the budget never reset. This threads a budget reset period through the edit flow end to end and reuses the shared duration dropdown so the options stay in sync with the rest of the UI. Resolves LIT-2651 * fix(proxy): validate member budget_duration and persist clears Reject budget_duration values that can't be parsed, are non-positive, or overflow date math before any write, so a bad value can't be persisted and later crash the budget reset job. Clearing the budget duration in the edit-member form now sends null and clears the column end to end, so the dropdown's clear control reflects a real change instead of being a no-op * chore(ui): regenerate schema.d.ts for member budget_duration Adds budget_duration to TeamMemberUpdateRequest/Response in the generated dashboard types so the Check UI API Types Sync gate passes |
||
|
tin-berri
|
21d2c3aa83
|
fix(ui): stop MCP playground tool calls from sending twice (#29821) | ||
|
ryan-crabbe-berri
|
001bda37d9
|
refactor(ui): route query-building networking calls through apiClient (#29815) |