mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
4746 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
tin-berri
|
b59ad212f4
|
feat(ui): add token endpoint auth method selector to MCP OAuth forms (#31739)
PR #31635 added a per-server token_endpoint_auth_method (client_secret_basic or client_secret_post) for upstream OAuth token endpoints, but it could only be set by editing the stored credentials JSON. This surfaces it in the dashboard as an optional selector directly under the Token URL field, in both the create form (OAuthFormFields, M2M and interactive flows) and the edit form. The field binds to credentials.token_endpoint_auth_method, which the backend already reads; the value is sent only when chosen, so leaving it blank keeps the existing setting and preserves the client_secret_post default. |
||
|
Mateo Wang
|
2633e8f8a8
|
fix(ui): include cache token columns in usage export (#32015) | ||
|
ryan-crabbe-berri
|
27069bd74f
|
feat(ui): shadcn migration foundation: Tailwind v4, shadcn init, antd cascade fix (#31995)
* feat(ui): shadcn migration foundation: Tailwind v4, shadcn init, antd cascade fix
Upgrade the dashboard from Tailwind v3 to v4 with CSS-first config: the
official upgrade codemod renamed utilities across 151 files, and
tailwind.config.js (plus the dead tailwind.config.ts) is replaced by
@theme tokens, @source globs, and @plugin directives in globals.css. The
Tremor safelist becomes @source inline patterns and the legacy tremor
theme tokens carry over verbatim. ui_colors.json was build-time only and
fed the dying Tremor palette, so its brand values are inlined and the
file removed; runtime theming replaces that path next.
shadcn is initialized with a hand-authored components.json (rsc,
cssVariables, baseColor gray) pointing utils at the existing
lib/cva.config.ts, which now exports cn (cva beta cx + twMerge) instead
of adding class-variance-authority as a second variant library. The two
ad-hoc cn helpers fold into it. Button lands as the canary primitive,
adapted to cva beta and React 18 forwardRef, with tests covering the
variant, twMerge, asChild, and ref seams. --radius is 0.5rem so the
shadcn radius scale reproduces Tailwind defaults and legacy rounded-*
classes render unchanged.
antd v5 emits unlayered CSS-in-JS that would beat every layered v4
utility, so AntdGlobalProvider now wraps the app in StyleProvider layer
and ConfigProvider cssVar, and globals.css declares
@layer theme, base, antd, components, utilities. antd wins over
preflight but yields to utilities, which is what lets migrated shadcn
pages coexist with legacy antd pages. Preflight stays global with the
three v3 behaviors pinned (default border color, button cursor,
placeholder color).
* fix(ui): restore tremor opacity tints removed by tailwind v4
Tailwind v4 removed the *-opacity-* utilities, but the precompiled
@tremor/react dist still composes them with shade-500 palette classes
(bg-opacity-10 over bg-<color>-500 etc.), so Badge, BadgeDelta, Callout,
light Icon and Button, BarList, and ProgressBar lost their tints and
rendered solid 500-shade fills. Adversarial review caught it; the
original smoke pages only exercised antd Tags.
tremor-v3-compat.css restores exactly the pairs tremor emits: for each
of the 22 safelisted colors, bg-opacity-{10,20,40}, hover/group-hover
bg-opacity-{20,30}, and ring-opacity-{20,40} against the -500 shade,
via color-mix into the utilities layer. Tremor's colorPalette maps both
background and iconRing to 500, so the -500 pairing covers every
composition in the dist; dark: variants are inert until dark mode ships.
The shim dies with @tremor/react at the end of the migration.
The upgrade codemod also missed two hand-rolled modal scrims using
bg-black bg-opacity-{30,50} (solid black under v4); now bg-black/30 and
bg-black/50. Removed the docker/build_admin_ui.sh copy of
enterprise_colors.json into the deleted ui_colors.json; that build-time
rebrand path is retired and its runtime replacement lands with the
theming phase.
* fix(ui): pair ring-opacity-40 with shade 300 in tremor compat shim
Tremor's colorPalette maps ring to shade 300, and the only consumer of
ring-opacity-40 (Icon variant outlined) composes it with that shade,
so the shade-500 rows were dead and outlined icon rings would render
at full opacity. Latent today (no dashboard usage of the outlined
variant); caught by adversarial review. ring-opacity-20 stays at 500
(iconRing), matching Badge and BadgeDelta.
|
||
|
tin-berri
|
b9df7fa705
|
fix(mcp): surface tools/list 401 auth failures as a challenge on single-server routes (#31921)
A 401 while listing tools (a missing or expired per-user OAuth token, or an upstream 401 for any auth_type) was swallowed to an empty tool list, so a single-server client got a 200 with no tools and no WWW-Authenticate challenge instead of a 401 it could re-authenticate against. Only oauth pass-through and delegate-to-upstream oauth2 servers surfaced it; every other auth_type, and the missing-token case for all of them, masked it. The surface-vs-absorb decision now keys on the route, not the auth_type. An upstream 401 in _fetch_tools_with_timeout becomes an MCPUpstreamAuthError regardless of auth_type, and the per-user OAuth challenge raised during client creation (a bare HTTPException 401 carrying a WWW-Authenticate header) is converted to the same type in _get_tools_from_server. The challenge is scoped to 401: a 403 (authenticated but forbidden, e.g. insufficient scope) is not a re-auth signal and degrades to an empty list like any other non-auth error, and the stdio-allowlist 403 (no challenge header) stays absorbed. The existing routing then does the right thing: single-server routes turn the error into a 401 + WWW-Authenticate, while the multi-server aggregator absorbs it to an empty list so one unauthenticated server does not fail the whole listing. On the UI tools page, an OBO (per-user authorization_code) server now shows the Authorize gate when the list call returns 401, not only when no credential row exists. The backend already refreshes a still-refreshable token on the list call, so a 401 means there is no valid token and none could be minted (expired with no usable refresh token), which is exactly when the user must reauthorize. |
||
|
yuneng-jiang
|
bea8c9380b
|
refactor(ui): drive cache settings form from a typed frontend schema (#31939)
* refactor(ui): drive cache settings form from a typed frontend schema
The Cache Settings form was dynamically generated from field metadata
shipped by the backend, and read its values back out of the DOM with
document.querySelector. That loses type safety and makes client-side
validation awkward, which is a poor fit for a form whose shape only
changes when a developer edits code.
Move the field definitions (name, label, type, default, help text, which
redis type they apply to, section, and validation rules) into a typed
frontend module and render them through antd Form with controlled state.
The GET /cache/settings endpoint is still used to populate current values,
and the save/test payload shape sent to POST /cache/settings and
/cache/settings/test is unchanged. Per-field validation now lives on each
field's antd rules, so an inline error can surface before and on submit;
this is where the upcoming Redis URL validation will slot in.
The backend's fields output in GET /cache/settings is no longer consumed
by the UI, but is left in place since removing it is a separate backend
change.
* refactor(ui): validate list-field JSON inline so bad input blocks save
sentinel_nodes and redis_startup_nodes had no validation rule, so
malformed JSON passed validateFields, was caught while building the save
payload, and the field was silently omitted; the user's cluster/sentinel
config was discarded with no feedback. Add a jsonListRule (same shape as
portRule) to both list fields so an invalid value surfaces inline and
blocks save.
* fix(ui): show valid-JSON examples for cache list fields and clarify the error
The Startup Nodes and Sentinel Nodes help text showed Python-style
single-quoted examples (e.g. [{'host': '127.0.0.1', 'port': '7001'}]),
which the JSON validator correctly rejects, so pasting the example we
display failed. Switch both examples to valid JSON with double quotes and
change the parse-error message to "Must be a valid JSON array (use double
quotes)" so the hint points at the fix. Also add a regression test
asserting a numeric field (Database Index) is included in the save payload.
* fix(ui): validate numeric cache fields as text so bad input blocks save
Numeric fields (Database Index, TTL, Max Connections, Similarity
Threshold) rendered as antd InputNumber, which silently coerces
non-numeric input to empty. Because the fields are optional, an invalid
entry like a full connection URL pasted into Database Index passed
validation and was silently dropped from the save payload.
Render numeric fields as text inputs with a validation rule (non-negative
integer for Database Index and Max Connections, number for TTL and
Similarity Threshold), mirroring how Port already works, so invalid input
is preserved, flagged inline, and blocks submit instead of vanishing. The
save payload still coerces these to real numbers. Adds a regression test
for a non-numeric value entered into a numeric field.
|
||
|
Sameer Kankute
|
b96f1aa686
|
fix(mcp): byom visibility, preview UX, and admin settings gating (#31809)
* fix(ui): show info message when MCP tool preview returns 403
Internal users submitting MCP servers hit an admin-only preview endpoint; replace the red connection error with a clear review notice while leaving other failures unchanged.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): let BYOM submitters see their approved servers
Approved user-submitted MCP servers defaulted to no access groups and allow_all_keys=false, so submitters could not see them after admin approval. Grant creator visibility for active submissions in get_allowed_mcp_servers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Improve dialogue box
* fix(security): restrict MCP semantic filter settings to proxy admins
Add an explicit PROXY_ADMIN check on PATCH /update/mcp_semantic_filter_settings
and hide Semantic Filter and Network Settings tabs from non-admin users in
the MCP Servers UI.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(lint): use list[str] instead of List[str] to satisfy UP006 budget
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(mcp): cache BYOM submitter server lookup with 60s TTL
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: fix ruff format and prettier formatting
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: preserve approved BYOM server visibility
* fix(mcp): keep no-mcp-servers opt-out absolute and gate BYOM union by key scope
The autofix in
|
||
|
ryan-crabbe-berri
|
3d644e1f9d
|
refactor(ui): colocate users page into route-level _components (#31897)
Moves the user-management component tree (view_users plus BulkEditUsers, edit_user, DefaultUserSettings, user_edit_view, and the view_users table/columns/info-view) out of the shared src/components dump into the users route segment under _components, now that the app router owns the route. The page imports from a trimmed ./_components barrel UserInfo moves into networking.tsx beside UserListResponse, its real owner: networking defines the user API response shapes that embed it, and previously reached up into a view folder (components/view_users/types) to import the type. Defining it in networking removes that backwards data-layer-to-view dependency and drains the view_users/ folder entirely. CreateUserButton and onboarding_link stay in components/ since the create-key flow also consumes them Relative imports in the moved files are rewritten to @/components/* absolute paths, and the eight pre-existing eslint-suppressions entries are re-keyed to the new paths so the move stays behavior and lint neutral Verified: the moved suites pass with the same 75 assertions as before the move, tsc and eslint are clean, and next build compiles the /users route |
||
|
ryan-crabbe-berri
|
2a9dbc4c0d
|
chore(ui): remove unused dep, delete dead file, and unblock knip (#31933)
Knip flagged remark-gfm as unused and date-fns as imported-but-undeclared, so drop remark-gfm (which prunes its transitive markdown subtree from the lockfile) and declare date-fns, which keyExpiryUtils.ts imports but only received transitively. Also delete the dead memory/components/index.tsx barrel, since nothing imports it once the page pulls MemoryView from its module directly Knip itself could not run: its Playwright plugin imports every config referenced by a --config flag in package.json scripts, and migration.serverRootPath.config.ts threw at import time when SERVER_ROOT_PATH was unset. Move that guard into a config-specific globalSetup so importing the config is side-effect-free; the check still fires loudly before any test runs when the prefix is missing |
||
|
Yuneng Jiang
|
1fe76dcedb
|
Revert "chore: remove _experimental/out (#31546)"
This reverts commit
|
||
|
ryan-crabbe-berri
|
3e0bd71ee9
|
feat(ui): disclaim that the Update API Key modal only rotates api_key (#31805)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(ui): disclaim that the Update API Key modal only rotates api_key An adversarial review of the credential-rotation work noted the modal always writes litellm_params.api_key, so models that authenticate with an Azure AD token, AWS credentials, or a Vertex service-account JSON are not rotated by it. Adds a warning Alert to the modal so users are not misled into thinking those secrets were rotated; broadening the modal to those providers is a follow-up * Update ui/litellm-dashboard/src/components/update_model_credentials_modal.tsx Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * style(ui): prettier-format the credential modal --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Krrish Dholakia
|
cca71a07c2
|
feat(mcp): add mcp_tool_search virtual tools for large tool catalogs (#31777)
* feat(mcp): add tool search virtual tools for large catalogs
When mcp_tool_search_enabled is set on a key's object_permission,
tools/list returns only mcp_tool_search and mcp_tool_call instead of
the full catalog. The LLM searches by keyword then calls discovered
tools by name, avoiding context bloat with 100+ tool deployments.
* fix(mcp): persist mcp_tool_search_enabled and route tool_call by name
The mcp_tool_search_enabled flag existed on the Pydantic models but the
Prisma schema lacked the column, so keys generated with the flag never
persisted it and tools/list kept returning the full catalog. Add the
column across all three schema.prisma copies plus a migration.
handle_mcp_tool_call passed server_name="" into call_tool, which built a
malformed prefixed name ("-<tool>") and failed to resolve the server.
Resolve the caller's allowed servers and dispatch through execute_mcp_tool
instead, matching how the normal /tools/call path routes.
* fix(mcp): filter list_tools to virtual tools on the protocol path
The REST surface (/mcp-rest/tools/list) returned only the two virtual
tools when mcp_tool_search_enabled was set, but the MCP protocol handler
(handle_list_tools, used by real MCP clients over streamable-http/SSE)
still returned the full catalog. Apply the same early return there so an
actual MCP client sees mcp_tool_search and mcp_tool_call instead of every
tool. call_tool was already intercepted on this path.
* fix(mcp): enforce IP + server filtering on virtual tool search/call
Review flagged that the virtual mcp_tool_search/mcp_tool_call path skipped
access controls the normal MCP flow applies. mcp_tool_call resolved allowed
servers from key permissions only, never applying IP filtering, so a caller
on a public IP could invoke a tool on a server marked
available_on_public_internet: false. mcp_tool_search listed the raw catalog
via global_mcp_server_manager.list_tools, exposing tool names/schemas that
/tools/list would hide and ignoring per-key/per-server tool filters.
Route both virtual handlers through the same filtered paths used by the
normal MCP flow: search now calls _list_mcp_tools and call resolves servers
via _get_allowed_mcp_servers, both threaded with the request client IP so
filter_server_ids_by_ip applies. execute_mcp_tool then enforces the server
allowlist and per-key tool permissions. Thread client_ip through
_list_mcp_tools/_get_tools_from_mcp_servers and pass it from the REST and
SSE call sites.
* fix(ci): ruff format server.py and sync dashboard API types
ruff format normalizes the list_tools client_ip changes in server.py, and
schema.d.ts gains the mcp_tool_search_enabled object-permission field so the
generated dashboard types match the proxy OpenAPI spec.
* style(mcp): drop quoted annotations and sort imports
Clears UP037 on the virtual tool handler signatures (redundant with
from __future__ import annotations) and I001 on the list_tools import block.
* refactor(mcp): extract virtual-tool dispatch and host progress capture
Pulls the mcp_tool_search/mcp_tool_call interception and the host
progress-callback setup out of mcp_server_tool_call into helpers, keeping
that handler under the strict cyclomatic-complexity ceiling after the
client_ip threading. No behavior change.
* test(mcp): cover SSE virtual-tool dispatch and host progress helpers
Adds unit tests for _dispatch_virtual_mcp_tool (non-virtual passthrough,
flag-disabled rejection, search/call routing with client_ip),
_capture_host_progress_callback, and the protocol list_tools virtual
early-return, covering the new server.py paths.
* fix(mcp): forward per-request auth headers through virtual tool handlers
The virtual mcp_tool_search/mcp_tool_call path intercepted the request
before the normal header extraction ran, so client-supplied per-request
auth (Authorization for upstream pass-through, x-mcp-auth-<alias>) was
dropped and execute_mcp_tool/_list_mcp_tools received None. Thread
mcp_auth_header, mcp_server_auth_headers, oauth2_headers, and raw_headers
from both the REST and SSE call sites through the handlers so upstream MCP
servers that require pass-through auth can be listed and called.
* fix(mcp): preserve requested server scope in virtual tool calls
A scoped MCP session (/mcp/<server>/ or header-scoped) carries an
mcp_servers scope that the normal call path passes into routing so the
session can only reach that server. The virtual-tool branch dropped it and
resolved with mcp_servers=None, letting a scoped session call mcp_tool_call
for any server the key can access. Thread the context mcp_servers scope
through _dispatch_virtual_mcp_tool into both handlers so search and call
resolve against the same scoped server set.
* fix(mcp): convert virtual tool errors to isError on the protocol path
The virtual-tool dispatch ran before the protocol handler's HTTPException
and guardrail handling, so a rejected virtual call (e.g. an out-of-scope
403 from execute_mcp_tool) raised out of mcp_server_tool_call and broke the
MCP JSON-RPC stream instead of returning an isError CallToolResult. Move
the dispatch inside the same try that wraps call_mcp_tool so virtual-tool
errors get the same isError conversion as normal tool calls.
* fix(mcp): spend-log virtual tool calls on the REST path
The REST virtual-tool branch returned before common_processing_pre_call_logic,
so execute_mcp_tool ran without a litellm_logging_obj and virtual mcp_tool_call
invocations were not spend-logged or guardrail-checked like normal calls. Run
the same pre-call pipeline in the call branch and thread the resulting
litellm_logging_obj through handle_mcp_tool_call into execute_mcp_tool.
* fix(mcp): reject virtual tool call when key has no accessible servers
handle_mcp_tool_call passed an empty allowed_mcp_servers list into
execute_mcp_tool; an unprefixed local tool name then fell through to the
local registry, which has no server permission check, so a key with only
mcp_tool_search_enabled and no server grants could run operator-configured
local tools by name. Reject with 403 before dispatch when no servers are
accessible, matching call_mcp_tool.
* docs(mcp): document virtual tool_search module and parity rule in AGENTS.md
* style(mcp): apply ruff format at repo line-length (120)
* fix(mcp): add mcp_tool_search_enabled to ObjectPermissionDict and customer test fixture
* chore: trigger CI
* fix(mcp): mirror pre-call pipeline, guard imports, coerce top_k, honor include_disabled_tools
- SSE mcp_tool_call now runs common_processing_pre_call_logic so it spend-logs and runs guardrails like the REST path (P1)
- coerce_top_k avoids ValueError on non-integer top_k from clients (both REST and SSE)
- guard mcp.types import in tool_search behind runtime/TYPE_CHECKING per package convention
- admin list with include_disabled_tools returns the real catalog even when mcp_tool_search_enabled is set
|
||
|
ryan-crabbe-berri
|
833406a711
|
fix(ui): rotate model credentials in a dedicated modal so a normal save can't overwrite secrets (#28089)
* feat(ui): add provider auth editing to the model edit view
Provider API keys / auth could previously only be changed by hand-editing
the raw litellm_params JSON, so there was no first-class way to rotate a
model's key. Adds an Authentication section that renders the correct
provider-specific fields (reusing ProviderSpecificFields) keyed off the
model's custom_llm_provider; fields are blank ("leave blank to keep
current") so untouched secrets are preserved and only entered values are
PATCHed and encrypted at rest.
Resolves LIT-3169
* refactor(ui): simplify model auth editing; fix stale credential branch
Drop the onFieldsResolved/authFieldKeys round trip: the parent now resolves
provider auth field keys itself via the new useProviderAuthFieldKeys hook
(same metadata ProviderSpecificFields renders), removing the report-up effect
and its stable-reference footgun. ProviderSpecificFields keeps only
excludeKeys (real need: suppress duplicate visible inputs).
Fix the stale Authentication branch: derive it from the live
litellm_credential_name form value (Form.useWatch) instead of the server
snapshot, so clearing/adding a credential mid-edit shows the right UI. Also
skip inline auth updates entirely when a named credential is selected, so we
never submit a credential name and raw inline auth together.
* fix(ui): don't leak freshly-entered model auth secrets to display/console
The auth values a user types are still sent in the PATCH request, but:
- strip them from the locally-stored litellm_params after save so the
read-only LiteLLM Params JSON doesn't render the plaintext key
- remove the debug console.log in modelPatchUpdateCall that dumped the
full update payload (incl. api_key / vertex_credentials) to the browser
console on every model update
Backend stores these encrypted and returns them masked on refetch.
* fix(ui): don't require blank auth fields in model edit context
Auth fields render blank ('leave blank to keep'), but required metadata
(e.g. OpenAI api_key) added a required validation rule that blocked
onFinish entirely — making it impossible to save any unrelated edit
without re-entering the secret. Add a disableRequired prop to
ProviderSpecificFields and set it in the model edit Authentication section.
* fix(ui): rotate model credentials in a dedicated modal so a normal save can't overwrite secrets
The model edit form seeded the read-only LiteLLM Params textarea with the whole
litellm_params blob and re-sent all of it on every save. Because /model/info
redacts secrets by masking them ("azur****BBCC") rather than removing them, any
save re-encrypted the asterisk mask over the real value and silently destroyed
credentials such as azure_ad_token, aws_session_token, watsonx token/zen_api_key
and the OCI key fields. api_key, client_secret, vertex_credentials and the AWS
access/secret keys were safe only because the backend strips those entirely
Credential rotation now lives in a dedicated UpdateModelCredentialsModal that
PATCHes only the fields the user types, decoupled from the params blob; the
backend already merges partial litellm_params, so the rest of the deployment is
left untouched. The general edit form drops masked values from both the textarea
seed and the outbound payload, so a normal save can never carry a redacted secret
Also removes the now-unused inline auth section and its excludeKeys and
useProviderAuthFieldKeys plumbing, strips secret-leaking console.logs from the
provider upload handler and the model-update response, and fixes a
react-hooks/use-memo error that was failing the frontend-lint CI job
* chore(ui): ratchet no-explicit-any lint metric to 2013
Removing the credential-echoing console.log (and its info: any param) from the
provider upload handler dropped the tracked count by one; update the committed
baseline so the Check lint budgets CI step is not stale
* refactor(ui): scope the model credential modal to api-key rotation only
Narrows UpdateModelCredentialsModal to a single API Key field. On submit it
PATCHes only { api_key }, so the backend merge leaves every other deployment
param untouched; a model authed via azure_ad_token, AWS keys, or a Vertex JSON
won't have anything to rotate here yet, which is the intended scope for now.
Drops the multi-field provider rendering this added earlier, which also removes
the now-unused disableRequired prop from ProviderSpecificFields and reverts that
shared component to its prior shape. The "Update API Key" trigger button is now
an antd Button rather than a TremorButton, so the feature introduces no tremor.
* refactor(ui): convert the model detail toolbar buttons from tremor to antd
Switches Test Connection, Re-use Credentials and Delete Model to antd Button so
the toolbar matches the Update API Key button and no longer mixes libraries;
Delete Model uses antd's danger styling instead of hand-rolled red classes
* style(ui): make the api-key modal submit button primary and drop the Need Help link
|
||
|
yuneng-jiang
|
776b272689
|
Merge pull request #31735 from BerriAI/litellm_lit_4057_router_settings_routing_groups_save
fix(ui): fix Router Settings Loadbalancing tab save (LIT-4057) |
||
|
ryan-crabbe-berri
|
4f41a9e140 |
test(ui): drop the e2e typecheck CI gate, keep the typed import for the editor
The e2e runs against the real proxy, so a contract drift already fails the test at runtime; tsc only checks the spec against schema.d.ts, a generated snapshot, so a backend change with a stale snapshot would pass tsc while the live test still catches it. The dedicated tsconfig + script + CI step were circular ceremony for that. Keep the zero-runtime-cost type-only import, which still catches mistakes in the editor, and make its comment honest about what enforces the contract. |
||
|
ryan-crabbe-berri
|
3971469b71 |
test(ui): harden Router Settings e2e and make its typing a real CI gate
Address an adversarial review of the Loadbalancing e2e: - The "typed against the backend schema" claim was hollow: nothing type-checked e2e_tests (the root tsconfig excludes it and no CI step runs tsc), so a contract drift would compile and run unchanged. Add e2e_tests/tsconfig.json, a typecheck:e2e script, and a CircleCI step so the schema typing actually gates. - The two describe blocks both mutate the proxy's shared router_settings, and the Loadbalancing save echoes the whole settings object, so they could clobber each other under local fullyParallel. Run the file serially. - patchRouterSettings swallowed a failed seed, which surfaced later as a misleading UI timeout. Assert the write succeeded, and rely on the server-side merge instead of echoing the whole settings object back (drops a cast and a GET). - Empty routing_groups already reproduces the bug, so drop the non-empty seed and its model coupling. |
||
|
ryan-crabbe-berri
|
540c860a97 |
test(ui): add typed e2e for Router Settings Loadbalancing save (LIT-4057)
Drives the real save flow against a live proxy: seeds a present routing_groups array (the LIT-4057 trigger) via the typed /config/update contract, changes num_retries on the Loadbalancing tab, and asserts the POST returns 200 instead of 422, the success toast appears, and the value still shows after a reload (the ticket's "refresh shows old values" symptom). The round-trip is typed against the OpenAPI-generated backend schema (ConfigYAML write, RouterSettingsResponse read) through a type-only import, so a backend contract drift fails the type check. |
||
|
ryan-crabbe-berri
|
30141f86f8 |
test(ui): make router settings save tests resilient to async timing
Address Greptile P2: the routing_groups test read setCallbacksCall.mock.calls[0][1] immediately after the now-async save handler, so any latency in the mock would throw an opaque TypeError instead of a clean assertion failure. Assert through toHaveBeenCalledWith inside waitFor with expect.not.objectContaining, dropping the index access and the cast. Also drop the ticket id from the test names. |
||
|
ryan-crabbe-berri
|
9968499aab |
fix(ui): fix Router Settings Loadbalancing tab save (LIT-4057)
The Loadbalancing tab rendered routing_groups as a generic text input and sent its array value back as the JSON string "[]", which fails Pydantic list validation on POST /config/update and returns 422. routing_groups has its own dedicated Routing Groups tab, so this tab must neither render nor write it; exclude it the same way retry_policy and model_group_retry_policy are excluded for the Model Retry Settings tab. The save was also fire-and-forget: setCallbacksCall was not awaited, so the rejected promise escaped the try/catch and the success toast fired unconditionally, showing success even when the backend rejected the change. Await the call, gate the success toast on resolution, and surface the error. |
||
|
ryan-crabbe-berri
|
7ed25de120
|
fix(ui): allow any git host on the skills add form (LIT-4053) (#31652)
* fix(ui): allow any git host on the skills add form (LIT-4053)
The skills add form only accepted GitHub URLs: its URL parser bailed on
any host that did not start with github.com, so GitLab, Bitbucket, and
self-hosted repos (and any repo subfolder on them) were rejected before a
request was ever sent. The backend already accepts arbitrary git hosts
via its url and git-subdir sources, with no host allowlist, so this was a
client-side restriction only.
Generalize the parser into an exported, host-agnostic parseSkillSource:
GitHub URLs keep their github / git-subdir shorthand, every other host is
treated as a raw repo url, and an optional Subfolder path field turns any
repo into a git-subdir source (url + path). When a pasted GitHub
tree/blob URL already encodes a subfolder, the field is cleared and
disabled so a contradictory source can never be submitted.
The parser is hardened to match the backend contract: query strings and
fragments are stripped, the host match is case-insensitive and drops a
leading www., the extracted and field-entered subfolder paths are both
validated against the same regex the server uses, a real file-extension
allowlist (not "any dot") decides whether a trailing blob segment is a
file, a branch-only tree URL falls back to the repo, non-GitHub URLs
require at least an org/repo, and the suggested skill name is kebab-cased
so it satisfies the name field's own rule.
The git-subdir source is now handled in the display helpers
(getSourceDisplayText, getSourceLink, formatInstallCommand), which
previously showed it as "Unknown source" with no link. The submit path
is fully typed (RegisterPluginRequest plus an AddPluginFormValues
interface), removing the two prior any usages; as a result an
author with an email but no name is dropped rather than sent, since the
backend requires the author name.
No backend changes. Tests cover the full host/subfolder matrix at the
parser level plus form-submit assertions on the exact source payload.
* refactor(ui): sync skill register types to the generated OpenAPI schema, surface backend errors
Replace the hand-maintained, already-drifted API types for the skills add
flow with the generated ones from schema.d.ts: PluginAuthor now aliases
components["schemas"]["PluginAuthor"], the registration payload is a new
SkillRegisterRequest (the generated RegisterPluginRequest envelope with
source narrowed to our PluginSource union, since the backend types source
as a loose string map, and version kept optional since the backend
defaults it), and the dead, mismatched RegisterPluginResponse is deleted.
registerClaudeCodePlugin's inline payload type (which was missing the
git-subdir path field entirely) is replaced with SkillRegisterRequest, so
the networking layer and the form can no longer drift from the backend.
Error handling: the add-skill form swallowed the real failure and always
showed "Failed to register skill". registerClaudeCodePlugin already
derives the backend message and throws it, so the form now surfaces it
("Failed to register skill: <reason>"), and the networking helper falls
back to the raw body / status when the error response is not JSON instead
of throwing a JSON parse error. A regression test asserts the backend
message reaches the user.
* fix(ui): reject credentialed git URLs on the skills form
A repo URL with embedded user-info (user:token@host) passed the raw-host
parser and was stored verbatim as the skill source, which is served on
the unauthenticated /public/skill_hub and marketplace.json feeds, leaking
the credentials. Reject any host segment containing '@'.
* fix(ui): validate skill repo URLs through one WHATWG URL gate
Replace the ad-hoc string parsing (stripScheme / splitHost / manual
scheme, @, ?# checks) with a single parseRepoUrl gate built on the URL
parser, so every malformed/unsafe class is handled in one place and the
URL stored on the public skill feeds is always canonical. It enforces
https (rejecting http/ssh/git/file/javascript/data and protocol-relative
//host), rejects embedded credentials (user:token@host, including
userinfo-confusion like github.com@evil.com), rejects IP-literal hosts
(loopback/private/metadata and obfuscated/IPv6 forms), and rebuilds the
stored url from origin+pathname so query strings, fragments, and trailing
slashes can never be published. The GitHub org/repo shorthand is now
charset-validated like the other paths, so junk can't reach the stored
repo. Closes both Veria findings (credentialed and http sources) plus the
adversarial-review follow-ups, with regression tests for each class.
|
||
|
yuneng-jiang
|
f8a2ea7378
|
Merge pull request #31426 from BerriAI/litellm_/cranky-hamilton-21b5d0
fix(ui): stop Request Logs page from overflowing horizontally and size its columns |
||
|
ryan-crabbe-berri
|
3dce3daff6
|
feat(proxy): type Customer Management response_model for OpenAPI coverage (#31043)
* feat(proxy): type Customer Management response_model for OpenAPI coverage Add response_model to the five remaining untyped /customer operations (block, unblock, new, update, delete) so the generated OpenAPI schema documents a concrete response body. new/update reuse the canonical LiteLLM_EndUserTable (matching info/list); block, unblock, and delete get small dedicated models in litellm/types/proxy/management_endpoints/customer_endpoints.py. Together with the already-typed info/list/daily-activity routes this brings the Customer Management group to full response_model coverage. Regression tests assert each public /customer/* route declares the expected response_model and that /customer/new surfaces a typed schema in app.openapi(), so dropping a response_model fails CI. * fix(proxy): keep budget_id in typed customer responses Address review feedback on the Customer Management response_model typing. Greptile flagged that response_model=LiteLLM_EndUserTable on /customer/new and /customer/update silently drops fields the raw Prisma model_dump() echoed. Checking the schema, budget_id is the only such scalar column that was missing from the Pydantic model (created_at/updated_at/tpm_limit do not exist on litellm_endusertable), so add budget_id to LiteLLM_EndUserTable. This restores budget_id on new/update and also fixes the pre-existing gap where /customer/info and /customer/list (already typed) dropped it, which the UI Customer type expects. A regression test pins budget_id surviving the response_model filter on /customer/update. Also document UnblockUsersResponse.blocked_users via a Field description: it holds the users that remain blocked after the call. The key name predates this PR and is kept to avoid a backwards-incompatible rename on a beta route. * fix(proxy): keep nested budget fields in customer responses response_model=LiteLLM_EndUserTable nests the budget as the narrow write allowlist LiteLLM_BudgetTable, which silently drops the server-managed fields the customer endpoints used to return (budget_reset_at, created_at). Introduce CustomerResponse, a thin response model that nests LiteLLM_BudgetTableFull (the repo's budget response model), and apply it on /customer/new, /customer/update, /customer/info and /customer/list. list also builds CustomerResponse so its budget isn't narrowed at construction time. created_by/updated_at/updated_by remain omitted, matching how budgets are returned elsewhere. The shared LiteLLM_EndUserTable is left untouched: it's constructed in many places that pass narrow budget instances, and pydantic v2 won't coerce a budget instance into a wider nested model. Typing only at the response boundary (where the handler hands FastAPI a dict) sidesteps that. A regression test pins budget_reset_at + created_at through the filter and asserts the internal audit fields stay out. * test(proxy): add golden-master characterization tests for customer responses Lock the exact JSON body each customer-object endpoint (info/list/new/update) and delete return today, so the upcoming type-safety refactor of the handlers is only allowed to land if it reproduces these byte for byte. Pins null-field inclusion, the nested budget shape (server fields kept, audit fields dropped), and object_permission reverse-relation stripping. Green against current code. * refactor(proxy): make the customer response flow type-safe Replace the untyped dict + bolt-on response_model pattern on the customer object endpoints with explicit typed construction. A single mapper, _to_customer_response, validates a DB row into CustomerResponse at one Any -> typed seam; new/update/info/list now return it (or a list of it) and carry real -> CustomerResponse / -> List[CustomerResponse] return annotations, and delete returns DeleteCustomersResponse. basedpyright now verifies the handlers' return shapes instead of a runtime filter doing it silently. This also deletes the four copy-pasted object_permission reverse-relation cleanup loops: pydantic's extra=ignore drops those undeclared fields during validation, so the loops were dead code (proven by the golden-master tests, which stay byte-for-byte green). basedpyright errors on the file drop from 140 to 116, all from removed dict plumbing. CustomerResponse stays a thin subclass of LiteLLM_EndUserTable so it inherits the existing validators/config unchanged (behavior preservation); only the nested budget type is widened. * refactor(proxy): annotate customer response mapper param as BaseModel Address review nit: the mapper's untyped `record` added an ANN001 violation. The incoming rows are pydantic v2 models, so type the param as BaseModel rather than object (object has no model_dump, which would just move the problem to basedpyright). This clears the ANN001 and also drops three basedpyright unknown-type violations the untyped param was adding. * style(test): ruff format customer endpoint tests * test(proxy): give customer budget test update mocks a valid model_dump The type-safe response refactor validates the update result via _to_customer_response (CustomerResponse.model_validate(record.model_dump())). These budget tests mocked the end-user update to return a bare MagicMock, so model_dump() yielded a MagicMock that fails validation. Give each update mock a minimal valid dict; the tests assert on the prisma calls, not the body. * chore(ui): regenerate API types from proxy OpenAPI spec * fix(ui): make generated API types stable across Python versions Python 3.13 strips a docstring's common leading indentation at compile time while 3.12 keeps it, so app.openapi() emits differently-indented description strings depending on the interpreter. The dashboard type generator ran locally on 3.13 and in CI on 3.12, so schema.d.ts drifted and the "Verify schema.d.ts matches the proxy OpenAPI spec" check failed Normalize every description through inspect.cleandoc in the spec dump so the output is identical regardless of interpreter, then regenerate |
||
|
Mateo Wang
|
72bcb748b9
|
chore: remove _experimental/out (#31546)
* chore: remove _experimental/out * fix(ci): recreate _experimental/out before copying UI build output The build scripts cp the Next.js output into litellm/proxy/_experimental/out, which was removed from git. cp failed because the target directory no longer existed; mkdir -p recreates it before the copy. * fix(proxy): make UI serving resilient to a missing _experimental/out Removing the committed UI export means the source/test tree no longer ships litellm/proxy/_experimental/out. Three things assumed it was always present and broke once it was gone: - get_favicon hard-coded the built favicon path and 404'd without it; it now falls back to the bundled swagger/favicon.ico - the /_next and /ui static mounts raised at construction when the export was absent, so the whole UI-setup block was swallowed and no mounts registered; they now use check_dir=False - _restructure_ui_html_files was a nested function only exposed as a module attribute when that block happened to succeed; it is now a real module-level function test_admin_ui_export_serves_nested_extensionless_routes validated the committed artifact, whose premise this PR removes; it now drives the same MCP OAuth callback restructure guarantee through a synthetic export. * chore(greptile): ignore generated _experimental/out so review fits the file limit * Revert "chore(greptile): ignore generated _experimental/out so review fits the file limit" ignorePatterns is applied after Greptile counts the files changed, so it does not bring the diff under the file limit; the config had no effect. |
||
|
Yuneng Jiang
|
256b5aadfb
|
fix(ui): revert Request ID width to default, tighten Session ID
Drop the explicit size on Request ID so it falls back to the default width like the other reverted columns. Narrow Session ID from 160px to 120px since its truncated value needs less room |
||
|
Yuneng Jiang
|
84d7a32020
|
fix(ui): revert Duration and TTFT column widths to default
The explicit 90px/80px sizes were too narrow for the Duration (s) and TTFT (s) headers once the sort arrows were factored in, cramping the header labels. Dropping the size lets these two columns fall back to the default width like before |
||
|
Mateo Wang
|
884cdc1537
|
fix(anthropic): drop unsignable thinking blocks and allow null signature in logging (LIT-4007) (#31654)
* fix(anthropic): drop unsignable thinking blocks and allow null signature in logging (LIT-4007) Open-source reasoning models (DeepSeek-R1 and distills, Qwen3/QwQ, IBM Granite 3.2 via vLLM/Ollama/OpenRouter/DeepSeek) return reasoning_content with no Anthropic-style signature, which LiteLLM represents as a thinking block with a null signature. Two failures resulted. First, ChatCompletionThinkingBlock.signature was a required str, so building the StandardLoggingObject raised a ValidationError on signature=None and the success log record was silently dropped while the request still returned 200; relaxing it to Optional[str] lets the log build. Second, replaying such a turn to a real Anthropic model forwarded the null-signature thinking block unchanged and Anthropic rejected it with 400 thinking.signature.str; since Anthropic verifies the signature cryptographically, a null, empty, or missing signature cannot be repaired, so anthropic_messages_pt now drops the unsignable thinking block while preserving the assistant text and keeping genuinely signed blocks. * style: use builtin generics for thinking-block filter helpers * fix(ui): regenerate schema.d.ts for nullable thinking-block signature |
||
|
ryan-crabbe-berri
|
ddba2e2b15
|
refactor(ui): colocate search-tools into route-level _components (#31658) | ||
|
ryan-crabbe-berri
|
5e5b09709c
|
perf(ui): load virtual-keys team filter from the fast v2 endpoint (#31638)
* perf(ui): load virtual-keys team filter from the fast v2 endpoint The virtual-keys table sourced all teams through fetchAllTeams, which hits the unpaginated /team/list. On a proxy with 125 teams that call takes ~9.5s, so the Team ID filter and the team-alias/budget columns sat empty for that whole window. The key list itself does not carry team_alias or team_max_budget, so the table genuinely needs a team lookup and cannot just drop the fetch. Add useAllTeams, which pages the fast /v2/team/list to completion (~0.6s per 100-team page, so ~1.2s for 125 vs ~9.5s), and point VirtualKeysTable at it instead of fetchAllTeams. The allTeams shape, the filter searchFn, the column lookups, and the loading indicator are all unchanged; only the source endpoint changes. fetchAllTeams stays for its other callers. * test(ui): tighten team-filter test readability and robustness Address adversarial review of the added tests. Rename the single-page useAllTeams test to match what it asserts (one request for a one-page result) rather than implying it guards the early-return, and drop the unread, misleading total: 125 from the mock page response. Scope the created_by alias-over-email assertion to the key's table row so it checks the visible cell value; the hover popover that also holds the email is portaled out of the row, so the previous document-wide negative assertion was relying on antd's lazy popover mounting. * fix(ui): scope useAllTeams cache by access token The previous /team/list query keyed on accessToken, so a user switch in the same SPA session produced a distinct cache entry. useAllTeams dropped that, so team IDs and aliases could be briefly reused across users until the staleTime expired. Put accessToken back in the query key to restore per-identity isolation, and add a regression test that a token switch triggers a refetch rather than serving the cached list. |
||
|
ryan-crabbe-berri
|
0e5aee1838
|
fix(ui): keep virtual-keys filters across delete and refresh (LIT-4080) (#31533)
* fix(ui): keep virtual-keys filters across delete and refresh (LIT-4080) Filtering virtual keys by User ID and then deleting a key reset the filter to show all keys, and re-clicking Fetch did not re-apply it. The page ran two competing fetch paths: useKeys (React Query) fetched the page unfiltered while a separate useFilterLogic hook held its own filteredKeys list and, on any refresh, only re-applied Team and Organization client-side, silently dropping the User ID and Key Alias filters. Delete refreshed through the unfiltered useKeys path, so the filtered view collapsed back to everything VirtualKeysTable now owns its filter state and feeds every filter (team, organization, key alias, user id, key hash) straight into the useKeys options, so the filters are part of the React Query key. Any refetch or invalidation re-runs the same filtered query, which makes the reset-on-delete bug structurally impossible. Free-text inputs are debounced with @tanstack/react-pacer, sorting and pagination are server-side, and changing a filter or sort resets to page 1 Delete now invalidates keyKeys.lists() from key_info_view, matching the create path, instead of prop-drilling a refetch; the window "storage" refetch effect is removed. The dual-path useFilterLogic hook (and its test) are deleted Regression coverage: VirtualKeysTable threads an active User ID filter into the useKeys query and clears it on reset, useKeys encodes filter options in its query key so a filter change refetches, and key_info_view invalidates the keys list on delete * refactor(ui): simplify virtual-keys table data flow VirtualKeysTable now fetches its own teams and organizations via useOrganizations and the existing all-teams query instead of taking them as props, so the prop-drill through UserDashboard and the two page callers (page.tsx, ApiKeysDashboard) is gone along with their redundant organization state and fetch Filter state collapses from a useState plus a useDebouncedState mirror into a single source whose debounced copy is derived with useDebouncedValue, and one typed toKeyListFilters adapter maps it to the key/list query options. Behavior is unchanged; same 300ms debounce and the same reset timing The unused onSortChange/currentSort props and their sync effect are removed since no caller passed them, leaving sorting fully internal Adds a created_by_user alias-over-email regression test that fails if the display precedence is swapped * test(ui): add required last_active to useKeys mock fixtures The KeyResponse type requires last_active, so the typed mockKeys fixtures were missing it. Add it so the file type-checks cleanly. * chore(ui): ratchet lint budgets after virtual-keys refactor Deleting filter_logic.tsx and simplifying VirtualKeysTable lowered the no-explicit-any (2026 to 2016) and complexity (128 to 127) counts, so the eslint-metrics.json baseline was stale and failed the frontend-lint budget gate. Regenerate it, and drop the now-dead filter_logic.tsx suppression entry for the file this PR removed. * fix(ui): show a loading state for data-backed filter dropdowns The Team ID and Organization ID filters source their options from async hooks (teams / organizations). While that data was still loading the dropdowns rendered 'No results found', so they looked empty rather than loading. Add an opt-in loading flag to FilterOption that the searchable select surfaces as a spinner and a 'Loading...' empty state, and wire it from the teams and organizations query loading states. While loading, the filter no longer caches an empty initial-options list, so the real options appear once the data arrives. |
||
|
michelligabriele
|
d7654d07ab
|
feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration (#31215)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(proxy): add AES-256-GCM at-rest credential encryption with versioned format and re-encryption migration * test(proxy): add behavior scenarios for credential migration endpoints * fix(proxy): scan covered tables in encryption check, fix CI lint and route types * fix(proxy): migrate callback_settings credentials, clear CI lint/recursion gates, add encryption endpoint+CLI tests * fix(proxy): correct dry-run/real-run migrated vs residual-legacy counters in config and SSO walkers * fix(proxy): make callback-vars residual detection gate-independent in encryption check |
||
|
Sameer Kankute
|
8e30cfbeb1
|
feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents (#30950)
* feat(a2a): support a2a-sdk 1.x proxy routing for 0.3 and 1.0 agents Bump a2a-sdk to 1.x and wire send/stream through compat conversions so the proxy accepts A2A 1.0 JSON-RPC while preserving 0.3 wire clients. Co-authored-by: Cursor <cursoragent@cursor.com> * Add user controlled protocol version in agents * Fix exeception mapping * Fix a2a base url * Add e2e test for a2a * Fix lint * Fix lint * fix(a2a): harden card version detection and header isolation coverage Use protocolVersion when inferring agent card wire format, assert distinct httpx cache keys in the header-isolation test, and suppress targeted basedpyright errors for optional SDK imports. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): suppress reportArgumentType for SDK compat types and fix streaming trace ID - Add pyright: ignore[reportArgumentType] to SendMessageSuccessResponse id= and result= args in _send_message, and SendStreamingMessageResponse root= in _stream_messages, where a2a-sdk compat types diverge from basedpyright's inferred signature, reducing the reportArgumentType count back within budget. - Fix streaming trace ID in astream_a2a_message to use str(request.id) when available instead of always generating a new uuid4(), restoring JSON-RPC request-ID correlation for observability. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style(a2a): expand SendStreamingMessageResponse for black formatting Move pyright: ignore comment to the root= argument line so Black accepts the expanded multi-line form. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): fix 2 reportArgumentType errors without suppression - main.py: narrow logging_obj from object|None to Optional[Logging] via isinstance check before A2AStreamingIterator call, fixing the "Logging | object" argument type mismatch at line 699. - a2a_endpoints.py: extract response_dict with explicit isinstance(dict) guard before passing to normalize_jsonrpc_response, fixing the "LLMResponseTypes | dict[str, Any]" type mismatch at line 835. - Remove spurious pyright: ignore comments added in previous commits that were not suppressing the actual errors. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(a2a): rewrite upstream URL for 1.0 agent cards in getAuthenticatedExtendedCard 1.0 upstream agent cards store the endpoint URL in supportedInterfaces[0].url rather than a top-level url field. The previous guard only rewrote url when it existed at the top level, so after normalize_agent_card lowered a 1.0 card to 0.3 the upstream internal address leaked into the url field of the 0.3 response. Fix: rewrite both url and supportedInterfaces[0].url to the proxy address before calling normalize_agent_card, ensuring the upstream address is never visible to downstream clients regardless of the upstream card's wire format. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: extend _served_version to all PascalCase methods; add direct httpx-client isolation proof - _served_version now checks `_PASCAL_TO_WIRE` membership instead of two hardcoded names, so GetTask/CancelTask/etc. are promoted to 1.0 wire format alongside SendMessage — prevents mixed wire formats mid-session - test_create_a2a_client_uses_fresh_httpx_client now asserts a2a_client_a._litellm_httpx_client is not a2a_client_b._litellm_httpx_client (direct proof that header bleed cannot occur), in addition to the cache-key inequality check Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: id:0 silently dropped in version_convert; explicit continue in stream retry - version_convert.py: replace `request_id or ""` with `str(request_id) if request_id is not None else ""` in both _send_result_to and _stream_result_to; id=0 is valid JSON-RPC and must not be coerced to "" which breaks response correlation - main.py: add explicit `continue` after the A2ALocalhostURLError retry in _execute_a2a_stream_with_retry so the control flow (retry → next iteration → stream_succeeded guard) is unambiguous Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve a2a retry and discovery card urls * Fix black * Fix test * fix(a2a): avoid KeyError in discovery log after 0.3→1.0 card normalization When a 0.3-style agent card is normalized to 1.0, the top-level url key is replaced by supportedInterfaces; log the already-computed proxy_url instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): preserve taskId when lowering push notification config set params Flatten 1.x create envelope fields before parsing into TaskPushNotificationConfig so 1.0 clients forwarding to 0.3 upstream keep taskId and config. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): ignore unknown fields in message/send proto fallback ParseDict in _build_message_send_params now matches other inbound paths so 1.0 clients with extra proto fields are not rejected with -32602. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): normalize tasks/list params and response across protocol versions Convert list task entries on the response path and lower ListTasksRequest params including status filters when forwarding 1.0 clients to 0.3 upstream. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(a2a): avoid reportArgumentType in _lower_list_tasks_params; use local var instead of _parse return Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(a2a): drop private SDK symbol in tasks/list status lowering _lower_list_tasks_params imported _CORE_TO_COMPAT_TASK_STATE, a private a2a-sdk symbol that could disappear on a patch release and silently break status-filter lowering. Derive the 0.3 wire string from the public protobuf enum name instead (TASK_STATE_<NAME> maps to the 0.3 value once the prefix is dropped and underscores become dashes) and validate the result against the 0.3 TaskState enum's own values via a fully-typed pure helper. Behavior is unchanged for every state; unspecified or unrecognized states still drop the filter. Adds parametrized regression tests covering dashed wire values (input-required, auth-required) and the unspecified drop. * fix(a2a): drop redundant push-notification envelope key; unify MessageToDict import _flatten_create_push_notification_params used `config or pushNotificationConfig`, which short-circuits so a co-present pushNotificationConfig key was never popped and leaked into the flattened params. Pop both keys unconditionally and prefer config when present. Adds a regression test on the helper that fails on the old leak. Also import MessageToDict from a2a.compat.v0_3.conversions in _lower_list_tasks_params to match every other conversion helper in the module instead of pulling it straight from google.protobuf.json_format. * fix(a2a): reject invalid message/stream params early with -32602 _handle_stream_message built MessageSendParams lazily inside the stream_response() generator, so malformed 1.0 params surfaced as a generic -32603 after the 200 status line was already committed. The non-streaming path validates up front and returns -32602 (Invalid params). Validate eagerly before returning the StreamingResponse and emit -32602 on failure so both paths reject malformed params identically. Adds a regression test asserting the streamed error code is -32602. * fix(a2a): raise clear error when non-streaming send ends on an update event _send_message fed the SDK iterator's last event straight into SendMessageSuccessResponse, whose result only accepts Message or Task. A non-standard upstream whose final event is a TaskStatusUpdateEvent or TaskArtifactUpdateEvent made the response construction raise an opaque pydantic ValidationError. Guard the converted result and raise a clear RuntimeError instead, consistent with the no-response guard above it. Adds regression tests for the Message happy path and the update-event rejection via an injected fake client. * test(a2a): lock in clean merged agent-card URL without PROXY_BASE_URL Regression coverage proving _build_merged_agent_card produces no double slash in supportedInterfaces[0].url when PROXY_BASE_URL is unset and request.base_url carries a trailing slash. get_custom_url routes through join_paths, which rstrips the base, so the f-string join stays clean. * style(a2a): modernize type annotations to satisfy strict ruff budget After merging the black->ruff-format migration from base, the A2A files owned by this PR still used Optional[X]/quoted annotations that pushed UP037/UP045 over their lowered ceilings. Convert to X | None, drop the now-unnecessary quoted local annotation in _send_message, and remove the imports left unused by the rewrite. Type semantics are unchanged. * style(a2a): type a2a_endpoints dict params as dict[str, Any] The merge with the formatter-migration baseline tightened the reportUnknownArgumentType ceiling; bare dict annotations made every value Unknown and pushed the codebase total over cap. Annotate the JSON-RPC params, body, metadata, and litellm_params dicts as dict[str, Any] so their values are typed, dropping the unknown-argument count back under the ceiling. No behavior change. * fix(a2a): guard localhost retry against a missing agent card handle_a2a_localhost_retry rewrote the card URL and called create_client with whatever agent_card it received. The caller resolves the card from the SDK client (Optional), so a None card reached set_agent_card_url and create_client, surfacing an opaque SDK error instead of a clear one. Add an early RuntimeError guard mirroring the httpx-client check, drop the now always-true card None-check on the stash line, and cover it with a regression test. * style(a2a): disable reportUnknownArgumentType in a2a-sdk boundary modules The lint env type-checks without the optional a2a-sdk/protobuf installed, so every call into the protobuf-generated compat conversions counts as an Unknown-typed argument and the new A2A code pushed the codebase reportUnknownArgumentType total over its ceiling. These three modules are the A2A SDK boundary; turn the rule off file-wide with a documented reason instead of scattering dozens of per-line ignores across every SDK call. * fix(a2a): tolerate unknown fields when lowering 1.0->0.3; align streaming trace id Two issues greptile flagged: version_convert: the 1.0->0.3 lowering paths (_send_result_to, _task_to, _stream_result_to) called ParseDict without ignore_unknown_fields=True, so a 1.0 upstream response carrying vendor extensions raised and best-effort fell back to passing the un-lowered 1.0 shape to a 0.3 client. Set the flag to match the agent-card path and every inbound path; unknown fields are now dropped and the result is correctly lowered. main.py: asend_message_streaming derived X-LiteLLM-Trace-Id from the JSON-RPC request id, unlike asend_message which uses the logging object's litellm_trace_id. Prefer the logging trace id (then request id, then a uuid) so streamed and non-streamed calls correlate under the same trace. Adds regression tests for both, including the stream-event lowering path. * style(a2a): apply ruff format to a2a protocol and proxy modules Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
234263fdda
|
fix(router): persist global retry_policy via /config/update (#29540)
* fix(router): persist global retry_policy via /config/update (LIT-3152)
The Admin UI Model Retry Settings tab POSTs
{router_settings: {retry_policy: {...}}} to /config/update, but the
field was dropped on two write-side layers so it never reached the
router. UpdateRouterConfig did not declare retry_policy, so
dict(exclude_none=True) stripped it before the DB upsert. And even when
fed directly, Router.update_settings had no "retry_policy" entry in
_allowed_settings, so the assignment was a silent no-op. The DB row
stayed at {"model_group_alias": {}}, llm_router.retry_policy stayed
None, and the UI fell back to defaultRetry = num_retries = 2 on refresh.
Declare retry_policy on UpdateRouterConfig as a plain dict, and add a
retry_policy branch to update_settings that coerces dict payloads to
RetryPolicy before setattr, mirroring Router.__init__. get_settings
already lists retry_policy, so reads work once writes land.
* fix(router): guard retry_policy type in update_settings
Mirror Router.__init__ semantics in update_settings: only assign
retry_policy when it is None or a RetryPolicy (after dict coercion).
Previously a non-dict, non-RetryPolicy value (e.g. a YAML typo like
retry_policy: 5 flowing through /config/update) was stored verbatim,
deferring the failure to request time in get_num_retries_from_retry_policy
instead of being dropped at write time.
* refactor(ui): harden Model Retry Settings flow and validate retry_policy at the boundary
Types UpdateRouterConfig.retry_policy as RetryPolicy and model_group_retry_policy as Dict[str, RetryPolicy] so /config/update validates the payload and rejects malformed counts instead of silently persisting them; the apply path in update_settings keeps coercing the stored dict back to RetryPolicy
Makes the Model Retry Settings tab the single owner of retry_policy and model_group_retry_policy so the generic Router Settings page no longer renders or writes them, replaces the fire-and-forget save with a react-query mutation that only shows the success toast after the write resolves, surfaces real errors, disables Save while in flight, and re-reads authoritative state on success, and sends both the global and per-group policies atomically so edits in the inactive scope are no longer dropped
Decouples the retry-scope selector from the All Models filter and defaults it to Global, seeds the displayed default from num_retries (falling back to 2), and gives per-group rows real inherit semantics so an empty input shows the global value as a placeholder with a Reset control, keeping 0 ("no retries") distinct from inheriting the global value
* fix(keys): align router_settings examples with typed RetryPolicy and resync UI artifacts
model_group_retry_policy is now Dict[str, RetryPolicy], so the {"max_retries": 5} sample in the key-generate test and the /key/generate and /key/update docstrings no longer validate; they now use a valid {"gpt-4": {"RateLimitErrorRetries": 5}} shape.
Regenerated eslint-metrics.json (no-explicit-any drifted 2027 -> 2026) and schema.d.ts (new RetryPolicy schema, retry_policy field, model_group_retry_policy value type) so the UI build and api-types-sync checks pass
* test(router): pin retry_policy persistence end to end (LIT-3152)
The existing retry_policy tests exercise UpdateRouterConfig and Router.update_settings in isolation, so they would all still pass if a regression flipped ConfigYAML.router_settings back to a loose dict or stopped add_deployment from applying the stored row. This drives the real handler chain an Admin UI save triggers: update_config writes the LiteLLM_Config row, the apply path forwards it to the live router, and get_config serializes it back, pinning retry_policy across persist, apply, and read-back.
* fix(teams): use valid model_group_retry_policy example in router_settings docstring
Same stale {"max_retries": 5} example the key endpoints carried; model_group_retry_policy maps a model group to a RetryPolicy, so the team /team/new and /team/update docs now show {"gpt-4": {"RateLimitErrorRetries": 5}}. Regenerated schema.d.ts to match.
* fix(ui): load retry settings via deferred fetch to satisfy set-state-in-effect
The Model Retry Settings effect called loadRetrySettings synchronously; eslint-plugin-react-hooks (react-hooks/set-state-in-effect) traces into it and flags the setState calls, failing frontend-lint. Split the loader into fetchRouterSettings + applyRouterSettings and run the fetch in an inline async IIFE with a cancellation flag, so state is applied in the post-await callback rather than on the effect's synchronous path. Behavior is unchanged and onSuccess still refreshes via loadRetrySettings.
* fix(ui): match CI rendering of RateLimitError 429 docstring in generated schema
gen:api run on a dev env (python 3.13 / newer fastapi) rendered the RateLimitError response description with 4-space indentation, but CI regenerates it with 8-space under its frozen python 3.12 toolchain, which is the canonical committed form. The Check UI API Types Sync job regenerates and diffs, so restore that block to the CI rendering; verified byte-identical to the pre-existing committed version.
* fix(ui): pin RateLimitError 429 docstring to CI's frozen schema rendering
Base #29619 regenerated schema.d.ts on a newer FastAPI that renders the RateLimitError response description at 4-space indent, but the Check UI API Types Sync job regenerates under the frozen python 3.12 toolchain, which renders 8-space. Merging base pulled in the 4-space form; restore the 8-space rendering so the generated types match what CI produces (verified byte-identical to the pre-#29619 committed form), which also corrects the base drift once this PR merges.
|
||
|
ryan-crabbe-berri
|
ac56320f26
|
fix(agents): show an agent's attached virtual key in the UI (#29619)
* fix(agents): show an agent's attached virtual key in the UI
The A2A agent detail view never surfaced which virtual key was attached to
an agent, so after assigning a key during agent creation there was no way to
see it again. Surface the attached key(s) in the agent detail view, derived
from the key table's agent_id foreign key the same way spend is already
joined into the agent response.
Backend adds an agent_id filter to /key/list (mirrors team_id) and enriches
GET /v1/agents and GET /v1/agents/{id} with a non-secret key summary (alias,
masked key_name, hashed token id). The frontend renders a Virtual Keys
section in the agent detail view that lists the agent's keys and links
through to the key detail, and the list view drops its fetch-500-keys-and-
filter-client-side workaround in favor of the enriched response. The orphaned
AgentCard and AgentCardGrid components, left behind when the agent list
switched from a card grid to a table, are removed
* fix(agents): redact attached virtual keys for non-admins
_attach_keys_to_agents joins keys onto the agent response by agent_id with
no caller scoping, but _redact_sensitive_agent_fields never cleared the new
keys field. A non-admin able to view an agent therefore received the alias,
masked name, and hashed token of every key attached to it, including keys
owned by other users or teams; the old client-side path used the scoped
key list, so this was a visibility regression. Clear keys in the redaction
path so only admins see attached-key metadata.
Adds an endpoint-level regression test asserting keys is populated for admins
and null for non-admins, and a list-view test covering the Active vs Needs
Setup badge that lost coverage when the agent card tests were removed.
* fix(agents): satisfy strict lint and resync key/list types
- use builtin list/dict generics in the new agent key helpers to stay
under the UP006 strict-rule ceiling
- swap @tremor/react for antd Typography in agent_virtual_keys (tremor is
being phased out; the new component was the only unsuppressed import)
- regenerate schema.d.ts so the /key/list agent_id query param is typed
* style(agents): prettier-format key hook test and agent_info
|
||
|
Yuneng Jiang
|
76be4461ca
|
feat(ui): give Request Logs columns explicit widths and tighten the dense ones
Now that the page-overflow bug is fixed by letting the main pane shrink, bring back per-column sizing purely to control widths. Columns declare explicit pixel sizes and the table derives its min-width from getCenterTotalSize(), so it stretches to fill a wide card but scrolls once the columns no longer fit. The shared DataTable applies this only when columns declare sizes, leaving the other consumers on their existing fluid layout Trim the columns that were eating horizontal space without earning it: Request ID and Key Hash drop ~30% (Key Hash now narrower than Key Alias, which is the more useful of the two), and Duration and TTFT shrink to fit their short numeric values |
||
|
Yuneng Jiang
|
014754be94
|
fix(ui): let dashboard main pane shrink so wide tables scroll instead of overflowing
The Request Logs page pushed the whole page past the viewport horizontally. The cause was the app shell flex layout: <main className="flex-1"> is a flex item, and flex items default to min-width: auto, so they refuse to shrink below their content's intrinsic width. The logs table is intrinsically ~2300px across its 16 nowrap columns, so main grew to that width and dragged the page with it; the table's own overflow-x-auto wrapper never got the chance to scroll Add min-w-0 to main so it can shrink to the available width, at which point the existing overflow-x-auto wrapper engages and the table scrolls inside its card. This applies to every dashboard page, not just logs Also drop the dead max-w-screen class on the logs container (not a real Tailwind utility, so it was a no-op), and revert the earlier column-sizing attempt which targeted table-layout rather than the actual containment problem |
||
|
Mateo Wang
|
01efcc1b74
|
fix(ui): stop listing bedrock_mantle models under the Bedrock provider (#31478)
The Add Model form's getProviderModels rolled any litellm_provider that
starts with the selected provider's slug into that provider's list. Because
"bedrock_mantle".startsWith("bedrock_") is true, bedrock_mantle/* models
(OpenAI-compatible, served at bedrock-mantle.{region}.api.aws) showed up
under plain Amazon Bedrock, where that model string routes to bedrock-runtime
and fails.
Exclude standalone sub-providers from the prefix rollup so bedrock_mantle/*
only appears under the Amazon Bedrock Mantle provider, while bedrock_converse
and other genuine sub-variants keep rolling up under Bedrock.
|
||
|
Yassin Kortam
|
f55d13ebba
|
fix(team): persist budget_duration on /team/member_add member budgets (#31443)
/team/member_add could not set budget_duration on an individual member budget. add_new_member created the budget row with only max_budget and allowed_models, and TeamMemberAddRequest had no budget_duration field, so a member added with an explicit per-member budget while the team ran a recurring member budget got a lifetime cap instead of a recurring allowance. Thread budget_duration from TeamMemberAddRequest through _process_team_members into add_new_member, and pull the member-budget resolution into a helper that writes budget_duration plus a computed budget_reset_at. When only a budget_duration is supplied and the team has a default member budget, the default is cloned and its reset window overridden so the member keeps the default's max_budget rather than becoming uncapped; a duration with no team default creates a window-only budget. Invalid durations are rejected with a 400 before any DB write, symmetric with /team/member_update. The available-team self-join bypass only grants the ability to join, so reject per-member budget and model controls (max_budget_in_team, budget_duration, allowed_models) for non-admin self-join callers in _validate_team_member_add_permissions, before any DB write. Otherwise a self-joining non-admin could set their own cap, reset window, or model scope past the team default; admins, team admins, and org admins are unaffected and a clean self-join still inherits the team default budget. Resolves LIT-4052 |
||
|
Sameer Kankute
|
133da06aa3
|
chore: litellm oss staging (#31185)
* fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped
The Total Tokens Over Time and Total Requests Over Time AreaCharts on the
Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow
once totals pass the hundred-million mark — leading digits of labels like
"100.00M" / "4500.00M" got clipped against the chart edge. The requests
chart was worse: it formatted with toLocaleString(), so billion-scale
request counts produced "1,000,000,000" (13 chars) and overflowed
immediately.
Fix in two places so neither alone has to carry the whole margin:
- activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and
switch the requests chart to the shared valueFormatter so it uses the
same compact k/M/B suffixes as the tokens chart.
- value_formatters.tsx: add a >= 1e9 branch to valueFormatter /
valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping
every formatted label at most 7 chars.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* docs(readme): add Deploy on AWS/GCP with Terraform section
Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.
Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): add 1-click deploy buttons for AWS + GCP
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.
AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): move AWS + GCP deploy buttons next to Render button
* docs(readme): unify deploy button sizes and badge styles
* docs(readme): bump deploy button height to 48 to match Render/Railway
* docs(readme): bump AWS/GCP badge height to compensate for SVG padding
* docs(readme): bump AWS/GCP badge height to 72
* docs(readme): bump AWS/GCP badge height to 84
* fix(readme): make deploy buttons same height (48px)
https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc
* docs(readme): flag GCP project ID substitution in image_registry
* docs(readme): equalize deploy button heights and fix Cloud Shell button font
GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.
Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
* docs(readme): collapse Railway deploy anchor to a single line
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
* Add Claude Fable 5 cost map entries as a data-only hotfix
Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano
Three bugs in model_prices_and_context_window.json:
1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens
were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K
max output, but the values were set as max_input=128000,
max_tokens=272000. This caused token limit errors when sending
prompts over 128K tokens to GPT-5 Pro.
2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was
272000, but GPT-5.4 Mini shares the same 1,050,000 token context
window as GPT-5.4. This was inconsistent with the azure/ variants
which already correctly had 1,050,000.
3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini,
max_input_tokens was 272000 instead of 1,050,000.
Source: OpenAI model documentation and contextwindows.dev which
aggregates official context window sizes.
Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini
should be 400K; their 272K values are correct per OpenAI docs)
* fix: also correct max_output_tokens for gpt-5-pro (272000→128000)
Per reviewer feedback, max_output_tokens was left at 272000 while
max_tokens was corrected to 128000, causing an internal inconsistency.
Both should be 128000 per OpenAI docs.
* fix(cost): price gpt-image generated output tokens as image tokens (#31147)
The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return
usage with no output token breakdown — litellm's `ImageUsage` has no
`output_tokens_details` field — so generated-image OUTPUT tokens were priced at
the text rate (`output_cost_per_token`) instead of the image rate
(`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x
undercount on the dominant cost component (image output is ~74% of spend). This
also affects azure gpt-image, which shares this calculator.
The OpenAI gpt-image cost calculator re-implemented usage handling instead of
reusing `calculate_image_response_cost_from_usage`, the shared helper that
azure_ai/gemini/vertex_ai already use. That helper classifies generated output
tokens as image tokens when the provider does not itemize output, and splits
text/image when it does.
Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage`
(pre-transformed chat Usage objects are still costed directly). Adds a regression
test for the no-breakdown ImageUsage case (gpt-image-2).
* fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098)
A bare application-inference-profile ARN passed as bedrock/arn:... fell
through to the invoke route, which cannot derive a provider from the
opaque profile id and raised 'Unknown provider=None'. The converse route
needs no provider, so detect these ARNs in get_bedrock_route and route
them to converse, matching the behavior of the already-documented
bedrock/converse/arn:... workaround.
Explicit invoke/ prefixes still win, and they remain a dead end for these
ARNs by design (no provider derivable). System-defined inference-profile
ARNs that embed a known model, and other opaque ARN types
(provisioned-model, imported-model, custom-model-deployment) that are
frequently invoke-only, are deliberately left on their current routes;
tests guard both boundaries.
* fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060)
_add_tool_choice_required_message appended the "select a tool" prompt to
the caller's messages list in place, so transform_request corrupted the
caller's conversation history and appended a duplicate prompt on every
retry. Build and return a new list instead so the call stays idempotent.
Adds a regression test asserting the input messages list is unchanged
across repeated transform_request calls.
Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com>
* fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996)
gpt-4o-transcribe and compatible ASR backends return a diarized_json
response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8.
TranscriptionUsageDurationObject typed seconds as int, so parsing the
response raised a pydantic ValidationError (int_from_float). That error
surfaces as an APIConnectionError which the router treats as retryable, so
it keeps re-calling the upstream (200 every time) until the upstream
rate-limits and returns 429 to the caller.
OpenAI specs this field as a float (see openai SDK UsageDuration.seconds),
so widen seconds to float. With the parse succeeding there is no exception
left to retry, which removes the loop.
Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com>
* fix(deepseek): drop non-function tools before chat completions call (#30910)
* fix(deepseek): drop non-function tools before chat completions call
DeepSeek's /chat/completions only accepts tools of type "function".
Requests bridged from /v1/responses can carry responses-API-native tool
types, for example a Codex CLI tool typed "namespace", which DeepSeek
rejects with "unknown variant 'namespace', expected 'function'" so the
whole request fails (issue #30722).
Filter unsupported tool types in the DeepSeek request transform so the
function tools still go through; when nothing callable remains, also drop
the now-dangling tool_choice and parallel_tool_calls
Fixes #30722
* test(deepseek): cover async tool filtering and document tool_choice assumption
Add an async_transform_request regression test so the sync and async tool
filtering paths cannot silently diverge, and document in _drop_unsupported_tools
that only non-function tools are dropped, so a function-named tool_choice always
references a surviving tool
* feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840)
* feat(ui): surface team budget on key overview when key has no own budget (#30801)
* feat(ui): surface team budget on key overview when key has no own budget
* fix(ui): replace IIFE with derived variable and use find() for team budget display
* fix(anthropic): emit replayable streaming thinking blocks (#31022)
* feat(proxy): read cold-storage prompts back in the logs detail view (#30364)
* feat(proxy): read cold-storage prompts back in the logs detail view
When a deployment offloads prompts and responses to cold storage instead of
Postgres, the spend-log row holds only "{}" placeholders plus a
metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed
nothing. The detail endpoint only read the placeholder columns and never
fetched the object back.
Resolve the payload per row based on actual content, not a config flag: if
Postgres has content, return it; otherwise read the exact stored object key and
fetch from the configured cold storage backend through ColdStorageHandler.
Reading the persisted key is a single GET. The key embeds a microsecond
timestamp that cannot be reconstructed from the millisecond-precision startTime
column, and listing the day's prefix to match on request_id would be too
expensive for this per-open path.
Also teach the detail drawer's pretty-view parser to accept a bare messages
array. The cold storage payload carries the prompt as a top-level messages list
with no proxy_server_request, so without this the output rendered while the
input stayed blank.
ColdStorageHandler gains an optional injected logger so the resolver can be unit
tested without monkeypatching. Postgres-stored prompts are unaffected: the fast
path returns the existing columns and the request-body object still renders the
same way.
* Update litellm/proxy/spend_tracking/spend_management_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure
Add unit tests for ColdStorageHandler (injected logger, graceful None when no
logger is configured, and resolution of a configured logger from the callback
registry) and a regression test asserting a cold storage backend exception
degrades to the Postgres values instead of surfacing a 500.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068)
* fix(mavvrik): advance metricsMarker after upload + fix scheduler startup
Two bugs fixed:
1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a
successful GCS upload, so metricsMarker stayed at 0 and every daily run
re-exported the same dates in an infinite catch-up loop.
Fix: add _update_metrics_marker(date_epoch) called at the end of deliver()
after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS
file is already committed). A 410 raises consistent with the rest of the
destination.
2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call
has triggered lazy instantiation of MavvrikFocusLogger, so it found no
logger instance and silently skipped registering the daily export job.
Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call
_init_custom_logger_compatible_class to force instantiation before
the APScheduler job is registered.
* fix(mavvrik): catch up from earliest window when metricsMarker=0
When the connector is freshly registered, metricsMarker=0 parses to None.
The catch-up block was guarded by `if last_ingested and ...` which skipped
it entirely for None, so only yesterday was exported instead of the full
_MAX_CATCHUP_DAYS window.
Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup).
The existing > 7 day warning only fires for non-None markers that are old.
* fix(mavvrik): use now as end_time for yesterday's export window
LiteLLM_DailyUserSpend rows for a given date get their updated_at
bumped by the spend flush job throughout the next morning. The core
database query filters on updated_at, so capping end_time at midnight
(yesterday + 1 day) missed any spend rows flushed after midnight.
Fix: pass now (cron fire time) as end_time for the daily "yesterday"
window so all fully-settled rows are captured regardless of when the
flush job ran.
Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per
row in the exported FOCUS CSV.
* fix(mavvrik): also use now as end_time for catch-up windows
* fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class
Calling it with only logging_integration raised TypeError at proxy startup
because internal_usage_cache and llm_router have no defaults. Also fix test
name to reflect the actual status code (5xx not 4xx) used in the mock.
* ci: retrigger CI run
* feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757)
* Add optional `instruction` passthrough to the rerank API
vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction`
field (folded into the model's chat_template_kwargs and consumed by the
chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently
dropped it: RerankRequest / OptionalRerankParams had no such field, so the
outgoing body was rebuilt without it.
Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(),
get_optional_rerank_params, and the hosted_vllm transformation into the
request body, only when non-None. When callers omit it, model_dump(exclude_none)
drops the field and the outgoing request is byte-for-byte unchanged — fully
backward-compatible. (DeepInfra already forwards `instruction` via
non_default_params; this formalizes the field in the shared types.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: thread `instruction` as a typed param + cover rerank_utils
Per PR review (greptile P2 + codecov):
- Make `instruction` a typed, named argument on the rerank provider interface
instead of recovering it from the opaque `non_default_params` blob. Adds
`instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params`
and every provider override, and forwards it explicitly from
`get_optional_rerank_params`. hosted_vllm now reads the named param directly.
It is still also surfaced in `non_default_params` so providers that read it
there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction`
as a named param rather than leaving it in **kwargs.
- Add get_optional_rerank_params unit tests (present + absent) to cover the
previously-uncovered threading line flagged by codecov.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scan rerank `instruction` through request guardrails
The rerank guardrail translation (CohereRerankHandler.process_input_messages)
only scanned `query`, so the newly added `instruction` field reached the
backend model unscanned. Since instruction-aware rerankers (hosted vLLM /
Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller
could place content there to bypass configured rerank request guardrails.
Generalize the handler to scan every user-controlled text field (`query` and
`instruction`) in one apply_guardrail call and write each sanitized value back
by index. Query-only requests are unchanged (single-element list at index 0);
non-string fields are left untouched. Adds tests covering instruction
scanning, PII masking write-back, and the non-string case.
Addresses the Veria AI security review on PR #30757.
* test: narrow Optional results before len() to satisfy basedpyright budget
The lint gate (basedpyright delta-vs-base budget) flagged one new
reportArgumentType: len(result.results) where results is
List[RerankResponseResult] | None. Assert results is not None first to
narrow the type before len()/indexing.
* fix: read rerank `instruction` from kwargs to satisfy basedpyright budget
The basedpyright delta-vs-base gate flagged one new reportArgumentType: the
Router forwards rerank calls via an untyped `**kwargs` unpack
(`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a
typed named param on the public `rerank`/`arerank` entrypoints made pyright
check that key against `str | None`, adding an error at router.py with no real
safety gain. Read `instruction` from kwargs in `rerank` instead.
It remains fully typed where it matters - threaded as a typed argument through
`get_optional_rerank_params` and each provider's `map_cohere_rerank_params`
(the original Greptile P2 ask). Whole-repo reportArgumentType is back to the
base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(github_copilot): synthesize empty choices at the provider seam (#30929)
Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with
choices=[], either carrying Anthropic-native content blocks or, for the
max_tokens=1 probe Claude Code sends, no content at all. github_copilot
is dispatched through the OpenAI SDK handler, which calls
convert_to_model_response_object directly and never invokes
GithubCopilotConfig.transform_response, so the empty-choices guard there
surfaced as a 500
Instead of synthesizing choices inside the shared
convert_to_model_response_object (which would silently turn empty choices
into a fabricated success for every provider), add a no-op
transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig
overrides it to synthesize choices from Anthropic-native content, reusing
its existing parsing, and the OpenAI SDK handler routes its parsed
response through the hook before generic conversion. The core utility
keeps treating empty choices as an error for all other providers
Fixes: https://github.com/BerriAI/litellm/issues/30927
Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com>
* fix(router): stop fallback lookups from mutating the router fallbacks config (#30624)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in
|
||
|
Sameer Kankute
|
687a62e561
|
fix(cli): mint per-session agent credential on lite login (#31072)
* fix(cli): mint per-session agent credential on lite login
The `lite login` command was producing a shared UI session token that broke agent use in three ways: a $0.25 budget cap (from max_ui_session_budget) that killed agent sessions in minutes, a fixed identity "cli-jwt-token" shared across every user preventing per-session spend attribution, and auth gated behind EXPERIMENTAL_UI_LOGIN so the token was rejected on default deployments.
This fixes all three. Each login now generates a unique cli-session-{uuid} token with no per-key budget cap (enforced via shared team/user counters instead), and the decrypt path activates for any non-sk- token without requiring EXPERIMENTAL_UI_LOGIN.
* fix(cli): address review feedback on EXPERIMENTAL_UI_LOGIN gate and e2e test
Restore EXPERIMENTAL_UI_LOGIN=false as an explicit opt-out: operators who set it to false keep the old boundary; unset (new default) and true both attempt NaCl decryption, which fails closed for non-blob tokens.
In the e2e test: replace the silent Redis fallback with pytest.skip so a missing Redis instance is explicit rather than silently degrading to a directly-minted token. Write the seeded flow back as JSON (proxy reads it via json.loads on cache fetch) instead of Python repr, and build the updated flow immutably.
* fix(key-management): cap CLI session token delegation budget to team ceiling
A CLI session token intentionally carries max_budget=None to avoid a per-session LLM spend cap. The key-generation delegation check (GHSA-q775-qw9r-2r4g) previously skipped non-admin callers with max_budget=None, treating them as having unlimited delegation authority. This allowed any internal user with a lite login session to mint virtual keys with arbitrary budgets.
Adds is_session_token=True to UserAPIKeyAuth for CLI session tokens and uses the caller's team budget as the delegation ceiling in that case, so the effective limit is min(requested_budget, team.max_budget) rather than unbounded.
* chore: regenerate dashboard OpenAPI types
The is_session_token field added to UserAPIKeyAuth cascades to the
dashboard schema. Regenerate types from the updated OpenAPI spec.
* fix(key-management): block personal key budget delegation from CLI session tokens
When team_table is None (personal key, no team_id in request), the personal key
has no team-budget enforcement at request time. A session token therefore cannot
delegate any explicit max_budget for a personal key -- that would open a budget
bypass path. Block the request with a clear 400 directing the caller to use a
team_id instead.
* test(auth): add unit coverage for non-admin CLI session token production path
* fix(type-check): use model_validate in _return_user_api_key_auth_obj to fix reportArgumentType gate
UserAPIKeyAuth(**user_api_key_kwargs) spread triggers a basedpyright
reportArgumentType error for each named field in UserAPIKeyAuth because
the dict's inferred value type (str | Span | LitellmUserRoles | Unknown)
is not assignable to each field's specific type. Adding is_session_token:
bool introduced +2 more such errors, breaching the gate cap.
model_validate accepts an untyped dict without per-field argument checking,
which eliminates the +2 new errors and also ratchets down the pre-existing
333 errors at those call sites. basedpyright-code-budget.json is updated
to reflect the new lower baseline (1814, down from 1934).
* fix(type-check): ratchet down reportArgumentType baseline only
The previous lint-budget-update captured all baselines from the local
environment, raising many ceilings vs the merge-base and failing the
non-gating budget_ratchet_check. Restore staging's values for every
rule and only lower reportArgumentType (1934 -> 1814) to reflect the
reduction from switching to model_validate in _return_user_api_key_auth_obj.
* fix(auth): set max_budget on CLI session token to enforce max_ui_session_budget
CLI session tokens were missing max_budget, so _virtual_key_max_budget_check
had no per-session ceiling to enforce. Operators relying on max_ui_session_budget
could be bypassed for the full token lifetime. Mirrors the existing UI token path.
* revert(auth): remove max_ui_session_budget from CLI session token
max_ui_session_budget defaults to $0.25 and is sized for the UI chat
pane (10-min sessions). CLI sessions are 24-hour tokens for real work;
capping them at that ceiling would throttle users under their actual
user/team budget. Budget enforcement for CLI sessions is via the shared
user and team counters as originally intended.
* fix(auth): cap CLI session at max_ui_session_budget only when user and team have no budget
When neither the user nor their team has a budget configured, CLI sessions
were fully uncapped. The poll endpoint now looks up the real user and team
objects from DB; if both have no max_budget, it passes litellm.max_ui_session_budget
as the token's per-key ceiling. Users or teams that already have a budget
configured are unaffected and continue to rely on the shared counters.
* fix(auth): fix black formatting and update test mock for cli_poll_key budget lookup
The get_user_object and get_team_object async calls in cli_poll_key were
not mocked in the existing test, causing MagicMock await errors. Patch
both functions at the auth_checks module level. Also apply black formatting
to ui_sso.py which CI rejected.
* fix(auth): skip fallback budget cap when team lookup fails for cli session token
* test(auth): pin cli session budget cap to user/team budget presence
The session_max_budget fallback in cli_poll_key only applied
max_ui_session_budget when neither the user nor the resolved team had a
budget. The existing coverage exercised only the team-lookup-failure
branch. Add two regression tests: a user with a configured budget must
not receive the fallback cap, and a session with no user and no team
budget must fall back to max_ui_session_budget. Mutating either guard
out of the branch now fails these tests.
* fix: remove CLI poll session budget cap
* revert(auth): restore CLI session fallback budget cap
Bugbot autofix (
|
||
|
Yuneng Jiang
|
93aca51251
|
Merge branch 'litellm_internal_staging' of github.com:BerriAI/litellm into litellm_/cranky-hamilton-21b5d0 | ||
|
Yuneng Jiang
|
4a6f0dbd8c
|
fix(ui): size Request Logs table columns so it scrolls instead of overflowing
Tremor's Table forwards className to a wrapper div rather than the inner table element, so the table-fixed class never reached the table and it stayed table-layout: auto. Across 16 whitespace-nowrap columns that expanded the table far past the viewport Give each spend-logs column an explicit pixel size and drive the table width from getCenterTotalSize(), matching the Virtual Keys table. The shared DataTable applies this only when columns declare sizes, so the other consumers keep their existing fluid layout |
||
|
Sameer Kankute
|
7eacdd5258
|
chore: litellm oss staging 250626 (#31305)
* fix(anthropic): support Bearer auth for custom api_base endpoints (Fixes #30926) * style: format common_utils.py with black * fix(anthropic): extract api_base from litellm_params in batches/files validate_environment * fix(anthropic): scope Bearer key check to custom api_base endpoints * fix(streaming): reset Anthropic message_start cursor (output_tokens=1) when no message_delta arrives The Anthropic streaming protocol emits `message_start.usage.output_tokens=1` as a placeholder cursor; the real cumulative output count only arrives in the final `message_delta` event. When a stream is cancelled before `message_delta` lands (common for thinking models on long-tail prompts), ChunkProcessor._calculate_usage_per_chunk's last-wins accumulator left completion_tokens stuck at 1. Because 1 is truthy, the `completion_tokens or token_counter(text=...)` fallback in calculate_usage() never fired, and requests were billed for 1 output token even when several thousand tokens of text had actually streamed. Fix: track whether any chunk's completion_tokens exceeded 1 (saw_non_cursor_completion). If the only update we saw was the cursor, reset completion_tokens to 0 so the text-based fallback estimates from the real completion content. Legitimate 1-token completions (model returns "Yes." etc.) are unaffected in practice — token_counter on a 1-token completion_output also yields ~1, so billing stays approximately correct. Tests: - TestAnthropicCursorBug (6 cases) — pins the post-fix behavior - TestNonAnthropicStreamingIntact (2 cases) — guards against regression on providers without the cursor pattern All 8 new tests pass; 9 existing streaming_chunk_builder_utils tests still pass. * fix(streaming): scope cursor reset to anthropic provider + recognize message_delta arrival Addresses both Greptile P2 threads on PR #30420: CLASS A — Anthropic-specific heuristic was applied globally ============================================================ The `completion_tokens == 1 and not saw_non_cursor_completion` reset lived in provider-neutral `streaming_chunk_builder_utils.py`. Any non-Anthropic provider that legitimately reports completion_tokens=1 in a single usage chunk (perfectly normal for short OpenAI / Bedrock / Vertex single-token replies with stream_options.include_usage=true) would have its value silently rewritten to 0 and re-billed via token_counter — producing a different number than what the provider actually charged. Fix: gate the reset on `custom_llm_provider == "anthropic"`, resolved from the first chunk's `_hidden_params` (the same field set by streaming_handler.py:722 on the live path). Unknown / missing provider is treated as non-Anthropic and skips the reset, so newer providers and custom plugins are also safe by default. CLASS B — `saw_non_cursor_completion` missed legitimate single-token replies ============================================================ Previous condition was `usage_chunk_dict["completion_tokens"] > 1`, which never fires for an Anthropic stream where the model legitimately emits exactly one output token (e.g., "Yes."). Anthropic still sends message_start (output_tokens=1, the cursor) AND message_delta (output_tokens=1, the real value) — same value, but two distinct usage events. The old check couldn't tell that apart from a cancelled stream where only message_start landed. Fix: track `completion_usage_updates` and flip `saw_non_cursor_completion` when EITHER (1) the value exceeds 1 (definitely not a placeholder), OR (2) we've seen >=2 completion-bearing usage events (positive evidence that message_delta arrived). Cancelled cursor-only streams still have exactly one event and still hit the reset; cache chunks with completion_tokens=0 don't count toward the threshold. Tests ============================================================ - _make_chunk now sets `_hidden_params["custom_llm_provider"]` (default "anthropic") so the gate is exercised by every existing test — none of them needed assertion changes besides the legitimate-single- token case, which now expects exactly 1 (was a fuzzy 0..3 range). - New: test_anthropic_cache_only_chunks_after_message_start_still_resets - New: test_non_anthropic_provider_completion_tokens_one_not_reset - New: test_unknown_provider_completion_tokens_one_not_reset 11/11 tests pass. * chore: add Co-authored-by trailer for attribution Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> * fix(anthropic): preserve messages cache usage * style(anthropic): format messages cache usage helper * fix(anthropic): accept integral float cache token counts Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(anthropic): accept integral float cache token counts * test(anthropic): cover cache usage edge cases * fix(gemini): preserve thoughtSignature for server-side tool responses When Gemini API returns toolCall and toolResponse parts, they might have different thoughtSignatures. Previously, LiteLLM merged them into a single dict, overwriting the response's thoughtSignature with the call's. This fix extracts them separately and re-injects them correctly. TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * fix(gemini): address PR comments on thoughtSignature handling - Fix orphan-response thoughtSignature regression by copying thought_signature to response_thought_signature - Add missing assertions in existing tests - Add new unit tests for orphan-response signature handling TAG=agy CONV=755b21d0-3200-40bc-bd1a-bb58a378a9a6 * feat(mcp): include server alias and server_id in mcp_info response - Add alias and server_id fields to mcp_info object in /mcp-rest/tools/list endpoint - Update rest_endpoints.py to surface alias from server config - Add test coverage in test_mcp_server.py and test_rest_endpoints.py Fixes #31015 * fix(proxy): reject non-finite spend via validate_finite_spend A NaN/-inf spend would bypass spend >= max_budget enforcement. Add a shared finite-value guard, defined above the litellm.proxy.* imports to avoid the module-level cyclic-import warning. * fix(proxy): require admin for any /key/update spend, reject non-finite Gate the admin check on the presence of `spend` (not a value diff): the DB spend lags the live cross-pod counter, so an "unchanged" spend on the non-admin path let a key owner / team member overwrite the live counter below real usage. Also reject NaN/+-inf spend before the DB write. * fix(proxy): invalidate spend counter on /user/update spend change A direct spend change on /user/update wrote the DB row but left the warm cross-pod counter at the stale value, so enforcement kept reading the old spend. Invalidate spend:user:{user_id} after the write (reseed-from-DB), and reject non-finite spend before the write. * fix(cache): route Bedrock semantic-cache sync embedding through the Router (#28244) The semantic cache's embedding model is a proxy Router alias whose AWS credentials (aws_role_name, aws_session_name) live only in the Router deployment's litellm_params. The sync embedding paths called litellm.embedding() directly, bypassing the Router, so they could neither resolve the alias nor assume the configured role; cross-account Bedrock semantic caching failed with "bedrock:InvokeModel is not authorized". On Redis this surfaced at proxy startup because redisvl's CustomTextVectorizer eagerly fires a dimension-probe embedding during cache construction, while llm_router is still None. Fix A: make the sync paths mirror the already-correct async paths. A shared, dependency-injected helper (litellm/caching/_embedding_router.py) decides whether to route through llm_router.embedding(...) when the model is a Router deployment, else fall back to direct litellm.embedding(...). Redis and qdrant sync set_cache/get_cache now precompute the embedding and pass vector= to the backend, exactly as the async astore/acheck already do. Both async _get_async_embedding methods are unified onto the same helper and now forward the caller's full metadata instead of a hand-picked subset. Fix B (Redis only): defer redisvl index construction from __init__ into a lazy, memoized llmcache property, so the dimension-probe embedding fires on first cache use, after llm_router is wired. A failed build is not memoized, so a transient outage recovers on the next request. Known limitation: resolve_embedding_router gates on an exact model-name match (same as the shipped async path); wildcard/alias/team-public routes still fall back to direct embedding. Tracked as a follow-up. * fix(cache): harden embedding-router and shrink Any surface (review) Address review feedback on the semantic-cache aws-role fix (#28244): - resolve_embedding_router now skips deployment entries missing model_name instead of raising KeyError on a malformed model_list (Greptile P2); add a regression test that fails on the old direct-key access. - Replace the `**kwargs: Any` passthrough on the four cache _get_embedding / _get_async_embedding helpers with an explicit, typed `metadata: Optional[Dict[str, Any]] = None` parameter. The helpers only ever consumed kwargs["metadata"], so this is behavior-preserving, makes the forwarded field obvious at the call site, and removes three bare-Any annotations (keeps the strict-rule ANN401 budget within ceiling). - Note in _build_llmcache that redisvl's dimension-probe embedding adds one extra billable embedding on the first cache request (Greptile P2). * fix(bedrock_mantle): correct responses routing for openai.gpt-5.x models Dashboard Test Connection for bedrock_mantle/openai.gpt-5.4 and openai.gpt-5.5 was failing with maximum recursion depth errors and "model does not exist" Route detection in the bedrock provider matched route tokens by plain substring, so the bedrock_mantle/ prefix was mistaken for the mantle/ invoke route and the body model was rewritten to bedrock_openai.gpt-5.5; route tokens now only match at a path-segment boundary so the bare model name is preserved A responses-mode model whose provider has no responses config bounced forever between the responses API and chat completions; the responses to completion fallback now tags its call so completion() does not bridge back, breaking the loop The Test Connection endpoint hardcoded the test mode to chat, which disabled mode auto-detection for responses-only models; the default is now None so the mode is detected from model capabilities acompletion() now drops a duplicate acompletion kwarg before building the partial and treats model_info=None as an empty dict to avoid a NoneType crash * test(bedrock_mantle): cover route guard and bridge flag; fix reportArgumentType regression Adds the regression coverage codecov flagged on the two responses to completion bridge guard lines and the bedrock route-prefix helper. The handler tests drive both the sync and async fallback paths with litellm.completion and litellm.acompletion mocked, and assert the forwarded kwargs carry _skip_responses_api_bridge=True, so dropping either flag line fails the suite. The common_utils tests assert that bedrock_mantle/openai.gpt-5.x no longer resolves to the mantle route while the genuine mantle/ and bedrock/mantle/ ids still do, exercising both branches of _model_has_route_prefix. Also aligns update_messages_with_model_file_ids model_id to Optional[str], matching its Responses API sibling, so the defensive model_info fallback no longer introduces a new reportArgumentType in completion(); the file-id lookup narrows model_id before the dict get * chore(ui): sync generated OpenAPI types for optional test_connection mode The test_model_connection mode body param default changed from chat to None so the mode is auto-detected from model capabilities, which makes the field optional in the proxy OpenAPI spec. Regenerate the committed schema so the dashboard types match: mode becomes optional and the description and default JSDoc follow the spec, keeping the Check UI API Types Sync gate green * refactor(bedrock): match all explicit route prefixes at path-segment boundary Migrates the remaining substring route checks to the existing _model_has_route_prefix helper so every explicit route token matches only as a leading path segment, consistent with get_bedrock_route and the mantle route. Covers _explicit_converse_route, _explicit_claude_platform_route, _explicit_invoke_route, _explicit_agent_route, _explicit_agentcore_route, _explicit_converse_like_route, _explicit_async_invoke_route and _explicit_openai_route. This also stops invoke/ from substring-matching async_invoke/. Route precedence and order are unchanged, and a note on the segment invariant is added to the helper docstring * test(bedrock): cover explicit route prefix segment matching Exercises all eight migrated _explicit_*_route helpers (converse, converse_like, invoke, async_invoke, agent, agentcore, claude_platform, openai) directly: each matches its token as a leading path segment and rejects the token glued to a preceding segment, so reverting any method to the old substring check fails the suite. Also asserts invoke/ no longer matches async_invoke/ models, the concrete improvement of the segment-boundary migration * test(proxy): assert negative spend is allowed (one-time grant use-case) Negative spend is intentionally permitted so admins can grant extra allowance for the current budget period only, without raising the recurring budget ceiling. Cover it explicitly in validate_finite_spend and via the /user/update invalidation test. * fix(google_genai): forward native generateContent top-level fields Google's native generateContent REST body carries safetySettings, toolConfig, cachedContent and labels at the top level as siblings of generationConfig. The proxy's :generateContent endpoint spread them into agenerate_content as loose kwargs and then dropped them, so callers had to wrap them in extra_body for them to take effect; safetySettings, for instance, was silently ignored The provider config now exposes the native top-level field names and setup_generate_content_call collects whichever are present, merging them into the outgoing request body through the existing extra_body merge so they reach Google verbatim. An explicit extra_body still wins on conflict. The sync generate_content_stream path now also forwards systemInstruction, matching the other three entry points Fixes #12671 Claude-Session: https://claude.ai/code/session_016MFtMXokCjT8u6mvyASudK * fix(proxy): resolve env refs for DB-stored models * fix(proxy): restrict DB env ref resolution * fix(proxy): block team DB env ref resolution * fix(lint): resolve ANN401/UP045/C901 strict-gate violations - Replace Optional[X] with X | None (UP045) in 8 files - Replace Any return/param types with concrete types or object (ANN401) - Extract _make_api_key_auth_header helper to reduce get_anthropic_headers complexity below C901 threshold (17 → 14) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): preserve x-api-key for custom endpoints; opt-in Bearer via prefix Users who pass a key already prefixed with "Bearer " get Authorization: Bearer. All other keys continue to use x-api-key, preserving backward compatibility with custom api_base endpoints that expect x-api-key rather than Authorization. Also consolidates get_auth_header to reuse _make_api_key_auth_header helper, eliminating the duplicated custom-endpoint routing logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(anthropic): restore Bearer routing for non-sk-ant- keys on custom api_base The backwards-compat change broke existing tests that verify the intentional Bearer-for-custom-base behavior (Fixes #30926). Restore original logic while keeping the _make_api_key_auth_header helper for code deduplication. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(anthropic): gate Bearer-for-custom-base behind use_bearer_for_custom_base flag Previously the auth-header switch from x-api-key to Authorization: Bearer applied unconditionally for non-sk-ant- keys on a custom api_base, silently breaking existing deployments that proxied to gateways expecting x-api-key. Introduce use_bearer_for_custom_base: bool = False on _make_api_key_auth_header, get_anthropic_headers, and get_auth_header. validate_environment reads it from litellm_params so callers can opt in per-model without any API surface change. Tests updated to pass use_bearer_for_custom_base=True where Bearer behavior is asserted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(redis): apply namespace prefix in delete_cache and async_delete_cache (#29981) DEL was the only Redis cache operation that skipped check_and_fix_namespace, so it targeted the raw SHA256 hash (e.g. 3997c4...) rather than the namespaced key (litellm:3997c4...). This caused two problems: a Redis NOPERM error on deployments with an ACL restricting DEL to the litellm:* pattern, and a silent no-op on all other deployments since the un-prefixed key was never stored. * style(anthropic): reformat common_utils.py with Black (--target-version py312) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: preserve cache metadata and spend counters * style: apply ruff format to streaming_iterator.py * refactor: reduce complexity of usage/spend helpers to satisfy strict ruff gate Extract Anthropic message_start cursor reset into _reset_anthropic_cursor_completion_tokens and the cross-pod spend-counter invalidation into _invalidate_user_spend_counter_if_changed, keeping both _calculate_usage_per_chunk and _update_single_user_helper under the max-complexity ceiling. Use builtin generics in the new signatures so no new UP006 violations are introduced. Behavior unchanged. --------- Co-authored-by: rupak-eng <rupakji99@gmail.com> Co-authored-by: songkuan-zheng <252822057+songkuan-zheng@users.noreply.github.com> Co-authored-by: songkuan-zheng <songkuan-zheng@users.noreply.github.com> Co-authored-by: Kannan Priyadharshan <kpd2204@gmail.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Marco Georgaklis <mgeorgaklis@google.com> Co-authored-by: Anjaiah Methuku <anjaiahspr@gmail.com> Co-authored-by: Andrii Butko <booandrew23@gmail.com> Co-authored-by: Kent <kingdooo@gmail.com> Co-authored-by: kunal2002 <k.nayyar2002@gmail.com> Co-authored-by: Ali Khan <alirazakhan.offi@gmail.com> Co-authored-by: jesco-absolut <team@srswti.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Matt Hill <mhill@dataminr.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Mateo Wang
|
6db55e0aa5
|
feat(mcp): add mcp_xff_num_trusted_hops to harden X-Forwarded-For client IP resolution (#31257)
* feat(mcp): add mcp_xff_num_trusted_hops to harden XFF client IP resolution MCP per-server IP access control reads the client IP from X-Forwarded-For and trusts the leftmost entry. Behind an append-style proxy or load balancer (AWS ALB, nginx with $proxy_add_x_forwarded_for, HAProxy, Envoy, Cloudflare), a client can prepend an arbitrary value to the header, so the leftmost entry is attacker-controllable even when the direct peer is a trusted proxy. An attacker can therefore spoof an internal IP and reach servers marked available_on_public_internet=false. This adds an optional mcp_xff_num_trusted_hops general setting modelled on Envoy's xff_num_trusted_hops. When set to N, the client IP is read N entries from the right of the chain (where N is the number of trusted appending proxies in front of the gateway) instead of the leftmost value, so any entries a client prepends are ignored. It composes with mcp_trusted_proxy_ranges, which still validates the direct peer, and only takes effect once that check passes; without a validated direct peer the gateway keeps failing closed, so hop counting cannot be abused by a direct-to-pod attacker. The chain must contain at least N valid entries or resolution fails closed. Default is unset, preserving existing behaviour. * chore(ui): regenerate dashboard schema for mcp_xff_num_trusted_hops * fix(mcp): warn when mcp_xff_num_trusted_hops is below the minimum A 0 or negative value is silently treated as disabled, which could leave an operator believing they enabled append-style X-Forwarded-For hardening while client IP resolution stays on the spoofable leftmost value. Emit a warning, consistent with how the module already surfaces invalid CIDR config, so the misconfiguration is visible in logs. * fix(mcp): reject mcp_xff_num_trusted_hops < 1 at config-parse time Add a ge=1 bound to the ConfigGeneralSettings field so the update_config_general_settings path rejects 0 and negative values with a clear validation error instead of accepting them, and self-documents the valid range. The runtime warning stays as defense-in-depth for raw-dict config that bypasses model validation. * style(mcp): black-format ip_address_utils.py * fix(mcp): fail closed when mcp_xff_num_trusted_hops is set but invalid A present-but-invalid mcp_xff_num_trusted_hops (non-integer, or below 1) previously made _resolve_num_trusted_hops return None, which the caller treated identically to "unset" and silently fell back to the legacy leftmost X-Forwarded-For value. An operator who set the value to harden client IP resolution but typo'd it would get weaker security than before, with no fail-closed signal. Model the setting as a tagged union (_HopCountUnset, _HopCountInvalid, _HopCount) so the three states are distinct: unset keeps the legacy path, a valid count drives hop-counting, and an invalid value fails closed (returns "") instead of reverting to the spoofable leftmost address. The caller matches on the union exhaustively. Add a parametrized regression test asserting get_mcp_client_ip returns "" for 0, -1, "abc", and 1.5 even with a spoofed internal leftmost entry, and update the resolver unit tests for the new return type. |
||
|
tin-berri
|
f426912ba1
|
fix(mcp): resolve toolset tools by the server's known prefix (#31254)
* fix(mcp): resolve toolset tools by the server's known prefix
Toolsets store {server_id, bare tool_name} and reconcile that against the
live prefixed tool name at list time. The reconciliation chopped the live
name at the first MCP_TOOL_PREFIX_SEPARATOR with no server context, so a
server whose prefix contains the separator (a hyphenated alias, or the
UUID server_id used as the prefix when a server has no alias) had its
tools silently dropped from /toolset/<name>/mcp while listing fine
everywhere else. Strip the exact known prefix for the tool's server_id
instead of guessing the boundary, on both the resolve and filter sides
Also render toolset tools as {server-prefix}-{tool} in the dashboard
picker result and chips; this is display only, the persisted record
stays {server_id, bare tool_name}
Resolves LIT-3419
* test(mcp): add focused unit tests for strip_known_server_prefix
Cover the LIT-3419 cases directly on the helper with real MCPServer
objects: clean prefix round-trip, hyphenated alias, UUID server_id
fallback, unprefixed passthrough, and the server=None legacy fallback
|
||
|
ryan-crabbe-berri
|
fa307fe9e5
|
fix(ui): render logos under a custom server_root_path (#31156)
The App Router migration moved pages to deeper path segments and the proxy can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local logo asset paths were emitted without the server root prefix, so they resolved off the origin root and 404'd. Route every local logo src through a single resolver that prefixes the live server root path and leaves external URLs untouched, fixing provider, guardrail, vector store, callback, MCP and audit-log logos at any route depth and root path. |
||
|
ishaan-berri
|
4efce809d0
|
feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134)
* feat(proxy): add logging_endpoints package init
* feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out
* feat(proxy): register callback_logs_router
* test(proxy): add logging_endpoints test package init
* test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling
* refactor(proxy): move callback-logs request/response models to litellm/types/proxy
* refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging
* test(proxy): update callback-logs tests for class-based replayer and separated types
* fix(proxy): cover /v1/callbacks/ in backend component allowlist
The new /v1/callbacks/logs route was dropped by both component
allowlists, failing test_gateway_plus_backend_covers_full_app. It's an
admin-only spend-logging route, so it belongs on the backend (control
plane) alongside the existing /callbacks family.
* refactor(proxy): use builtin dict/list generics in callback-logs endpoint
Switch Dict/List from typing to builtin dict/list to satisfy the ruff
strict-rule budget (UP006).
* refactor(proxy): use builtin dict/list generics in callback-logs types
UP006: builtin generics over typing.Dict/List.
* chore(ui): regenerate schema.d.ts for /v1/callbacks/logs
Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/
CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the
dashboard types in sync with the proxy OpenAPI spec.
* fix(proxy): force stream=False when replaying callback logs
A replayed StandardLoggingPayload is a terminal, fully-aggregated event —
the producer (e.g. the rust realtime gateway) already collected the whole
session before POSTing. Marking the rebuilt Logging object as streaming made
async_success_handler wait for a complete_streaming_response that never
arrives, so the spend log was never written. Realtime sessions now land in
LiteLLM_SpendLogs.
* feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs
integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait
(base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger
— the first concrete logger, owning a bounded channel + background worker that
batches and POSTs to the Python proxy's /v1/callbacks/logs.
* feat(litellm-rust): RealTimeStreaming per-session log collector
1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id
per event (never buffers frames); log_messages() builds one StandardLoggingPayload
on session close and fans out to the CustomLogger callbacks. request_id == the
OpenAI realtime session id (sess_…), with the gateway id as fallback.
* feat(litellm-rust): wire realtime logging into the splice (lock-free observe)
The collector is owned on the splice task and observed via a synchronous &mut
callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic
on the per-frame hot path. On session close the bridge flushes one payload.
AppState carries the registered loggers; main spawns the proxy logger.
* docs(litellm-rust): ai-gateway realtime logging architecture
* docs(litellm-rust): document request-log egress to the LiteLLM control plane
Add a 'Request logging' guide to the ai-gateway README: how to point the gateway
at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the
admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session
behavior.
* feat(litellm-rust): make log-egress tunables env-overridable
Channel capacity, batch size, and flush interval now read from
LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS,
falling back to the DEFAULT_* consts on missing/invalid/non-positive values.
Grouped behind an EgressTunables::from_env() read once at logger construction.
* docs(litellm-rust): document log-egress tuning env vars
* docs(litellm-rust): require constants in a crate-level constants.rs
Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings
go in src/constants.rs, not inline in feature modules; env-overridable tunables
keep their DEFAULT_* value there.
* refactor(litellm-rust): move ai-gateway constants into constants.rs
Per the new rule: the log-egress defaults (proxy base, ingest path, channel
capacity, batch size, flush interval) and the realtime provider default move to
crates/ai-gateway/src/constants.rs; modules import from it.
* ci: run logging_endpoints tests in the proxy-infra coverage shard
tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading
job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on
codecov/patch despite being ~98% covered locally. Add it to proxy-infra's
test-path so the test is exercised under --cov.
* fix(litellm-rust): hash the master key before logging — never send the raw credential
Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which
fans out to spend logs and every callback (Langfuse/Datadog) and could be
recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's
hash_token); the field is named *_hash and the proxy stores it verbatim when it
isn't sk-prefixed, so the DB value is identical with zero plaintext exposure.
* fix(litellm-rust): observe realtime logging on upstream events only
Greptile P1: observe ran on the client->upstream arm too, so an authenticated
client could send a fabricated response.done and inflate its own spend log.
session.created/response.done are server->client events; observe the upstream
arm only.
* feat(proxy): bound callback-logs batch + return per-record failures
Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000,
env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and
return per-record {index, error} failures so a caller (the rust gateway) can
distinguish a transient callback error from a structurally bad payload.
* chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field
* fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant
It doesn't need to be env-configurable (only the rust egress tunables are). As an
os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires
every env key to be documented in the (separate-repo) config_settings.md. Plain
constant → not scanned → code-quality + documentation checks pass.
* docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes
* docs(litellm-rust): tighten the README request-logging section
* docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback)
* docs(litellm-rust): drop em-dashes from the request-logging section
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
|
||
|
ryan-crabbe-berri
|
f2f6cacb19
|
feat(ui): track frontend lint counts in a committed snapshot (#31157)
* feat(ui): track frontend lint counts in a committed snapshot Persist the eslint budget-rule counts (no-explicit-any, complexity, max-depth) to eslint-metrics.json so the trend is queryable straight from git history and can later feed a dashboard. A CI drift check regenerated from the same lint report keeps the snapshot honest, so a PR that shifts a count has to run npm run lint:metrics and commit it * fix(ui): harden lint-metrics drift check and eslint failure handling Make the drift comparison symmetric over the union of committed and actual keys so a phantom rule left in eslint-metrics.json (for example after a rule is dropped from eslint-budgets.json) is caught instead of silently passing. Only swallow eslint's lint-errors exit code in the generator and rethrow anything else, so a fatal eslint failure surfaces its real output rather than a confusing ENOENT on the missing report |
||
|
ryan-crabbe-berri
|
8f4389246d
|
fix(ui): persist budget window deletion on virtual keys (#31107)
Deleting every budget window from a virtual key looked like it saved but reverted on reload, while editing a window persisted. The key edit form set budget_limits to undefined once the window list was emptied, and JSON.stringify drops undefined keys, so /key/update received no budget_limits field at all and model_dump(exclude_unset=True) skipped the existing clear-on-empty branch. Sending [] instead lets the backend store JSON null and clear the stored windows, matching how it already treats an explicit empty list Resolves LIT-3742 |
||
|
yucheng-berri
|
a8a1472428
|
fix(deps): bump osv-flagged dependencies to clear known CVEs (#31122)
Bumps the 12 packages osv-scanner flags on litellm_internal_staging, taking the scan from 24 known vulnerabilities to zero. vcrpy goes to 8.2.1 first so aiohttp can move to 3.14.1 (vcrpy <= 8.1.1 cannot import aiohttp 3.14), then the two aiohttp ignore entries are dropped from osv-scanner.toml. The langchain stack moves together since langchain 1.3.9 requires langgraph 1.2.x. Runtime deps cryptography (48.0.1), starlette (1.3.1), python-multipart (0.0.32), pydantic-settings (2.14.2) and pypdf (6.13.3) are bumped via relock, and the dashboard's js-yaml, ws and form-data overrides are bumped too. Also removes the paths filter on the OSV workflow so it runs on every PR rather than only when a lockfile changes, which is why it never showed up on recent code-only PRs |
||
|
yuneng-jiang
|
a5b75e8bab
|
fix(ui): keep team Organization optional for proxy admins in single-org setups (#30861)
The Create Team form auto-selected, disabled, and required the Organization field whenever exactly one organization existed, regardless of role. For a proxy admin the organization is optional, so single-org setups could not create a standalone team even though the field is presented as optional. Gate the single-org preselect, the disabled state, and the restrictive help text on the org-admin role so they apply only to org admins, who must scope a team to their organization. Proxy admins now keep an optional, clearable, empty organization field regardless of how many organizations exist, matching the multi-org behavior. The /team/new endpoint already accepts a null organization, so this was a UI-only restriction. |
||
|
mubashir1osmani
|
21cd1d1a4f
|
fix(router): isolate all per-deployment pricing overrides from sibling deployments (#31021)
* fix(router): isolate all per-deployment pricing overrides from sibling deployments CustomPricingLiteLLMParams is the authoritative set of per-deployment pricing fields, used to strip overrides from the shared backend-alias key so one deployment cannot pollute a sibling that shares the same backend model. It had drifted from ModelInfoBase: tiered and per-unit cost fields such as input_cost_per_token_above_272k_tokens, cache_read_input_token_cost_above_*, output_vector_size, ocr_cost_per_*, and the regional uplift multipliers were absent, so a deployment overriding any of them leaked the override into litellm.model_cost under the shared key and every sibling read the wrong rate via /model/info (LIT-3897). Add the missing fields so the denylist covers every ModelInfoBase pricing field, and guard against future drift with a test asserting the two stay in sync, plus a regression test that a tiered override stays isolated to its own deployment model_id key. * chore(ui): regenerate schema.d.ts for custom pricing fields |