* fix(router): isolate all per-deployment pricing overrides from sibling deployments
CustomPricingLiteLLMParams is the authoritative set of per-deployment pricing
fields, used to strip overrides from the shared backend-alias key so one
deployment cannot pollute a sibling that shares the same backend model. It had
drifted from ModelInfoBase: tiered and per-unit cost fields such as
input_cost_per_token_above_272k_tokens, cache_read_input_token_cost_above_*,
output_vector_size, ocr_cost_per_*, and the regional uplift multipliers were
absent, so a deployment overriding any of them leaked the override into
litellm.model_cost under the shared key and every sibling read the wrong rate
via /model/info (LIT-3897).
Add the missing fields so the denylist covers every ModelInfoBase pricing
field, and guard against future drift with a test asserting the two stay in
sync, plus a regression test that a tiered override stays isolated to its own
deployment model_id key.
* chore(ui): regenerate schema.d.ts for custom pricing fields
* feat(mcp): scaffold outbound_credentials package with typed Result
PR1 of the MCP v2 outbound-credential migration. Adds the
litellm/proxy/_experimental/mcp_server/outbound_credentials/ subpackage with a
hand-rolled Ok | Error Result union (pure stdlib + typing_extensions, no new
dependency) and its package surface. Nothing imports this on a live request path
yet, so production behavior is unchanged; later PRs add the typed config
vocabulary, the resolve_credentials dispatch, and the v1 graft.
* feat(mcp): add outbound_credentials typed vocabulary (#31049)
PR2 of the MCP v2 outbound-credential migration, stacked on the result.py
scaffolding. Adds types.py (the AuthConfig discriminated union over seven frozen
per-mode configs, CredError as an expression @tagged_union, Subject, ServerSpec,
and the parse_auth_spec_kind boundary parser) and httpx_auth.py (NoOpAuth,
StaticHeaderAuth). Pulls in expression>=5.6.0,<6.0 on the proxy extra for the
tagged union. Construction-time tests prove illegal mode/field combinations are
rejected. Nothing is wired onto a request path yet; the resolver lands next.
Realtime websocket sessions emit events outside the OpenAIRealtimeEvents
union (e.g. rate_limits.updated, response.function_call_arguments.delta,
surfaced when logged_real_time_event_types="*"). Building
LiteLLMRealtimeStreamLoggingObject revalidated every stored event against
the 16-member union, producing thousands of ValidationErrors per session
(12,670 for a ~281-event session). That synchronous work blocked the
asyncio event loop, degrading realtime time-to-first-audio and dial latency
and tripping readiness probes, and the raised error discarded the session
usage so no cost was tracked.
Type results as SkipValidation[OpenAIRealtimeStreamList] and serialize the
events verbatim, so already-formed event dicts are not revalidated. The
flood drops from 12,670 errors to 0 and the combined usage survives to the
cost calculator.
Resolves LIT-3919
Resolves LIT-3920
POST /team/new with any budget_limits returned 500 because
jsonify_team_object serialized members_with_roles but left budget_limits
as a raw Python list, which Prisma's Json column rejects. /team/update
and /key/generate worked only because each json.dumps the windows itself.
Serialize budget_limits in the shared helper, guarded by isinstance(list)
so the pre-serialized /team/update path is unaffected.
Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.
- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
async). Large agentic tool-use / thinking streams can make assembly re-raise
as APIError from inside the except-StopIteration handler, where the sibling
except does not catch it, so it escaped __next__/__anext__ and dropped the
request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
stream_chunk_builder returns None or raises, rebuild usage from the
message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
(it is read from there, not from kwargs), matching the gemini/cohere/openai
handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
operators don't have to reverse-map spend back to a key
* docs: add MCP server change guidelines
* Add outbound_credentials directory and update AGENTS.md
Updated AGENTS.md to include new outbound_credentials directory and its files, along with additional comments on existing files.
---------
Co-authored-by: tin-berri <tin@berri.ai>
The login flow stores a post-login return URL in the litellm_return_url
cookie (5 minute TTL). globalSetup snapshots cookies into the per-role
storageState that every spec reuses, so when the snapshot races ahead
of the app consuming that cookie, each test inheriting it gets
redirected to the stale URL (/ui/?login=success) mid-assertion the
first time it mounts a page. That one rogue navigation is behind the
recurring e2e failures whose call logs all show "navigated to
/ui/?login=success" while waiting for an element; which specs die
varies run to run with snapshot timing. Clear the cookie right before
saving the snapshot so no test starts with a pending redirect.
The request logs table column displayed "Key Name" while its accessor
(metadata.user_api_key_alias) and the corresponding filter both use the
"Key Alias" label; this aligns the column header with that naming.
* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools
GET /v1/mcp/tools returns the MCP tools available to the calling key, the same
data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of
which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so
virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"])
got a 403 listing tools one way but not the other.
Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no
management write counterpart, so it does not need the method-aware carve-out
used for server discovery.
* test(proxy): parametrize MCP inference route check over the full endpoint set
The Add Model provider dropdown is driven by provider_create_fields.json
(served at /public/providers/fields), and Bedrock Mantle had no entry, so it
could not be selected even though the backend provider, its models, and the UI
enum/logo mappings already existed.
Add a bedrock_mantle entry exposing the credential fields the provider actually
honors: an optional bearer api_key for BYOK, the AWS SigV4 chain, a region, and
an api_base override. Selecting it now populates the bedrock_mantle models from
the cost map via the existing getProviderModels filter.
Also resolve the provider logo when getProviderLogoAndName is given the enum key
(e.g. BedrockMantle) rather than the slug, which the dropdown passes; previously
only slugs that lowercase-matched their key (like bedrock) resolved a logo.
* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel
A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.
* refactor(ui): centralize no-mcp-servers sentinel in a shared constant
The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.
* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes
Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.
AWS auth parameters in the Bedrock and SageMaker path could be expanded against
the process environment when credentials were built. Config-sourced references
are already expanded at load time, so restrict expansion to that path: a
reference still present at request time is treated as caller-supplied input and
is left as-is, and the web-identity helper rejects environment-variable
references before resolving the token.
Also rework the ambient AWS_* fallback as a single pass that pairs each value
with its own env-var name, fixing a latent index misalignment that left
AWS_EXTERNAL_ID unresolved.
Adds regression tests covering the resolution behavior.
GET /model/info returned an empty list for a team key whose team only has team-scoped BYOK deployments, even though /v1/models and a master key both returned them. _get_caller_byok_team_scope resolved the caller's allowed teams only from user_api_key_dict.user_id and the bound user's team memberships. A team or service key has user_id=None, so the helper returned an empty set and _byok_row_outside_caller_teams then dropped every team BYOK row
This includes the key's own team_id in the allowed-team set across every non-admin branch, since a team key is authoritatively scoped to its team regardless of whether a bound user is resolvable or a formal member of that team
Completes the work in #30025, which aligned /v1/model/info with router deployments but missed the team-key case in the scope helper it introduced
* fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body
* test: cover bedrock key-tag litellm_metadata pre-seed in common_checks
Add a regression test asserting key-level tags on a bedrock passthrough
request land in litellm_metadata and never leak into the provider-facing
metadata field, which closes the codecov/patch gap on the auth_checks
pre-seed line. Also drop the now-stale comment that hardcoded
metadata["headers"]; the headers are written under whichever metadata
field _get_metadata_variable_name selects.
* refactor(auth): pre-seed litellm_metadata from LITELLM_METADATA_ROUTES
The auth-time pre-seed in common_checks hardcoded "bedrock", so any other
route later added to LITELLM_METADATA_ROUTES would reintroduce GH#30629 (key
tags leaking into the provider-facing metadata field) without a matching
update here. Key off the shared constant instead, and extend the regression
test to cover a non-bedrock metadata route so the route-agnostic behavior is
locked in.
* fix(auth): pre-seed litellm_metadata before header-tag merge
apply_client_tag_policy_pre_auth runs in user_api_key_auth.py before
common_checks, so it resolved get_metadata_variable_name_from_kwargs to
'metadata' (litellm_metadata was not yet present). common_checks then
pre-seeded litellm_metadata on LITELLM_METADATA_ROUTES, after which
apply_key_tags_pre_auth and _tag_max_budget_check both targeted
litellm_metadata, leaving header tags stranded in metadata and invisible
to per-tag budget enforcement on Bedrock and other matching routes.
Extract the pre-seed into LiteLLMProxyRequestSetup.pre_seed_litellm_metadata_for_route
and invoke it before apply_client_tag_policy_pre_auth so all tag merges
and the budget-check read agree on the same metadata key.
* test(auth): guard early litellm_metadata pre-seed wiring for header tags
Bugbot's autofix added a pre-seed of litellm_metadata in
_run_centralized_common_checks before apply_client_tag_policy_pre_auth, so
x-litellm-tags header tags land in litellm_metadata and stay visible to
_tag_max_budget_check on LITELLM_METADATA_ROUTES. Its test replayed that call
order in the test body, so removing the production call site still passed.
Add a wiring-level regression that drives the real _run_centralized_common_checks
and asserts header tags land in litellm_metadata (not metadata) for bedrock and
/v1/messages. Dropping the pre-seed call site now fails the test.
---------
Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and
default_user_id) instead of emails. /user/daily/activity passed
entity_metadata_field=None, so every user entity in the breakdown
carried empty metadata; the chart could only fall back to the user_id.
The frontend resolved labels from a separately paginated user list, so
any spender not on a loaded page showed as a UUID.
Resolve the email/alias for the user_ids actually on the page (mirroring
how api key metadata is already resolved) and attach it to the entity
metadata, so the chart labels each spender with their email and falls
back to the UUID only when no email is on file. get_daily_activity gains
an optional resolve_entity_metadata hook so the user endpoint can do this
page-scoped lookup without loading the whole user table.
Resolves LIT-3889
Map the SCIM enterprise extension block
(urn:ietf:params:scim:schemas:extension:enterprise:2.0:User) onto SCIMUser
so create and PUT persist employeeNumber, costCenter, organization,
division, department, and manager into LiteLLM_UserTable.metadata under
scim_enterprise, and round-trip them back out on read. This lets financial
reporting group spend by fields like cost center and department.
The enterprise block holds directory-only HR attributes, so it is kept out
of the generic user management responses (/user/info, /v2/user/info, and
/user/list), which non-proxy-admin callers such as team and org admins can
use to read other users. The data still lands in metadata for reporting and
still round-trips through the SCIM read endpoints, which build their response
from the user row directly.
Resolves LIT-3617
Adds an optional litellm_settings.scim_admin_group. When configured, the
global proxy role is recomputed from a user's resulting groups on every SCIM
write that can change membership: user create/PUT/PATCH and group
create/PUT/PATCH/DELETE, plus the existing-email upsert path. Membership in
the admin group grants PROXY_ADMIN and its absence demotes to the non-admin
default, enabling just-in-time elevation and automatic demotion without a
re-login. When the setting is unset the role is never touched, so current
behavior is preserved and a misconfigured IdP can never unexpectedly grant
admin.
* fix: validate proxy request body and nested fields
Ensure caller-supplied request fields cannot override server-side deployment
configuration, and apply request-body validation consistently to nested
structures. Adjusts router kwarg handling and client-side credential handling
for base-url overrides
* test: cover router strip ordering and advisor clientside credential gate
* fix: clear deployment credentials on client base-url override
When a request overrides api_base/base_url, recompute the deployment's
litellm_params (clearing the deployment's own api_key) and drop the cached
client built for the original endpoint, so the deployment credential is not
reused for the client-supplied endpoint. Adds regression tests that assert the
credentials actually forwarded to litellm.completion/acompletion.
* fix(proxy): require api_key alongside api_base override
A request that overrides api_base/base_url but supplies no api_key still
left the proxy carrying a server credential: once the override clears the
banned-param opt-in, the provider re-resolves a key from the environment
(api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in
main.py) and forwards it to the caller-controlled URL. Popping the
deployment api_key only changed which server key leaked.
Gate is_request_body_safe so a permitted api_base/base_url override must
also carry a non-empty caller api_key; reject otherwise. The env
resolution in main.py is left as the provider boundary.
* fix(proxy): extend request-body banlist with five additional credential and session targeting fields
Yuneng's review found five deployment-owned request-body params still missing
from the denylist and the router strip set. Each lets a caller reach the
operator's provider credentials or retarget the outbound request:
aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region
retarget the OCI request, litellm_credential_name selects any server-loaded
credential by name with no ownership check, and runtimeSessionId resumes a
Bedrock AgentCore runtime session (AWS does not enforce session-to-user
mapping, so this is a cross-tenant session-resume vector).
Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to
_DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and
SDK direct calls are unaffected: the banlist gates the request body only, and
the router strip drops caller kwargs, never deployment["litellm_params"].
* test: rename arbitrary canary values in security tests to neutral placeholders
* fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip
P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS
(litellm_embedding_config, extra_body) for the banned-param check but not for
the api_key co-presence check, so a base override smuggled into one of those
nested dicts cleared the client-side-credentials opt-in without a paired
api_key and let the provider re-resolve a server credential from the
environment. Run _check_base_override_has_api_key on each nested config dict
too, so the requirement applies wherever a base override is permitted.
P1-B: the deployment-owned credential strip in the Router runs unconditionally
on every _completion/_acompletion, which is security-correct but silently
drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a
single warning (key names only, never values) when the strip removes a
non-empty value, so the backwards-incompatible behavior is visible without
gating the strip on a context flag that does not exist.
* fix(proxy): apply api_key co-presence to tool-entry base override
is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and
tools[]); the previous commit extended the api_key co-presence rule to root
and nested config dicts but not to tool entries. With
allow_client_side_credentials enabled, a tool entry carrying api_base/base_url
and no paired api_key cleared the gate, letting a provider interceptor fall
back to a server-side credential for a caller-controlled URL. Add the same
_check_base_override_has_api_key call to each tool dict and its nested function
dict, mirroring the symmetry already applied to the nested config keys. The
rule is unchanged: api_key must live in the same dict as the base override it
accompanies.
* test(proxy/auth): require paired api_key under extra_body opt-in
* fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running
* fix(advisor): narrow proxy-import guard to ImportError-family
* fix(router): gate api_key clear on base override behind litellm.proxy_is_running
* test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture
* style: use built-in generics in PR-added type annotations
* revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate
* revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes
* style: black-format advisor orchestration test
The "day-by-day by user and model" usage export overcounted spend by
repeating each user-day total once per model. generateDailyWithModelsData
iterated day.breakdown.models crossed with the entity's own
api_key_breakdown and added the api key's full daily spend to every model,
leaving the per-model object (modelData) unused and never matching the api
key to the model. A user who called N models got N identical rows, so the
column summed to N times the real spend; one reported export totaled
$167,953 against a true $21,353 (7.87x).
Attribute each model only the spend of the keys that actually used it by
intersecting the entity's api keys with modelData.api_key_breakdown. This
mirrors the already-correct pattern in activity_metrics.tsx, so per-model
rows now sum back to the user-day total and models a user never called no
longer appear.
The existing tests passed only because their mocks omitted
models[*].api_key_breakdown and never asserted numeric values. The mocks
now match the real /user/daily/activity payload, and new tests assert that
per-model spend sums to the user-day total and that uncalled models are
omitted; both fail on the old code.
Resolves LIT-3866
is_otel_v2_enabled() constructed a pydantic-settings model (_OTelV2Flag) on every
call, which re-scans the process environment and costs ~28us. The flag is read
multiple times along the proxy request hot path (auth, logging-callback setup,
proxy_server), so the cost compounded into a measurable per-request CPU overhead
and a throughput regression visible from v1.87.3 onward.
The flag is a process-level setting that is fixed at startup, so resolve it once
with lru_cache. Caching it alone restores throughput to the pre-regression
baseline in load tests. Tests that toggle the env now call cache_clear().
The AI Hub MCP Hub listed each MCP server's upstream URL in a table column and in the server Details modal, on both the authenticated dashboard and the public hub. The unauthenticated GET /public/mcp_hub endpoint also returned the url field via MCPPublicServer, so the upstream address was readable by any client even with the column gone. These surfaces are for end users discovering available servers, so the gateway-internal endpoint should not be exposed there.
Drop the URL column from both MCP Hub tables and the URL field from both detail modals, and remove url from MCPPublicServer so /public/mcp_hub no longer serialises it; schema.d.ts is updated to match. The public hub MCPServerData interface no longer declares url since the response omits it. Admin surfaces that configure the endpoint (the MCP server management page, the submissions review tab, the make-public form) and the authenticated /v1/mcp/server endpoint are untouched.
publicMCPHubColumns is lifted to a module-level export so both hub column sets get a mutation-killing regression test, public_model_hub.test.tsx covers the details modal hiding the url, and test_public_endpoints.py asserts /public/mcp_hub never returns url even when the server has one.
Google moved the role field off the top-level Interaction response schema onto the per-turn Turn schema, so the OpenAPI compliance canary test_interaction_response_fields started failing on main with "Output field 'role' not in spec". role on Turn is already asserted by test_turn_schema, so dropping it from the Interaction output_fields list realigns the test with the live spec without losing coverage.
* feat(sandbox): code interpreter interceptor on the Responses API
Route OpenAI's code interpreter to a configured sandbox (e2b) instead of
OpenAI's container, with no client change. A client calls /v1/responses with
a code_interpreter tool; the interceptor converts it to a function tool so the
model emits the code, runs that code in the sandbox via the phase 1 primitive,
feeds the result back, and lets the agentic loop continue.
Reuses the existing agentic-loop hooks (no new hook methods). The anthropic
agentic caller _call_agentic_completion_hooks gains an api_surface argument and
a responses execute path (_execute_responses_agentic_plan re-calls aresponses);
the responses handler invokes it after transforming the response. Web search and
compression interceptors are untouched.
Adds an api_base passthrough to the sandbox SDK and a sandbox_tools registry the
proxy parses, so the interceptor resolves a named tool to provider/key/base.
v0 limitation: no file upload or download yet; stdout and inline results flow
back, attaching input files and downloading produced files do not.
* feat(sandbox): re-inject code_interpreter_call so the response matches OpenAI
The native OpenAI Responses code interpreter returns a code_interpreter_call
output item (id, type, status, code, container_id, outputs) alongside the
message. The interceptor now re-injects an equivalent item via
async_post_agentic_loop_response_hook so a client gets the same response shape
whether the code ran in OpenAI's container or the sandbox: build_plan records
the executed code and the container id per call, and the post hook inserts the
code_interpreter_call before the message in the final response output.
* feat(sandbox): support streaming for the code interpreter interceptor
A stream:true /v1/responses request with code_interpreter previously broke,
because the agentic loop only runs on the non-streaming responses path. The
interceptor now forces stream=False in the pre-call hook (so the loop runs in
the sandbox) and the responses handler wraps the completed response back into a
synthetic stream via MockResponsesAPIStreamingIterator, so the caller still gets
SSE. The follow-up call and nested wrapping are guarded by stripping the
converted-stream flag from the follow-up request and only wrapping at the
outermost call (agentic loop depth 0).
* fix(lint): use builtin generics in code interpreter interceptor to satisfy UP006 budget
* fix(code-interpreter): gate sandbox execution, delete sandboxes, harden registry
Gate the agentic loop on a server-set interception marker and re-check
provider scope so an authenticated caller cannot trigger sandbox code
execution by naming their own function tool litellm_code_execution; the
marker is stripped from client requests at the proxy boundary and only
set when the pre-call hook actually converts a native code_interpreter
tool. Delete the sandbox once the final response is assembled instead of
leaking it until its own timeout, and prune expired cache entries by
deleting their containers too. Resolve sandbox params once at create time
and reuse them for run and delete. Clear the sandbox-tool registry before
re-registering so stale tools do not survive a config reload.
* fix(lint): use PEP 604 X | None unions to satisfy UP045 budget
* test(code-interpreter): cover execution-error and unparseable-argument tool-call paths
* fix(code-interpreter): rewrite forced code_interpreter tool_choice to the function tool
* test(sandbox): cover sandbox-tool registry resolution, reload clearing, and secret lookup
* test(code-interpreter): cover dict-shaped responses and object-attribute tool-call detection
* fix(proxy): strip client-supplied _code_interpreter_interception_converted_stream
A client could inject the converted-stream marker to force the completed
response to be re-wrapped as a synthetic SSE stream it never requested.
Add it to the untrusted root control fields alongside the other agentic
loop markers so the proxy strips it at the request boundary.
* fix(code-interpreter): isolate sandboxes by server-minted key and clear registry on tool removal
Key the per-request sandbox cache on a server-minted random token instead
of the caller-controlled litellm_call_id (sourced from the x-litellm-call-id
header). Two concurrent requests that send a colliding call id can no longer
share a sandbox container and read each other's code or files. The token is
minted in the pre-call hook when interception activates, stripped from client
requests at the proxy boundary, and survives the server-driven followups so a
single request still reuses one sandbox across the agentic loop.
Register sandbox tools unconditionally with an empty-list fallback so a config
reload that removes sandbox_tools clears the previously registered credentials
instead of leaving them resolvable in the process.
* refactor(sandbox): swap the tool registry atomically on reload
Build the new registry and rebind it in one assignment instead of clearing
then repopulating in place, so a concurrent resolve_sandbox_tool can never
observe a transiently empty or half-populated registry during a config
reload. clear_sandbox_tools now delegates to register_sandbox_tools([]).
* fix(code-interpreter): cap caller loop limit and emit OpenAI-shaped outputs
Strip max_agentic_loops at the proxy request boundary so an authenticated
caller cannot raise the agentic-loop ceiling to drive many upstream model
calls and sandbox executions from a single request; the loop stays bounded
by the server default.
Populate the re-injected code_interpreter_call.outputs with an OpenAI-shaped
logs array ([{"type": "logs", "logs": stdout}], or [] when there is no
stdout) instead of None, so clients that iterate over outputs or validate the
response through the OpenAI SDK's Pydantic model do not break.
* feat: plugin architecture — toggle between AI Gateway and external plugins
Adds a generic plugin system so any external service can register with
litellm and appear as a mode in the UI alongside the AI Gateway.
Backend (litellm/proxy/plugin_routes.py — new):
- GET /api/plugins: returns registered plugins from config; returns
plugin_key only to authenticated requests
- ANY /plugin-proxy/{name}/{path}: reverse proxies API calls to plugin
Config:
general_settings:
plugins:
- name: my-plugin
display_name: My Plugin
url: https://my-plugin.example.com
plugin_key: sk-... # plugin auth key, passed to iframe
UI:
- PluginModeContext.tsx: fetches /api/plugins, persists mode to localStorage
- leftnav.tsx: mode switcher dropdown at top of sidebar; plugin mode shows
plugin-specific nav items
- layout.tsx: renders iframe to plugin URL in plugin mode; passes plugin_key
as ?token= for auto sign-in
Plugin contract: expose GET /api/plugin-manifest returning
{ name, display_name, nav_items[], capabilities[] }. No litellm changes
needed to add new plugins — config only.
Reference implementation: LiteLLM-Labs/litellm-agent-control-plane
* feat: add Plugins tab to Admin Settings UI
Allows admins to add/edit/delete plugin registrations directly in the
litellm UI under Admin Settings > Plugins, instead of editing config.yaml.
Uses existing /config/field/update API to persist to general_settings.plugins.
Each plugin entry has: name (identifier), display_name, url, plugin_key.
* fix(ci): black, prettier, eslint, async-client violations
- Black: format plugin_routes.py and proxy_server.py
- Prettier: format PluginModeContext.tsx and PluginSettings.tsx
- ESLint: replace raw fetch() with createApiClient in PluginModeContext
- ESLint: use lazy useState initializer to read localStorage instead of
calling setModeState inside useEffect (react-hooks/set-state-in-effect)
- code-quality: replace httpx.AsyncClient per-request with
get_async_httpx_client() shared client (avoids +500ms overhead)
* fix(ci): schema.d.ts regen, Black proxy_server.py, ApiClientConfig fix
- Regenerate schema.d.ts for new /api/plugins routes
- Re-run Black 26.3.1 on proxy_server.py (matches CI version)
- Fix PluginModeContext: createApiClient requires getBaseUrl field
* fix: security hardening + CI fixes
Security (Greptile 1/5 → addressing all 3 findings):
- plugin_routes.py: add Depends(user_api_key_auth) to both /api/plugins
and /plugin-proxy/{name}/{path} — was an unauthenticated open relay
- plugin_routes.py: /api/plugins now returns plugin_key only to callers
with a valid litellm token (enforced by user_api_key_auth), not just
any header presence
- layout.tsx: replace ?token= URL param with postMessage(targetOrigin)
— token no longer exposed in browser history / logs / Referer headers
CI:
- backend/routes/allowlist.py: add /api/plugins and /plugin-proxy/ to
fix test_gateway_plus_backend_covers_full_app
- schema.d.ts: regenerated with enterprise routes included
- Black + Prettier formatting
* fix: regenerate schema.d.ts with enterprise routes included
Install litellm-enterprise workspace member before gen:api so audit and
other enterprise routes appear in the generated types, matching what CI
produces with uv sync --extra proxy.
* fix: exclude plugin routes from OpenAPI schema, restore upstream schema.d.ts
Both /api/plugins and /plugin-proxy/ are internal infrastructure routes,
not part of the public litellm API surface. Marking include_in_schema=False
prevents Python-version-dependent schema diffs from breaking the schema
sync check across different environments.
* fix: schema.d.ts - passing schema base + exact plugin route types from openapi-typescript
Use the CI-correct schema from a recently passing branch as base, then
inject plugin route entries (paths + operations) generated by
openapi-typescript from the plugin routes' OpenAPI spec. This avoids
Python-version-dependent formatting differences that made local gen:api
produce incorrect output.
* fix: schema.d.ts - insert plugin ops at correct route registration position
Plugin operations belong after delete_memory_v1_memory__key__delete
(memory_router is included immediately before plugin_router in proxy_server.py),
not after list_organization which is alphabetically but not registration-order.
* fix: schema.d.ts - correct op positions from hunk analysis
list_plugins_api_plugins_get goes after event_logging_batch op (hunk 1: line 33583).
plugin_proxy ops go after create_policy_policies_post (hunk 2: line 44634).
Previous location after delete_memory_v1_memory__key__delete was wrong.
* fix: schema.d.ts - proxy ops go before create_policy (after otel_spans)
* fix(security): restrict plugin_key to proxy_admin role only
Veria finding: plugin_key was returned to any authenticated caller.
Now only proxy_admin users receive plugin credentials in /api/plugins
response — regular internal users see plugin name/url but not the key.
* fix: update schema.d.ts docstring for list_plugins
* fix: clear plugin registry on config reload (Greptile medium)
register_plugins_from_config now replaces the registry instead of
merging, so plugins removed from config are unreachable immediately
without requiring a process restart.
* fix(security): encrypted token exchange for plugin iframe — no raw litellm credential exposure
The dashboard was sending the user's litellm bearer token to the plugin
iframe via postMessage, allowing a compromised plugin to act as that user.
Fix:
- GET /api/plugins/auth-token: proxy encrypts caller token with Fernet
keyed from LITELLM_SALT_KEY, returns ciphertext only
- UI postMessages the ciphertext (not raw token) to the iframe
- Plugin decrypts server-side with same LITELLM_SALT_KEY via POST /api/plugin-auth
- Raw litellm credential never leaves the proxy in plaintext
Additional hardening already in place:
- /plugin-proxy/* strips Authorization header, injects plugin_key instead
- plugin_key only returned to proxy_admin role via /api/plugins
- Plugin registry cleared (not merged) on config reload
Adds docs/plugin_architecture.md with plugin integration guide.
* fix(code-quality): use get_async_httpx_client in plugin_proxy
* fix: add /api/plugins/auth-token to schema.d.ts
* fix: use apiClient for auth-token fetch, copy correct layout.tsx and PluginModeContext
- Replace raw fetch() with createApiClient (fixes no-restricted-syntax ESLint rule)
- Copy correct layout.tsx with encrypted token + postMessage approach
- Copy correct PluginModeContext.tsx with accessToken prop injection
- Update schema.d.ts with auth-token path and operation entries
* fix: add plugin_auth_token operation to schema.d.ts
* fix(security): strip cookie/set-cookie + fix compressed response headers
Veria High: cookie header was forwarded to plugin backends allowing
capture of litellm JWT session cookies. Strip cookie on requests.
Strip set-cookie from responses so plugins cannot overwrite litellm
session cookies.
Greptile P1: httpx decompresses responses but resp.headers still
contained Content-Encoding/Transfer-Encoding/Content-Length from the
wire. Forwarding these caused double-decompression and length errors.
Now filtered via _RESPONSE_STRIP before returning to the browser.
* fix: update plugin_key help text — no more ?token= reference
* fix(security): disable follow_redirects to prevent SSRF
follow_redirects=True allowed a plugin backend to return a 3xx to an
internal URL, causing the proxy to fetch that internal service and relay
the response. Disabled: clients handle their own redirects.
* fix: forward user identity headers to plugin to address confused deputy
Plugins receive X-LiteLLM-User-Id and X-LiteLLM-User-Role so they can
enforce their own per-user access control before acting on requests that
arrive with the shared plugin_key credential.
* fix(security): restrict /plugin-proxy/* to proxy_admin role
Closes the confused deputy gap: regular users could invoke any plugin
endpoint using the shared plugin_key as a bearer credential. Now only
proxy_admin callers can use the plugin proxy route.
Plugin UIs communicate with the plugin service directly via the iframe
(using the encrypted token exchange); this proxy route is for
administrative/server-to-server access only.
* fix: update schema.d.ts for admin-only proxy route docstring
* fix(bug): use PassThroughEndpoint instead of None for get_async_httpx_client
get_async_httpx_client(llm_provider=None) raises TypeError — the function
concatenates the provider string and None is not a str. Use
httpxSpecialProvider.PassThroughEndpoint, the enum value used by other
internal proxy pass-through routes.
* fix(security): add 30s TTL to encrypted plugin auth tokens
Veria medium: encrypted tokens had no expiry, allowing indefinite replay.
Fernet embeds a timestamp; decrypt_token now passes ttl=30 so tokens
older than 30 seconds are rejected even with a valid HMAC.
Plugin's /api/plugin-auth must call litellm within 30s of the iframe
receiving the postMessage — normal browser behavior, tight enough to
close the replay window.
* feat(ui): topnav plugin switcher, embed plugins at their root
Builds on the plugin architecture already on this branch (encrypted-token
postMessage handshake, /api/plugins, PluginSettings) and removes the parts of the
embed that assumed a specific plugin's shape.
The mode switcher moves out of the sidebar into the topnav and lists AI Gateway
plus each registered plugin by its display_name. Selecting a plugin hides
litellm's sidebar entirely and renders the plugin full-bleed at its root url; the
plugin draws its own navigation inside the iframe. This drops the hardcoded
"Agent Control Plane" label and the hardcoded Sessions/Agents/Routines/... nav
groups (agentControlPlaneMenuGroups / acpPagePaths) that only matched the agent
platform and 404'd for a plugin that serves only / (e.g. the chat UI). The
encrypted-token postMessage flow is unchanged.
Note: embedding at root means a plugin must route internally from /; plugins that
previously relied on the /sessions entrypoint should redirect from their root.
* fix(security): audience-scoped identity claim replaces litellm token
Veria: shared LITELLM_SALT_KEY with plugins + encrypting user bearer token
created delegation/impersonation risk.
Architecture change:
- /api/plugins/auth-token now issues a plugin-scoped identity CLAIM
{user_id, user_role, plugin, exp} encrypted with HMAC(LITELLM_SALT_KEY, plugin_name)
- Each plugin holds only its own HMAC-derived key; cannot forge claims for
other plugins or recover LITELLM_SALT_KEY
- Claim contains NO litellm bearer token — compromised plugin learns caller
identity only, cannot act as that user against the proxy
- 30s TTL enforced in both Fernet header and explicit exp field
- LAP /api/plugin-auth verifies claim, returns its own master key to browser
(LAP key never exposed without valid claim)
* fix(plugins): allow registering plugins from the admin UI
Adding a plugin in the UI POSTs general_settings.plugins to /config/field/update,
which rejected it with "Invalid field=plugins passed in." because `plugins` was
not a field on ConfigGeneralSettings. Add a typed PluginConfig model and a
`plugins` field so the update validates and persists.
The in-memory plugin registry only refreshed at startup, so a plugin added via
the UI did not appear in /api/plugins (the view switcher) until a restart. Refresh
the registry from the new general_settings whenever the plugins field is updated.
While here, type the registry as dict[str, PluginConfig] instead of raw dicts so
list_plugins and plugin_proxy access typed attributes.
Fix the Plugin Key field copy: it is optional and only used to authenticate
litellm's server-side reverse proxy to a plugin's own backend
(/plugin-proxy/<name>/*). It is not involved in iframe auth, which forwards the
user's litellm token. Plugins that use the forwarded token leave it blank.
* fix: regenerate schema.d.ts with PluginConfig type and updated auth-token endpoint
* fix: use CI-compatible schema base for plugin entries
* fix(plugins): load DB-persisted plugins on startup
Plugins added through the admin UI are saved to DB general_settings, but the
registry only initialised from the YAML config at boot, so UI-added plugins
disappeared from the view switcher after a restart (the Plugins table still
listed them since it reads the DB directly). Refresh the registry from the DB
general_settings when it is merged in at startup.
* fix: add PluginConfig schema, plugins field, fix list_plugins return type
* fix: correct PluginConfig and plugins field positions in schema
* fix: correct plugins field position in schema (after pass_through_endpoints)
* fix: update PluginConfig.plugin_key description to match _types.py source
* fix: move plugins field after pass_through_request_timeout (correct alphabetical position)
* fix: redact plugin_key in config/field/info response
Veria medium: proxy_admin_viewer could read plugin_key via
GET /config/field/info?field_name=plugins. Now plugin_key is
replaced with *** in the response regardless of caller role.
The credential is only usable server-side.
* fix(security): correct plugin docs salt-key guidance, drop iframe clipboard-read
Address the two open Veria findings on the plugin architecture.
The plugin docs told external services to decrypt the iframe auth payload
with the proxy's LITELLM_SALT_KEY directly. That is both insecure and wrong:
the running code derives a per-plugin key as HMAC-SHA256(LITELLM_SALT_KEY,
plugin_name) and ships only a short-lived identity claim with no litellm
bearer token. Sharing the master salt would let a compromised plugin decrypt
any litellm secret recovered from a dump or backup. Rewrite the doc to match
the implementation: the proxy computes the per-plugin key once and provisions
it as a dedicated secret, the plugin validates the claim's audience and 30s
TTL, and LITELLM_SALT_KEY never leaves the proxy. Also refresh the now-stale
module and UI comments that still described the old shared-key token flow.
Drop clipboard-read from the plugin iframe's allow attribute so an untrusted
plugin can no longer read the user's clipboard; clipboard-write is retained.
* fix(ci): modernize PluginConfig typing, refresh budget baselines via merge
* fix(plugins): close iframe auth race and empty-plugins mode fallback
Address the two open Greptile behavioral findings.
The iframe auth handshake only posted the encrypted claim on the iframe's
`load` event. When the auth-token fetch resolved after the iframe had already
loaded, that listener never fired again and the plugin never received the
claim. Send the claim immediately as well as on subsequent loads so both
orderings are covered.
The plugin mode fallback guarded on a non-empty plugins list, so removing all
plugins left a user stranded on a stale mode with a blank iframe instead of
returning to the AI Gateway. Track a loaded flag and fall back to ai-gateway
once plugins have loaded whenever the stored mode is no longer registered,
including the empty-list case.
Add a PluginModeContext regression test covering the empty-list fallback and
the still-registered path.
* chore: re-trigger CI (GH Actions missed the prior head; re-run flaky live-API suites)
* fix(plugins): scope iframe auth claim to the active plugin
The iframe auth-token fetch omitted plugin_name, so the proxy always issued a
claim encrypted under the default plugin's per-plugin key. For any other active
plugin the iframe received a claim it could not decrypt and sign-in silently
broke, and because the cached claim was posted to whichever plugin was mounted,
a compromised iframe could replay the default plugin's claim. The active
plugin's name was also missing from the fetch effect's dependencies, so
switching plugins never refreshed the claim.
Request the claim with the active plugin's name, re-fetch when the active
plugin changes, and only deliver a claim while it still matches the mounted
plugin so one plugin's claim is never replayed to another.
* fix(plugins): never overwrite a stored plugin_key with its redaction placeholder
/config/field/info redacts every plugin_key to "***", so an admin editing a
plugin in the settings UI posted that placeholder straight back and the update
handler persisted "***" as the real credential, permanently destroying the key.
Preserve the stored credential on update: a blank or redacted plugin_key now
sources the existing key from the saved config, only a real value replaces it,
and a placeholder with no stored key is dropped rather than written. The edit
modal also starts the key field blank so an untouched save keeps the current
key, with the field labelled accordingly.
* fix(security): sandbox proxied plugin responses on the dashboard origin
The /plugin-proxy reverse proxy returned the plugin's body and content-type on
the litellm dashboard origin, so a compromised plugin could serve an HTML/JS
document that a proxy_admin navigates to and have it execute with the admin's
session against same-origin management APIs.
Force every proxied response inert: set Content-Security-Policy: sandbox (opaque
origin, scripts disabled) and X-Content-Type-Options: nosniff, applied after the
plugin's own headers so they cannot be overridden. The header construction moves
to a pure helper with a unit test covering the sandbox enforcement and the
existing wire/cookie header stripping.
* fix(plugins): recover to ai-gateway when the plugins fetch fails
The loaded flag was only set on a successful /api/plugins response, so when the
fetch failed a user with a plugin mode stored in localStorage stayed on the
blank plugin placeholder with no switcher to escape. Mark loaded in a finally
so the stored mode still falls back to ai-gateway on failure, and add a
regression test for the failed-fetch path.
* fix(security): never return plugin_key from /api/plugins
The plugin list endpoint returned the plaintext plugin_key to proxy_admin
callers, and the dashboard fetches /api/plugins on every load into React state,
so the credential was exposed to DevTools, memory snapshots, and any same-origin
script. The browser never uses the key; the proxy injects it server-side from
the registry and admin key management runs through the redacted
/config/field/info path. Drop plugin_key from the response for every caller and
update the regression test to assert it is never returned.
* chore(ui): regenerate schema.d.ts for updated list_plugins docstring
* fix(security): strip every litellm auth header before forwarding to plugins
The plugin reverse proxy only removed Authorization and x-api-key, but
user_api_key_auth also authenticates a caller via API-Key, x-goog-api-key,
Ocp-Apim-Subscription-Key, x-litellm-api-key, and any configured custom key
header. A malicious plugin could lure a proxy_admin into calling
/plugin-proxy/... with the litellm key in one of those headers; the request
authenticated locally and then forwarded the same key to the plugin, letting it
impersonate the admin.
Add a canonical SpecialHeaders.litellm_credential_header_names() that the auth
header enum is the single source for, and strip that whole set plus the live
general_settings.litellm_key_header_name from every forwarded request. New auth
headers added to SpecialHeaders are now stripped automatically. Regression tests
cover each credential header, the custom configured header, and the canonical
list's contents.
* feat(fireworks_ai): sync chat completions endpoint with full API surface
Add 23 missing request parameters to get_supported_openai_params():
seed, top_logprobs, min_p, typical_p, repetition_penalty,
mirostat_target, mirostat_lr, logit_bias, echo, echo_last, ignore_eos,
prompt_cache_key, prompt_cache_isolation_key, raw_output,
perf_metrics_in_response, return_token_ids, safe_tokenization,
service_tier, metadata, speculation, prediction, stream_options,
sampling_mask. Also add reasoning_history gated on supports_reasoning.
Fix prompt_truncate_length to prompt_truncate_len to match the actual
API parameter name. The old name was never in
DEFAULT_CHAT_COMPLETION_PARAM_VALUES, so it always went to extra_body
and was rejected by Fireworks; it never actually worked.
Normalize reasoning_effort boolean values to strings: True becomes
"medium", False becomes "none". The Fireworks OpenAPI schema
documents these as accepted types, but the server rejects non-string
values with HTTP 400 in practice. Integers pass through as-is since the
server is expected to validate them.
Auto-inject stream_options.include_usage=true when stream=true and the
user has not explicitly set stream_options. Without this, Fireworks
returns null usage in all streaming chunks, which is inconsistent with
the non-streaming behavior where usage is always present. If the user
explicitly sets include_usage=false, it is preserved.
Capture Fireworks-specific response fields in transform_response():
perf_metrics, prompt_token_ids, raw_output, and token_ids are now
extracted from the response and stored in response._hidden_params
(fireworks_perf_metrics, fireworks_prompt_token_ids,
fireworks_raw_outputs, fireworks_token_ids) so they are accessible to
logging, the proxy, and downstream consumers when the corresponding
request parameters are enabled.
Remove deprecated document inlining logic. Document inlining was
deprecated on 2025-06-30
(https://docs.fireworks.ai/updates/changelog#-document-inlining-deprecation).
This removes _add_transform_inline_image_block(), the file-to-image_url
migration in _transform_messages_helper(), and the
disable_add_transform_inline_image_block lookup. Current models that
support image input do so natively as VLMs. cache_control,
provider_specific_fields, and thinking_blocks stripping is retained.
Update get_provider_info() to look up supports_vision and
supports_pdf_input from the model cost map instead of hardcoding both
to True (which was based on the now-deprecated document inlining).
supports_prompt_caching remains True.
API docs: https://docs.fireworks.ai/api-reference/post-chatcompletions
Reasoning guide: https://docs.fireworks.ai/guides/reasoning
Prompt caching: https://docs.fireworks.ai/guides/prompt-caching
* fix fireworks chat api surface gaps
* Scope Fireworks thinking param to reasoning models
* style: fix black formatting
* fix(test): update minimax-m3 expected_vision to True
* test: cover non-dict content branch in transform_messages_helper
* fix(fireworks_ai): remove metadata from supported params to prevent internal metadata disclosure
* test(fireworks_ai): replace stale document-inlining capability test
The CircleCI-only litellm_utils_tests suite still asserted the old
behavior where document inlining made every Fireworks model report
supports_pdf_input and supports_vision as True. That premise was removed
in this change, so the test now reflects cost-map-driven capabilities:
unmapped models no longer advertise vision/PDF support while mapped VLMs
like minimax-m3 still do.
* test(fireworks_ai): add end-to-end regression for native OpenAI params
The existing coverage for the newly supported OpenAI-native params asserted
list membership in get_supported_openai_params or called map_openai_params
with a hand-built dict, both of which bypass the get_optional_params gate
(DEFAULT_CHAT_COMPLETION_PARAM_VALUES). That gate is what previously raised
UnsupportedParamsError for seed, top_logprobs, logit_bias, prompt_cache_key,
service_tier and prediction when drop_params=False. Assert the full path so a
revert of the supported-params additions fails the test instead of passing a
shallow membership check.
* test(fireworks_ai): fix test isolation in vision/inlining tests
Use monkeypatch in test_fireworks_ai_vision_capability_from_cost_map so the
LITELLM_LOCAL_MODEL_COST_MAP env var and litellm.model_cost are restored after
the test instead of leaking global state into the rest of the process.
Switch the document-inlining integration tests off deepseek-v3p1, whose
supports_vision is null in the cost map, onto minimax-m3 which is explicitly
supports_vision:true. The pass-through assertions no longer depend on a model
incidentally not being marked non-vision.
* fix(fireworks_ai): gate image rejection on exact vision capability
The image_url rejection read supports_vision via _get_model_cost_capability,
which falls back to hyphen-boundary substring matching when no exact cost-map
entry exists. A custom or fine-tuned model id that merely contains a known
non-vision model's short name (e.g. an id ending in -glm-5p2) inherited that
entry's supports_vision:false and hard-failed valid image_url blocks on a
vision-capable deployment.
Split the exact candidate-key lookup into _get_model_cost_capability_exact and
use it for the hard rejection so a fuzzy match can never block images; the
substring fallback stays a soft signal for capability reporting. Also rewrites
the fallback as a comprehension + max instead of an accumulating loop.
* feat(fireworks_ai): surface response fields on streaming responses
The Fireworks-specific response fields (perf_metrics, prompt_token_ids,
per-choice raw_output and token_ids) were only captured into _hidden_params in
transform_response, which runs for non-streaming completions; streaming chat
went through the default OpenAI chunk handler and dropped them.
Add a FireworksAIChatCompletionStreamingHandler that the provider now returns
from get_model_response_iterator. It reuses one extraction helper with
transform_response and attaches the fields to each streamed chunk's
provider_specific_fields, which is the channel litellm preserves when it
rebuilds streamed chunks (per-chunk _hidden_params is not carried through).
Per-choice token_ids/raw_output ride the content chunks; response-level
perf_metrics/prompt_token_ids ride the final usage chunk. Covered by an
end-to-end streaming test through litellm.completion(stream=True).
---------
Co-authored-by: Ahmad Shahzad <ahmad@shahzad.dev>
Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com>
Introduce a single, typed caller identity that is resolved once at the auth
boundary and read by reference downstream, instead of being re-derived from a
50-field key object or rebuilt from request metadata.
What this adds (litellm/proxy/auth/resolvers/), organized by responsibility:
- Principal: a small, frozen, identity-only value type (user / organization /
teams / project / end-user / roles / scopes / network), with its sub-models
and the role mapping. No budget or policy state; those stay on the key object.
- DbIdentityStore: the auth flow's resolver, owning both halves of resolving a
caller. resolve_key does the one combined_view lookup (cache, then DB via the
shared lower-level helpers, then write-back) and returns the key object, which
still flows for budget / rate-limit / policy unchanged. principal_from_key
projects the identity slice of that key object into a Principal, issuing no
lookup. user_api_key_auth resolves every key through the store rather than
calling get_key_object directly; auth_checks.get_key_object stays as the legacy
entrypoint for its other callers until they migrate.
- network: the X-Forwarded-For / trusted-proxy CIDR primitives live here in one
place. trusted_proxy_utils now imports them rather than keeping a second copy.
At the seam, user_api_key_auth projects one per-request Principal off the
resolved key object and stamps the request network context onto it once
(X-Forwarded-For is trusted only when trusted_proxy_ranges is configured). It is
attached to request.state.principal for the downstream consumers later phases
add. The projection is additive and defensive: a failure never rejects an
already-authenticated request, and a missing principal must be treated as deny
by any future reader. The Principal is always identifiable (credential_ref and a
stable subject off the token), never anonymous.
This is additive and changes no behavior today; it is the identity foundation
the spend-attribution and authorization phases build on.
The migrated /ui/api-keys route gates rendering on useAuthorized() but read userID from the AuthContext (useAuth), which hydrates asynchronously. On a hard refresh or deep link the route could render UserDashboard before AuthContext had populated userID, so UserDashboard hit its `userID == null` guard and showed "User ID is not set". The legacy index page avoided this by gating on AuthContext's own authLoading; the migration switched the gate to useAuthorized without aligning the identity source.
Read identity from useAuthorized (a synchronous cookie decode) so userID is populated whenever the route is authorized. useAuth is kept only for the backfill setters UserDashboard still expects, until the planned AuthContext consolidation removes them.
Refs LIT-3687
* feat(sandbox): add e2b code execution primitive
Add a provider-agnostic code execution primitive that runs model-generated
code in an isolated sandbox and returns the output, with e2b as the first
backend over raw httpx (no SDK dependency).
Public API: litellm.acode_interpreter_tool (ephemeral create -> run -> delete)
plus the low-level lifecycle litellm.acreate_sandbox / arun_code /
adelete_sandbox. Each is @client-decorated so operations are logged like
litellm.asearch. Backends implement BaseSandboxConfig; resolved via
ProviderConfigManager.get_provider_sandbox_config.
* fix(sandbox): address review feedback and CI gates
- document e2b provider in provider_endpoints_support.json and add a sandbox endpoint definition
- regenerate dashboard CallTypes after the sandbox call-type additions
- guard explicit timeout=0 instead of coercing it to the default
- require a ContainerHandle access token before running code; reject bare-id runs
- return False on a 404 delete now that the shared http handler raises for status
- skip non-JSON NDJSON lines and cap streamed output to bound memory
- move the real-network integration tests out of tests/test_litellm into tests/integration/sandbox
* fix(sandbox): satisfy strict ruff gate and scope star-exports
- modernize annotations in the new sandbox modules to PEP 585/604 (list/dict,
X | None) and drop the now-unnecessary quoted forward refs so the strict-rule
budget delta for UP006/UP037/UP045 returns to zero
- add __all__ to litellm/sandbox/main.py so 'import *' only re-exports the four
public entrypoints instead of leaking module-level imports
* fix(sandbox): drop quotes on sandbox config return annotation
utils.py uses 'from __future__ import annotations', so the quoted forward ref
tripped UP037; the unquoted union is lazily evaluated and keeps the strict-rule
delta at zero
* chore(sandbox): re-trigger automated review after addressing feedback
* test: point router/completion/triton tests at the local fake OpenAI endpoint
The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app)
takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted
proxy configs onto a job-local fake server but left these in-Python api_base
literals pointing at the dead host, so litellm_router_testing, local_testing_part1,
local_testing_part2 and llm_translation_testing still fail with a 404
"Application not found" when Railway is down
Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190)
through a shared helper, auto-start the canned server from the local_testing and
llm_translation conftests when nothing is already serving, and extend the server
with a Triton embeddings route and a slow-endpoint delay so the triton and
latency-timeout tests run fully offline. The deliberately broken fallback URL is
left as-is so fallback handling still has a failing upstream
* fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI
* fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target
* fix(tests): keep fake OpenAI mock alive across xdist workers
ensure_fake_openai_endpoint registered atexit on the worker that spawned
the subprocess, so under -n 4 the first worker to drain its queue would
terminate the shared mock while siblings were still hitting it. Detach
the child via start_new_session and drop the per-worker teardown; reuse
on /health handles re-runs and CI containers clean up themselves
* fix(watsonx): wrap string embedding input in array for WatsonX API
WatsonX text/embeddings expects inputs as []string; OpenAI clients often send a single string.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(watsonx): format watsonx embed transformation for black
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(watsonx): avoid UP006 in embed transformation strict lint gate
Use list[str] and branch-based input normalization instead of List and cast
so the watsonx embedding change does not add strict ruff UP006 violations.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add CI-parity mode and truncation-proof summary to strict ruff gate
* refactor: tolerant worktree cleanup and concrete GateInputs types
* fix: clean up temp dir when git worktree add fails
* fix: align lint-gate with CI by dropping unused --ci-parity path
The lint-gate Makefile target invoked ruff_strict_gate.py with --ci-parity,
which counted violations on a throwaway merge of base into HEAD against base
counts at the base tip. CI in test-linting.yml runs the same script without
--ci-parity on a PR-head checkout, taking the gather_fast path that counts on
the live tree against base counts at the merge-base. A local pass could
therefore disagree with CI.
Drop --ci-parity from the Makefile and remove the now-unused gather_ci_parity
branch and flag so there is one code path that both local and CI exercise.
The docstring claim that CI runs against the synthetic merge ref was also
wrong; the workflow checks out github.event.pull_request.head.sha.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The deprecated OldTeams component takes only accessToken, userID, userRole
and premiumUser; it ignores the teams prop these tests passed and instead
populates its table from the mocked teamListCall. The delete-warning block
never set teamListCall, and vi.clearAllMocks clears call history but not
implementations, so the table rendered the "Legacy Team" (keys.length 2)
left behind by the previous block's last test. Both delete tests therefore
ran against that leaked team: the keys-present case passed only because the
leaked count happened to be 2, and the no-keys case rendered the same warning
it asserted should be absent, so it failed.
Seed the team through the channel the component actually reads (teamListCall)
and drop the props it never consumes, so each test renders exactly the team
it declares. The keys-present case now uses a distinctive count so it can no
longer pass on a coincidental leak
Adds a "valkey-semantic" cache type so semantic prompt caching can run
against Valkey clusters (for example AWS ElastiCache for Valkey) using the
valkey-search module.
The existing "redis-semantic" backend cannot drive valkey-search. RedisVL
gates the connection on a RediSearch module version that valkey-search does
not report, and its SemanticCache index declares the prompt as a TEXT field,
which valkey-search does not implement. ValkeySemanticCache therefore talks to
valkey-search directly over redis-py: it builds a vector index from the field
types valkey-search supports (TAG for caller scope, VECTOR for the prompt
embedding) and runs KNN queries for retrieval. Prompt extraction, embedding
generation, and cached-response parsing are reused from RedisSemanticCache
since those are backend agnostic. The redis dependency is imported lazily in
the cache dispatch so importing litellm without redis installed still works.
It also fixes semantic-cache scope keys so similarity matching works across
reworded prompts. get_cache_key() hashed messages / prompt / input into the
litellm_cache_key that every semantic backend filters its KNN search on, so a
paraphrase landed in a different bucket and never matched, even far above the
similarity threshold. For semantic cache types the prompt-bearing params are
now excluded from the scope key and the server-set tenant identity
(user_api_key, team, org) is appended instead, restoring embedding matching
within a tenant while keeping cache entries scoped to the authenticated
key / team / org. The three semantic backends share this key, so the same
change fixes redis-semantic and qdrant-semantic.
Connections resolve from VALKEY_HOST / VALKEY_PORT / VALKEY_PASSWORD, falling
back to REDIS_* for drop-in compatibility, and passwordless clusters (IAM or
no-auth) are supported.
Resolves#29121Fixes#29086
The delete-team confirmation modal warned that a team's keys would be
deleted but said nothing about models. #29977 made team deletion also
delete the team's BYOK models, so the modal copy was understating what
gets removed.
The warning banner now mentions models alongside keys, and the
always-shown confirmation message does too so a team that has models but
no keys (the banner only renders when keys exist) still gets warned.
The "View Usage Guide" button on the legacy Usage page (shown when
DISABLE_EXPENSIVE_DB_QUERIES is set, i.e. SpendLogs has 1M+ rows) linked
to docs/proxy/spending_monitoring, which was removed from the docs and
now returns 404. Point it at docs/proxy/cost_tracking, which is live.
Fixes LIT-2724
A streaming request that breaks mid-flight, for example on a mid-stream read
timeout, still bills the provider for the chunks already delivered, yet the proxy
recorded that interrupted request as a zero-spend failure. An earlier revision
logged the recovered partial usage through the success path, which mislabeled a
failed request as a success and produced a misleading spend row
This recovers the partial usage where the failure is actually logged. The
streaming handler assembles the usage from the chunks seen so far and stashes it,
with its cost, on the logging object before firing the failure handlers. The
proxy failure hook lifts that usage and cost onto request_data before the
non-serialisable logging object is popped, and the spend-log writer records the
real partial spend on the failure row instead of a hardcoded zero;
get_logging_payload honors the recovered usage for the token columns and
_failure_handler_helper_fn preserves the recovered cost so the non-DB failure
loggers stay consistent
A request that recovers via a successful fallback is unaffected: the failure hook
only fires when the whole request fails, so the fallback's combined-usage success
row stays the single source of truth and there is no double counting
Resolves LIT-3825
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(otel): one v2 logger owns the global provider; scope tenant creds per exporter
The proxy published the OTel global TracerProvider before callbacks were
initialized, so no preset logger existed yet and a second generic logger was
built that won the global provider. Server spans then exported through a
different provider than the preset's gen-ai spans, orphaning the LLM span on
the preset backend. Publish after callback init and reuse the already-built
logger instead.
Separately, per-request tenant OTLP credentials were stamped onto every OTLP
exporter, leaking one backend's key onto a co-configured backend. Tag each
exporter with the preset that contributed it and apply dynamic credentials
only to the matching owner.
* fix(otel): satisfy Any-discipline on changed lines
Type the logger-selection parameter as Sequence[object] (isinstance narrows
it), cast the list[Any] global at the single call site, and pass model_copy a
typed dict[str, str] update so no changed line carries an Any value.
* fix(otel): annotate the untyped-global boundary with any-ok
select_global_otel_v2_logger consumes litellm._in_memory_loggers, a shared
List[Any] global this change does not own. A cast doesn't satisfy the
Any-discipline checker (it inspects the inner expression), and re-annotating the
global is out of scope, so mark the single boundary line any-ok.
* test(otel): cover the startup global-provider publish via injectable helper
The publish step lived inline in proxy_startup_event (a FastAPI lifespan unit
tests do not execute), so its lines were uncovered though the selection logic
was tested. Extract publish_global_otel_v2_provider, which selects the single v2
logger and publishes its provider through an injected setter, and unit-test that
the published provider is the selected logger's. proxy_server delegates to it.
* refactor(otel): select global provider from the registered owner, not a list scan
The startup publish picked the global TracerProvider by scanning
_in_memory_loggers for the first OpenTelemetryV2, re-deriving an answer the
factory already settled: the first logger built registers itself as
proxy_server.open_telemetry_logger, and every other v2 path (guardrail, identity
seeding, phase spans) routes through that owner via _registered_v2_logger. Pass
that owner into select_global_otel_v2_logger so the global provider reuses the
same logger instead of an independent, order-dependent guess; the list scan
remains the SDK-path fallback. The owner is injected at the proxy call site to
keep the helper free of hidden global reads.
* refactor(otel): type ExporterSpec.owner as an ExporterOwner enum
The owner field carried free-form strings that had to match preset callback
names. Introduce a str-based ExporterOwner enum (values equal to the callback
names, so per-request credential routing's owner==callback_name comparison still
holds) and have each preset tag its exporter with the enum member.
* refactor(otel): rename ExporterOwner.ARIZE to ARIZE_AX
Distinguish the hosted Arize AX backend from Arize Phoenix at the member level
while keeping the value 'arize' (the public callback name routing compares
against). Add a comment noting AX and Phoenix are separate backends.
* fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough
When a guardrail blocks a streaming request pre-call by raising
ModifyResponseException (or RejectedRequestError), chat_completion streams the
violation message back as a 200 by building a CustomStreamWrapper. It read the
logging object from the outer request body (`data.get("litellm_logging_obj")`),
but that dict never carries litellm_logging_obj -- it diverges from the
processor's data at function_setup, and only the processor copy (exposed as
e.request_data, already bound to `_data` here) gets the logging object
attached. CustomStreamWrapper.__init__ then dereferences
`logging_obj.model_call_details` on None and 500s the request with
"AttributeError: 'NoneType' object has no attribute 'model_call_details'".
Read logging_obj from `_data` (= e.request_data) in both streaming
passthrough handlers so the refusal streams correctly. The non-streaming and
the anthropic/responses passthrough paths were unaffected.
Adds a regression test asserting the wrapper receives the logging object from
e.request_data rather than None.
* test(proxy): cover RejectedRequestError streaming passthrough
The streaming logging_obj fix was applied to both the ModifyResponseException
and RejectedRequestError handlers, but only the former had a regression test.
Extract a shared helper and add a parallel test for the RejectedRequestError
streaming path so both handlers stay guarded against the None-logging_obj crash.
---------
Co-authored-by: Joseph Barker <joseph.barker@rubrik.com>
Switch the zizmor check to fail on any finding at medium severity or
above (advanced-security off, min-severity medium, annotations on) so it
can be promoted to a required check, and pin the engine to zizmor 1.24.1
through zizmor-action v0.5.6 for deterministic runs.
Clear the findings that were outstanding so the check passes: correct
mismatched action pin version comments, scope the proxy endpoint
workflow's id-token and pull-requests permissions to the jobs that use
them, and mark the server-root-path docker build as non-publishing while
dropping its shared gha build cache.
* feat(proxy): add configurable response headers middleware
Adds a small ASGI middleware that sets standard response headers
(X-Frame-Options, Content-Security-Policy frame-ancestors, X-Content-Type-Options)
on proxy and UI responses. Strict-Transport-Security is optional and gated
behind LITELLM_ENABLE_HSTS for HTTPS deployments. Values use setdefault so a
route that sets its own header is preserved.
* feat(proxy/ui): make login page credentials hint configurable
build_ui_login_form accepts a hide_default_credentials_hint parameter and
google_login reads LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (or general_settings)
so the legacy login page behaves consistently with the new UI. Also collapses
a duplicated branch and removes an unused variable and module-level constant.
* fix(proxy/ui): apply credentials hint flag on /fallback/login
The /fallback/login handler still rendered the default-credentials hint
regardless of LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT. Collapse its duplicate
branch and forward the flag, matching google_login, so all login surfaces
behave consistently. Adds regression tests for /fallback/login and makes the
ui_sso test helper restore os.environ so env vars do not leak across tests.
Cut the default "Virtual Keys" landing (?page=api-keys) over to a path route at
(dashboard)/api-keys. The dashboard is extracted into a shared ApiKeysDashboard
component used by both the new route and the index's inline render, so there's
no duplication. Adding the MIGRATED_PAGES entry repoints the sidebar item and
redirects ?page=api-keys to /ui/api-keys.
The index is the post-login landing and still hosts the legacy switch for the
not-yet-migrated pages (models, pass-through, usage) plus the invitation flow,
so it stays. The auto-redirect now fires only for an explicit ?page= param,
leaving the bare /ui/ landing to render inline; this keeps the return-URL
handling and the invitation_id flow (both of which run at the bare landing)
intact, where a blanket redirect would have dropped them. The new route uses
useAuthorized for the login gate, matching every other migrated route.
