mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
10113 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mubashir1osmani
|
ae5e7abe16 |
test(e2e): batch/file API e2e suite
Adds a black-box e2e suite under tests/e2e/batches/ for the batches and files API. Nothing is imported from the litellm codebase; the tests drive the live proxy over HTTP and verify state through the generated prisma client, so they catch real regressions rather than re-asserting internal calls. It covers the gemini managed-files upload, verified by reading the file back ACTIVE from the provider; a managed-object poll-cap guard for the #23472 OOM that seeds more than one page of rows into real Postgres and watches the poll cycle in the proxy logs; and a vertex streaming-upload memory guard for the LIT-3382 OOM (gated, for an environment with real memory headroom). The pure helpers have unit coverage, and the memory sampler raises rather than passing vacuously when the cgroup read is unavailable |
||
|
ishaan-berri
|
4efce809d0
|
feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through callbacks (#31134)
* feat(proxy): add logging_endpoints package init
* feat(proxy): add POST /v1/callbacks/logs to replay logging payloads through the success/failure callback fan-out
* feat(proxy): register callback_logs_router
* test(proxy): add logging_endpoints test package init
* test(proxy): cover /v1/callbacks/logs replay, admin guard, and partial-failure handling
* refactor(proxy): move callback-logs request/response models to litellm/types/proxy
* refactor(proxy): wrap callback-logs replay in CallbackLogsReplayer class with payload logging
* test(proxy): update callback-logs tests for class-based replayer and separated types
* fix(proxy): cover /v1/callbacks/ in backend component allowlist
The new /v1/callbacks/logs route was dropped by both component
allowlists, failing test_gateway_plus_backend_covers_full_app. It's an
admin-only spend-logging route, so it belongs on the backend (control
plane) alongside the existing /callbacks family.
* refactor(proxy): use builtin dict/list generics in callback-logs endpoint
Switch Dict/List from typing to builtin dict/list to satisfy the ruff
strict-rule budget (UP006).
* refactor(proxy): use builtin dict/list generics in callback-logs types
UP006: builtin generics over typing.Dict/List.
* chore(ui): regenerate schema.d.ts for /v1/callbacks/logs
Run npm run gen:api to add the CallbackLogRecord/CallbackLogsRequest/
CallbackLogsResponse types and the /v1/callbacks/logs path, keeping the
dashboard types in sync with the proxy OpenAPI spec.
* fix(proxy): force stream=False when replaying callback logs
A replayed StandardLoggingPayload is a terminal, fully-aggregated event —
the producer (e.g. the rust realtime gateway) already collected the whole
session before POSTing. Marking the rebuilt Logging object as streaming made
async_success_handler wait for a complete_streaming_response that never
arrives, so the spend log was never written. Realtime sessions now land in
LiteLLM_SpendLogs.
* feat(litellm-rust): CustomLogger callback layer posting to /v1/callbacks/logs
integrations/ mirrors litellm/integrations/: a sync, typed CustomLogger trait
(base contract), a typed StandardLoggingPayload, and LiteLLMPythonProxyAPILogger
— the first concrete logger, owning a bounded channel + background worker that
batches and POSTs to the Python proxy's /v1/callbacks/logs.
* feat(litellm-rust): RealTimeStreaming per-session log collector
1:1 with Python's RealTimeStreaming: observe() accumulates O(1) usage/model/id
per event (never buffers frames); log_messages() builds one StandardLoggingPayload
on session close and fans out to the CustomLogger callbacks. request_id == the
OpenAI realtime session id (sess_…), with the gateway id as fallback.
* feat(litellm-rust): wire realtime logging into the splice (lock-free observe)
The collector is owned on the splice task and observed via a synchronous &mut
callback threaded through providers::realtime::realtime() — no Arc/Mutex/atomic
on the per-frame hot path. On session close the bridge flushes one payload.
AppState carries the registered loggers; main spawns the proxy logger.
* docs(litellm-rust): ai-gateway realtime logging architecture
* docs(litellm-rust): document request-log egress to the LiteLLM control plane
Add a 'Request logging' guide to the ai-gateway README: how to point the gateway
at a LiteLLM proxy via LITELLM_PROXY_BASE_URL (+ LITELLM_MASTER_KEY for the
admin-only /v1/callbacks/logs POST), and the non-blocking / one-payload-per-session
behavior.
* feat(litellm-rust): make log-egress tunables env-overridable
Channel capacity, batch size, and flush interval now read from
LITELLM_LOG_CHANNEL_CAPACITY / LITELLM_LOG_BATCH_SIZE / LITELLM_LOG_FLUSH_INTERVAL_MS,
falling back to the DEFAULT_* consts on missing/invalid/non-positive values.
Grouped behind an EgressTunables::from_env() read once at logger construction.
* docs(litellm-rust): document log-egress tuning env vars
* docs(litellm-rust): require constants in a crate-level constants.rs
Mirror of Python's litellm/constants.py rule — magic numbers and fixed strings
go in src/constants.rs, not inline in feature modules; env-overridable tunables
keep their DEFAULT_* value there.
* refactor(litellm-rust): move ai-gateway constants into constants.rs
Per the new rule: the log-egress defaults (proxy base, ingest path, channel
capacity, batch size, flush interval) and the realtime provider default move to
crates/ai-gateway/src/constants.rs; modules import from it.
* ci: run logging_endpoints tests in the proxy-infra coverage shard
tests/test_litellm/proxy/logging_endpoints wasn't in any coverage-uploading
job, so callback_logs_endpoints.py showed only import-level coverage (~35%) on
codecov/patch despite being ~98% covered locally. Add it to proxy-infra's
test-path so the test is exercised under --cov.
* fix(litellm-rust): hash the master key before logging — never send the raw credential
Greptile/Veria P1: user_api_key_hash was the plaintext LITELLM_MASTER_KEY, which
fans out to spend logs and every callback (Langfuse/Datadog) and could be
recovered from logs. SHA-256 it (auth::hash_token, matching the proxy's
hash_token); the field is named *_hash and the proxy stores it verbatim when it
isn't sk-prefixed, so the DB value is identical with zero plaintext exposure.
* fix(litellm-rust): observe realtime logging on upstream events only
Greptile P1: observe ran on the client->upstream arm too, so an authenticated
client could send a fabricated response.done and inflate its own spend log.
session.created/response.done are server->client events; observe the upstream
arm only.
* feat(proxy): bound callback-logs batch + return per-record failures
Greptile P2: cap /v1/callbacks/logs at MAX_CALLBACK_LOG_RECORDS (default 1000,
env-overridable) so one POST can't trigger an unbounded callback/DB fan-out; and
return per-record {index, error} failures so a caller (the rust gateway) can
distinguish a transient callback error from a structurally bad payload.
* chore(ui): regenerate schema.d.ts for CallbackLogFailure / failures field
* fix(constants): make MAX_CALLBACK_LOG_RECORDS a plain constant
It doesn't need to be env-configurable (only the rust egress tunables are). As an
os.getenv var it tripped tests/documentation_tests/test_env_keys.py, which requires
every env key to be documented in the (separate-repo) config_settings.md. Plain
constant → not scanned → code-quality + documentation checks pass.
* docs(litellm-rust): trim ai-gateway ARCHITECTURE.md to one diagram + notes
* docs(litellm-rust): tighten the README request-logging section
* docs(litellm-rust): ARCHITECTURE.md is just the diagram (gateway = inference, spend = callback)
* docs(litellm-rust): drop em-dashes from the request-logging section
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
|
||
|
Mateo Wang
|
1b81148f2a
|
test: add e2e tests for spend, budgets and llms (#30869)
* tests: add e2e tests for spend, budgets and llms
* style: make chained comparison of status_code clearer
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* remove e2e_tests folder
* test: add spend tracking tests
* test: multi-window budgets coverage
* fix: p0 issues, added types and shared functions for each test suite
* chore: add config.yml
* test: passthrough endpoints stream/non-stream e2e
* style: carry clearer status_code comparison into renamed e2e dir
* fix: rename cost breakdown function
* fix: pydantic validation for budget info, dont allow explicit type cast
* refactor: migrate to gateway client
* test: add custom pricing tests
* chore: change master key
* test(e2e): address greptile review feedback
Remove the duplicate cache/cache_params block in the gateway config so the two
can't silently diverge under future edits. Reorder the soft-budget test to assert
the call isn't a budget block before require_successful_call, since that helper
hard-fails any non-2xx and left the budget-block check unreachable; the misleading
"skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it
so a failed delete doesn't leak a budget on the shared proxy. Scope the
spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup
import so a broader "pytest tests/" run isn't left with a mutated path.
* test(e2e): drop misleading skip comment on require_successful_call
require_successful_call fails hard, it does not skip; the trailing
comment was factually wrong. The function name already states intent,
so the comment is removed in both per-model and tag budget helpers.
* test(e2e): assert budget-isolation invariant before success check
On the should-still-succeed path of the per-model and tag isolation
tests, check is_budget_block before require_successful_call. If the
isolation bug fires the unaffected model/tag is blocked, so asserting
the specific 'blocked by X' invariant first yields the diagnostic
message instead of a generic upstream-failure. Matches the ordering in
test_soft_budget_e2e.py.
* fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows
* fix(e2e): run case init() inside try so partial-init failures tear down
run_case called case.init() outside the try/finally that runs teardown(), so a
case that registers cleanups progressively (create team, then user, then key)
and then fails partway through init() would leak the already-created entities on
the long-lived shared proxy. Move init() inside the try so teardown always runs.
Add a regression test that registers a cleanup then raises mid-init and asserts
the resource is still released.
* test(e2e): mark known pricing-leak isolation test xfail(strict)
test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy
gap (a deployment's custom per-token pricing leaks into the shared cost map for
sibling deployments of the same underlying model) and was left unconditionally
failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True)
so the suite stays green while the leak persists and turns into a failure the
moment isolation is fixed, prompting the marker's removal.
* refactor(e2e): make suite pass its shipped strict basedpyright config
The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright
--project tests reported four errors in it: three reportAny on the parametrize
ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed
autouse fixture _require_live_proxy. Replace the untyped lambda with a typed
_case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and
rename the fixture to require_live_proxy so basedpyright no longer treats it as an
unused private function (it is referenced only by pytest's autouse machinery).
basedpyright --project tests now reports zero errors.
* fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory
* test(e2e): run harness unit tests without a live proxy
The autouse session fixture skipped the whole tests/e2e session when no proxy
answered, which also skipped test_lifecycle.py, a pure unit test of run_case that
never touches the proxy. A regression test that silently skips gives no signal,
so the skip now lives in pytest_runtest_setup gated on the same e2e marker the
spend-log truncate guard already uses: live tests skip when no proxy is up while
harness unit coverage always runs. The liveness probe is cached with lru_cache so
it still runs once per session
* test(e2e): clean up gateway config comment debris
Fix the typo on the header comment and drop the orphaned namespace/ttl
comment remnants left indented under cache_params; the active values are
already set above. Flagged by greptile review.
* fix: add new tests, split gateway
* test(e2e): type the redis spend-counter probe for strict basedpyright
The new cold-counter reseed test drove its redis client untyped, so the strict
tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the
file landed: scan_iter/get came back unknown and the pool.map lambda had an
untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING
import (the runtime import stays lazy so the suite still skips, not errors, when
redis is absent), which resolves scan_iter to Iterator[str] and get to str | None,
and replace the lambda with a typed inner function mirroring _burst. basedpyright
--project tests is back to zero errors.
* test(e2e): xfail the known team multi-window failure and isolate member teardown
Greptile flagged two issues in the mirrored split-gateway commit. The team
multi-window budget test documents a real /team/new write bug (budget_limits go
straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and
/team/update paths) and was left as an unconditional hard failure, which would
turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing
isolation test so the suite stays green while the bug persists and flips to a
failure the moment the write is fixed and the marker should go.
The class-scoped member fixture in test_team_member_budget_e2e.py tore down its
key, user, and team sequentially with no exception isolation, so a failed
delete_key would strand the user and team on the long-lived shared proxy. Route
cleanup through a ResourceManager: register each delete progressively and run them
LIFO best-effort in a finally, so a partial-setup failure still releases what came
before and one failed delete never blocks the rest.
* test(e2e): set fast budget-reset cadence in gateway config so staging windows reset within e2e timeouts
* test(e2e): surface real /spend/tags errors instead of masking them as missing tags
The spend-tracking e2e client swallowed every non-200 from /spend/tags into an
empty list, so a real server error or a response-shape mismatch showed up only as
the generic "tag never appeared in /spend/tags" with no diagnostics. That masking
is what made the original cluster failure undiagnosable.
spend_by_tags now raises SpendTagsError carrying the actual HTTP status and body
for any non-Success result, and poll_tag_spend fails fast on a hard server error
rather than polling it into a timeout; eventual consistency only manifests as a
200 whose payload does not yet carry the tag, so only that case waits. The tag
test now reports the last observed status and asserts the endpoint returned 200
at least once, with no weakened assertions.
Hardening surfaced the real defect in the test itself: /spend/tags returns a
top-level JSON array (List[LiteLLM_SpendLogs]), but the client validated against a
SpendTagsResponse dict wrapper that never matched, so every call fell through to
the empty-list mask. Wired spend_by_tags to the existing TagSpends RootModel and
removed the dead SpendTagsResponse model. Verified against the real Postgres that
request_tags is stored as proper JSONB arrays and /spend/tags aggregates them
correctly, so there is no encoding bug to fix here.
* test(e2e): drop flaky test_tag_spend_matches_sum_of_tagged_logs
The test wrote tagged requests and polled /spend/tags expecting read-after-write
consistency. /spend/tags itself is fine; verified live that request_tags is stored
as a JSON array and the endpoint reflects a fresh tag within seconds, so the
failures were a timing flake under full-suite load rather than a real defect.
Coverage is retained by test_request_tags_round_trip (tags persist onto the row)
and the /spend/tags route probe in test_spend_routes.py.
Also remove the now-dead tag-spend scaffolding this test was the only user of:
poll_tag_spend, spend_by_tags, TagSpendPoll, SpendTagsError, the TagSpend/TagSpends
models, and their imports.
* test(e2e): widen budget-reset wait windows to de-flake wall-clock-aligned resets
The short-window reset tests asserted the reset landed within WINDOW_SECONDS + 45
(~75s), but the 30s budget window is wall-clock-aligned, so the reset can land up
to a full window after start, then the rescheduler (~15-20s) zeroes the spend, plus
poll and DB lag. A real run measured 84s, just over the 75s bound, and which of the
short-window siblings tripped flipped run to run. Widen the wait loops to 150s and
the elapsed assertions to WINDOW_SECONDS + 90 (120s for the key test). A genuinely
stuck rescheduler is still caught by the wait-loop timeout, so this only removes the
timing flake, not the regression signal.
* test(e2e): let the spend-counter reseed test reach a cluster-mode TLS redis
The test's _redis() built a standalone, non-TLS client on the docker-compose
defaults (localhost:6380), so against the EKS serverless ElastiCache (cluster-mode
+ TLS) it could never connect and the test skipped. Honor E2E_REDIS_SSL and
E2E_REDIS_CLUSTER so it builds a TLS RedisCluster client when the deploy provides
them, and E2E_REDIS_NAMESPACE so the counter is read with a direct GET (cluster-safe)
rather than a keyspace scan that can't span shards. The local standalone path and the
graceful skip-on-unreachable behavior are unchanged.
* test(e2e): take the direct-GET spend-counter path on E2E_REDIS_CLUSTER
The gateway's cache sets no namespace, so the counter key is the bare
spend🔑<hash>. Trigger the cluster-safe direct GET on E2E_REDIS_CLUSTER (not
only on E2E_REDIS_NAMESPACE) so the cluster deploy need not set a namespace it
does not use; the namespaced key is still tried first when a namespace is given.
* test(e2e): use REDIS_HOST/REDIS_PORT and drop the unused redis knobs
The runner is a standalone test pod, so the proxy's own REDIS_HOST/REDIS_PORT
names are unambiguous - no E2E_ prefix needed. The only deployed redis it talks
to is the serverless ElastiCache (always TLS + cluster), so that is inferred from
REDIS_HOST being set rather than carried as ssl/cluster knobs. Stage sets no cache
namespace (bare counter key, read directly on the cluster) and is passwordless, so
the namespace and password env are gone; the local namespace is still handled by
the standalone SCAN.
* test(e2e): replace the vacuous failure-row test with per-model attribution
test_failure_call_writes_failure_status_row had two skip hatches (the call did
not fail, or no failure row landed) and never asserted anything on this proxy -
gemini accepts an empty message (HTTP 200), and live failure-row logging is
non-deterministic across providers. Replace it with a deterministic check: one
key calling gemini-2.5-flash and claude-haiku-4-5 gets one spend row per call,
each carrying its own model and a nonzero cost, under distinct request_ids that
match the call's response id. Verified live on stage (gemini/gemini-2.5-flash
$0.00053, anthropic/claude-haiku-4-5 $0.000038, distinct ids matching the
responses). Failure-status row construction stays covered by the unit suite.
* test(e2e): assert /spend/logs returns the key's spend without 5xx
Regression for the intermittent 500s on /spend/logs (DB query / serialization
errors under load). The existing spend_logs() helper swallows non-success
responses into an empty list, so a 500 looks identical to 'rows not flushed yet'.
This test queries the endpoint directly and asserts a Success response on every
poll, failing loudly on any 5xx, then requires the call's nonzero spend to surface.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
|
||
|
tin-berri
|
bbef1b84ab
|
feat(mcp): graft v2 resolver onto _create_mcp_client (none + api_key static family) (#31058)
* feat(mcp): add v1 bridge + none/api_key resolver arms (unwired) PR4a of the MCP v2 outbound-credential migration, stacked on the resolver skeleton. Builds the bridge for the first live modes without wiring it onto the request path: - resolver.py: the none arm (NoOpAuth) and the api_key shared-key arm (StaticHeaderAuth from the config); the BYOK source and the other five arms stay not_implemented. - adapter.py: the v1 <-> v2 edge (to_subject, to_server_spec, raise_public, should_defer). to_server_spec maps only none + the static-header family and returns None to defer every other mode to v1. Imports v1, kept out of the package __init__ so the resolver core stays v1-free. - MCPClient gains an optional resolved_auth that feeds the factory's auth= slot, taking precedence over the SigV4 aws_auth; default None keeps current behavior. Nothing calls these from _create_mcp_client yet, so production behavior is unchanged; the graft lands in PR4b. Unit tests cover the two arms, the full mapping table, and the auth plumbing. * feat(mcp): graft v2 resolver onto _create_mcp_client for migrated modes Wire the none + api_key static-family resolver arms from PR4a onto v1's live request path. In _create_mcp_client's HTTP/SSE branch, to_server_spec decides per mode: a migrated mode resolves through the injected UpstreamCredentialProvider and feeds the resulting httpx.Auth into the new resolved_auth slot; every other mode returns None and falls through to the unchanged v1 construction. resolve_mcp_auth now runs only when the mode defers, so a migrated server skips the v1 token-exchange / M2M I/O. stdio is untouched: auth_type/auth_value never reach the upstream on the stdio path (_get_auth_headers is HTTP/SSE only), so there is nothing to graft there. No v1 code is deleted yet; resolve_mcp_auth's static return still backs stdio and the not-yet-migrated modes until later PRs retire it. * test(mcp): cover the v2-resolver graft in _create_mcp_client Regression tests for the PR4 graft. Migrated HTTP modes resolve through the provider into resolved_auth: none -> NoOpAuth, and the static api_key family emits the right header per scheme (X-API-Key, Bearer, token, raw authorization, base64 basic). Deferred modes (oauth2) and a missing static token fall back to v1's auth_value. A stdio server with a migrated auth_type still defers to v1, since httpx.Auth never reaches the subprocess. A resolver Error is mapped to the public HTTP contract (401) via an injected provider, exercising the DI seam. * fix(mcp): defer to v1 when an inbound credential would be overridden The graft attaches the resolved static credential as an httpx.Auth, whose auth flow writes its header after extra_headers. That silently overrode an inbound Authorization: a per-request mcp_auth_header override, or a header supplied via a guardrail hook / static_headers / forwarded caller header. v1 lets those win, so the graft had inverted the credential precedence for the migrated static modes. Mirror the v2 egress credential-isolation invariant: defer the request to v1 when mcp_auth_header is set, or when the header the resolved credential would write is already present in extra_headers. none writes no header, so it never defers. * test(mcp): cover the credential-isolation defer guard Regression tests for the precedence fix. A per-request mcp_auth_header override and an Authorization already present in extra_headers (guardrail hook like the JWT signer, static_headers, or a forwarded caller header) both defer a migrated static server to v1 so the inbound credential wins; none stays on v2 and does not clobber an inbound Authorization since NoOpAuth writes nothing. The deferred cases assert resolved_auth is None, which fails if the guard is removed. * refactor(mcp): resolve inbound-header conflict on v2 instead of deferring For an Authorization already supplied via extra_headers (a guardrail hook such as the JWT signer, static_headers, or a forwarded caller header), keep the request on the v2 path and skip resolved_auth rather than deferring to v1. The inbound header still wins since nothing overwrites it, but hooks no longer pin a v1 fallback, which is what lets resolve_mcp_auth be retired once the remaining modes migrate. The mcp_auth_header per-request override still defers to v1, since that value becomes the upstream credential rather than sitting in extra_headers; that defer falls away once the per-user modes stop writing mcp_auth_header. * fix(mcp): clear UP037 lint gate and fix allowed-servers test under the graft adapter.py uses `from __future__ import annotations`, so the quoted "UserAPIKeyAuth" / "MCPServer" annotations in to_subject/to_server_spec/_shared_key_spec were unnecessary and pushed UP037 over the strict-rule budget; drop the quotes. test_list_tools_only_returns_allowed_servers passed a MagicMock as user_api_key_auth. The graft now builds a Subject from the principal, and the MagicMock's non-string org_id/user_id fail Subject validation, so the listing came back empty. Use a real UserAPIKeyAuth instead (MagicMock for an injected dependency was the anti-pattern here). * test(mcp): assert config token via resolved_auth, not the headers dict test_mcp_server_config_auth_value_header_used inspected _get_auth_headers(), but the graft now carries the static credential on the client's httpx.Auth (resolved_auth) and writes the header at send time, so that dict is empty. Assert the header the StaticHeaderAuth emits onto the request instead. Both config keys (authentication_token, auth_value) stay covered. * chore(typecheck): set reportMatchNotExhaustive slack to 0 The previous slack of 3 put the ceiling at baseline + slack = 4, so a newly non-exhaustive match (for instance dropping an Error arm off a Result match) could land without tripping the gate. Setting slack to 0 pins the ceiling at the current baseline of 1, so any added non-exhaustive match now fails CI while the one pre-existing violation in router.py stays within budget |
||
|
tin-berri
|
6003187165
|
fix(mcp): let proxy admins assign MCP servers to teamless keys (#31126)
Creating or updating a key with a specific (non-allow_all_keys) MCP
server or access group failed with a 403 when the key had no team:
Key is not in a team. Only globally available (allow_all_keys) MCP
servers can be assigned
validate_key_mcp_servers_against_team computed the allowed set as
team servers + allow_all_keys servers. For a teamless key the team
set is empty, so the allowed set collapsed to just allow_all_keys
servers and any explicitly-picked server or access group was rejected.
This was asymmetric with runtime: get_allowed_mcp_servers honors a
teamless key's own object_permission.mcp_servers verbatim, with no
team gate and no allow_all_keys filter. So the create/update path
refused to persist a grant the run path would have served.
Thread is_proxy_admin into the validator from both call sites
(/key/generate and /key/update). When a key has no team and the
caller is a proxy admin, the requested servers and access groups are
folded into the allowed set so the existing subset checks pass. A
proxy admin can already reach every MCP server, so there is nothing
to escalate. Non-admins and every team-scoped key are unchanged.
Resolves LIT-3815
|
||
|
mubashir1osmani
|
56825926af
|
fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads (#31036)
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
|
||
|
Sameer Kankute
|
8bca05d311
|
fix(anthropic): sanitize tool_use ids on native /v1/messages path (#31094) | ||
|
mubashir1osmani
|
e0c8a6b483
|
fix(proxy): expand all-proxy-models sentinel in direct access lookup (#31153)
A user provisioned with "All Proxy Models" stores the literal "all-proxy-models" sentinel in user.models. get_direct_access_models looked that string up as a real model_name via get_model_list, which matched no deployment, so /v2/model/info marked every model direct_access=false and the Models + Endpoints page rendered empty for such users when they have no teams. The model dropdown / Playground worked because get_key_models already expands the sentinel to the full proxy model list, hence the inconsistency in the report. Expand the sentinel to all non-team deployment ids via get_model_ids(exclude_team_models=True), the same call the PROXY_ADMIN branch in the caller already uses. This fixes both /v1/model/info and /v2/model/info since they share _populate_team_access_on_models. Empty user.models stays "no direct access" to match get_key_models semantics. Fixes #22791 |
||
|
Krrish Dholakia
|
d0706c17fe
|
fix(anthropic): drop unsupported speed param with drop_params (#31152)
* fix(anthropic): drop unsupported speed param with drop_params Anthropic fast mode (speed) is Opus 4.6/4.7/4.8 on the direct API only. Strip speed when the model map lacks supports_speed and drop_params is set, for both chat completions and /v1/messages passthrough. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): allow supports_speed in model map schema The new supports_speed flag on Opus entries must pass JSON schema validation in test_aaamodel_prices_and_context_window_json_is_valid. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(review): raise on unsupported speed without drop_params Passthrough /v1/messages now raises UnsupportedParamsError when speed is unsupported and drop_params is false. Emit drop warning from map_openai_params when speed is silently skipped. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): gate speed param by routed provider, not just model id Vertex, Azure, and Bedrock reuse the shared Anthropic transform and strip their provider prefix first, so a bare `claude-opus-4-8` resolved to the direct-API model-map entry (`supports_speed: true`) and forwarded `speed` upstream, producing the same 400 that drop_params is meant to prevent. Gate fast mode on `custom_llm_provider == "anthropic"` so it stays on the direct Anthropic API across both the chat completions and `/v1/messages` passthrough paths, and collapse the duplicated drop/raise logic in map_openai_params into the shared `_maybe_drop_speed_param` helper. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
360adbe765
|
fix(mcp): resolve config-defined servers in per-user credential and env-var endpoints (#31171)
The per-user BYOK, OAuth (OBO), and env-var management endpoints resolved the target MCP server through a DB-only lookup (get_mcp_server / get_all_mcp_servers_for_user). A server defined in config.yaml lives only in the in-memory registry and never gets a row in LiteLLM_MCPServerTable, so those endpoints raised 404 "MCP Server <id> not found" (or 403 for non-admins) before any credential could be stored, leaving config-server users unable to connect and forced to re-authorize forever. Route all three through a single registry-aware resolver: DB first, then the in-memory registry (built into LiteLLM_MCPServerTable via _build_mcp_server_table, the same fallback fetch_mcp_server already uses), then the canonical get_allowed_mcp_servers authorization the MCP gateway enforces on tool calls. Admins get a 404 for an unknown id; non-admins get 403 for a missing-or-forbidden server so server ids stay non-enumerable. This also closes a gap where the two store endpoints performed no per-server authorization at all. |
||
|
ishaan-berri
|
1d5ab42e14
|
feat: add minimal rust router + axum ai-gateway calling router.realtime (2/2) (#31135)
* add CoreError::Routing variant for deployment selection failures * add minimal Rust Router (simple-shuffle) mirroring router.py spec * add litellm-router crate manifest * add ai-gateway POST /v1/realtime handler calling router.realtime * add ai-gateway health routes * wire ai-gateway routes into the axum app * add ai-gateway AppState holding the shared router * add ai-gateway axum server entrypoint * add litellm-ai-gateway binary crate manifest * docs: add ai-gateway folder-architecture AGENTS.md * register router + ai-gateway crates and axum/rand deps in workspace * update Cargo.lock for router + ai-gateway crates * split router: extract model_list types into deployment module * split router: extract routing policy into strategy module * split router: move Router orchestration into router module * router lib: wire submodules and re-export public API * add read_model_list helper reusing ProxyConfig env/secret resolution * add GIL-activity tracker (records acquisitions, 30s window) * add GET /health/gil endpoint for polling GIL activity * add pyo3 load_router_from_config bridge (feature-gated, load-time only) * register /health/gil route in ai-gateway * wire build_router: load from python config when feature enabled * add optional pyo3 dep + python-config feature to ai-gateway * update Cargo.lock for optional pyo3 dependency * fix: satisfy strict ruff budget (FA100) in read_model_list * test: cover read_model_list env resolution + empty config * ai-gateway: bind localhost by default, warn on bad PORT/missing keys, wire gateway key * ai-gateway: add gateway_key to AppState for realtime auth * ai-gateway: require bearer auth + map unknown model to 404 on /v1/realtime * ai-gateway: move python interop into python/ with load-time-only AGENTS.md * ai-gateway: document auth, gil, and python folder in AGENTS.md * core: add router module (model_list types + simple-shuffle selection) * ai-gateway: dispatch realtime via core router + providers (drop router crate dep) * update Cargo.lock: fold router into core * workspace: drop crates/router member and litellm-router dep * read_model_list: reuse ProxyConfig.get_config (includes + os.environ + DB) instead of thin yaml read * ai-gateway: constant-time bearer compare + 500 (not 503) for unconfigured key * ai-gateway: trim stored gateway key to match trimmed bearer token * ai-gateway: add subtle dep for constant-time comparison * workspace: add subtle dependency * update Cargo.lock for subtle * core router: make strategy a folder (one module per strategy, simple_shuffle) * providers: make realtime() a streaming splice (client stream <-> OpenAI) instead of collect * providers: add futures-channel dev-dep for the streaming live test * ai-gateway: make /v1/realtime a WebSocket (auth before upgrade, splice typed events) * ai-gateway: dispatch realtime as a stream splice * ai-gateway: route /v1/realtime via GET (WebSocket), drop POST * ai-gateway: enable axum ws feature + futures-util * update Cargo.lock for ws feature + futures-channel * core router: add has_deployment() for pre-flight model checks * ai-gateway: extract auth into auth/ module (single master key, LITELLM_MASTER_KEY) * ai-gateway routes: adopt router()-per-module template + merge in app() * ai-gateway: document auth/ + routes template in AGENTS.md * ai-gateway: realtime route as thin handler + service + transport * ai-gateway: auth as a RequireMasterKey extractor (idiomatic axum FromRequestParts) * ai-gateway: docs for auth extractor + simplified route template * ai-gateway: collapse realtime route to mod.rs + service.rs; docs for extractor/template * providers realtime: enforce idle timeout around the splice (reap stalled sessions) * ai-gateway: rename realtime service timeout param to idle_timeout --------- Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> |
||
|
Shivam Rawat
|
8dfe702e5b
|
fix(bedrock-mantle): honor api_base for VPC endpoint routing on bedrock/mantle/... (#31141)
The bedrock/mantle chat and messages paths hardcoded the public Mantle host and ignored api_base, so private VPC/VPCE/GovCloud endpoints could not be used. Route URL construction through a shared helper that prefers api_base and aws_bedrock_runtime_endpoint before falling back to the regional public host. Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Mateo Wang
|
286169d39b
|
fix(model_prices): correct regional processing uplift to gpt-5.4/5.5 series only (#31136)
* fix(model_prices): correct regional processing uplift assignment gpt-4.1, gpt-4o, gpt-5, and their variants were incorrectly carrying the 10% EU/US regional processing uplift multiplier. Per OpenAI's pricing docs, the uplift applies only to models released on or after 2026-03-05 (gpt-5.4 series and gpt-5.5 series). Removes the uplift from: gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, gpt-4o, gpt-4o-2024-08-06, gpt-4o-2024-11-20, gpt-4o-mini, gpt-5, gpt-5-pro, gpt-5-mini, gpt-5-nano. Adds the uplift to: gpt-5.4, gpt-5.4-mini, gpt-5.4-nano, gpt-5.4-pro, gpt-5.5, gpt-5.5-pro. * fix(model_prices): apply same regional uplift correction to backup file * fix(model_prices): add regional uplift to date-versioned gpt-5.4/5.5 siblings * test(model_prices): update data residency tests to use gpt-5.4 as the uplift model The tests were using gpt-5 which no longer carries the regional processing uplift after correcting which models have it. Switch to gpt-5.4 (released 2026-03-05, the cutoff date) and add a regression parametrize covering all pre-cutoff models to pin that they stay uplift-free. * test(batches): use gpt-5.4 for data residency uplift assertion batch_cost_calculator's data residency uplift test still pinned gpt-5, which no longer carries the regional processing uplift after this change. Switch it to gpt-5.4 (the canonical post-cutoff uplift model), matching the llm_cost_calc test update. --------- Co-authored-by: mgalbato <37748295+mgalbato@users.noreply.github.com> |
||
|
yucheng-berri
|
c2e06890ad
|
fix: tighten role-based visibility of config and MCP fields (#30587)
* fix: redact config and MCP secrets in read-only admin views GET /config/field/info and the MCP server list/detail endpoints returned secret-bearing fields to any caller with an admin view, including read-only admins. They now return those fields in full only to a full PROXY_ADMIN; every other caller gets the reduced, non-admin view, while non-sensitive fields remain readable. Regression tests cover the role-based visibility on both endpoints, including that a full admin still sees everything needed to populate the edit form. * fix: redact nested secrets in config field info for non-admins /config/field/info returned structured general_settings fields verbatim to any admin-view caller, so a view-only admin reading database_args received the nested aws_web_identity_token (a DynamoDB role-assumption credential) in plaintext. Recurse into dict/list field values and redact secret leaves for non-PROXY_ADMIN callers, leaving non-secret siblings and full-admin reads unchanged * fix: redact secret config values in /config/list for non-admins /config/list shared the same _user_has_admin_view gate as /config/field/info but returned each field value unredacted, so a view-only admin reading the list received pass_through_endpoints upstream Authorization headers verbatim. Route every general_settings value through a shared role-aware redactor (extracted from /config/field/info) covering the top-level and nested field paths, so non-PROXY_ADMIN callers get secret-bearing fields redacted while full-admin reads stay unchanged * chore(ci): allowlist _redact_secret_values_in_obj in recursive_detector The config secret redactor recurses over JsonValue, which is acyclic, and its depth is bounded by the operator-authored general_settings schema. Add it to the recursive_detector ignore list alongside the other bounded nested-redaction helpers (mask_dict, _redact_sensitive_litellm_params) * proxy: cap recursive secret redaction depth at 10 Match the cap on _redact_sensitive_litellm_params (the closest analog in the proxy, also recursive, key-name driven, returns a sentinel). The previous justification — bounded by operator-authored schema depth, JsonValue acyclic — is true today but is a property of the threat model, not an enforced invariant of the function. If a code path is ever added that pipes external input into general_settings (config import, migration tooling, JWT-driven settings, …) the assumption silently breaks. A local cap makes the invariant local. The cap branch fails closed: at _REDACT_SECRET_MAX_DEPTH the whole subtree is replaced with 'REDACTED' rather than returned verbatim. A future refactor that flips this to fail-open would let a deeply nested credential leak; the new regression test test_redact_secret_values_in_obj_fails_closed_at_max_depth guards against that. Updates the recursive_detector ignore-list rationale to point at the numeric cap rather than the structural argument. * test: actually exercise the depth cap in fails-closed test The previous fixture stored the leaf under the secret-named key 'aws_web_identity_token', which the recursor's key-name short-circuit redacts regardless of the cap — so the test passed both with and without the cap in place. Empirically confirmed: under an uncapped mutant the old fixture still hides the secret (key-name catches it), the new fixture leaks it (only the cap can stop it). Swap the leaf key to a non-secret name so the cap is the only redaction path exercised, making the test fail on mutation as advertised. |
||
|
Mateo Wang
|
f12c9bec48
|
fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119)
request_timeout was shadowed by router_settings.timeout: Router stored a single slot via `self.timeout = timeout or litellm.request_timeout`, so when a router timeout was set the configured request_timeout was never used. Provider calls with no per-model timeout (Bedrock especially) then fell back to the hardcoded 600s httpx client default. Mirrors PR #25701 and completes it on top of the CompletionTimeout work already on this branch. - Router: add an independent self.request_timeout and prefer it over router_settings.timeout in both _get_non_stream_timeout and _get_stream_timeout - http_handler: cached default clients now fall back to request_timeout instead of a hardcoded 600s - Replace the brittle `== 6000` default-detection heuristic with a single get_configured_request_timeout() resolver backed by an explicit request_timeout_explicitly_set sentinel (set from REQUEST_TIMEOUT env and litellm_settings), keeping the value-differs fallback for SDK assignment. This also fixes an explicit request_timeout of 6000 being coerced to 600 - CompletionTimeout no longer second-guesses the package default; the caller passes the explicitly-configured value or None Regression for LIT-2369. |
||
|
ishaan-berri
|
0a17c7c39f
|
feat: add LiteLLM Rust workspace with Mistral OCR bridge (#31033)
* docs(readme): add Deploy on AWS/GCP with Terraform section Adds a quickstart for the two published Terraform modules on the public registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste main.tf for each cloud, the one-time GCP Artifact Registry remote-repo command, and pointers to the registry pages for the full input surface. Sits inside the Get Started section, between the gateway/SDK table and Run in Developer Mode -- where someone scanning the README for "how do I deploy this" will land. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): add 1-click deploy buttons for AWS + GCP GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo and walks through `terraform apply` via the existing DeployStack tutorial (already shipped at terraform/litellm/gcp/examples/default/ TUTORIAL.md). User just picks a project. AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an in-browser, already-authenticated shell. User runs four commands (clone + cd + cp tfvars + terraform apply) once inside. There's no native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN "Launch Stack" + CodeBuild would be needed for that, and that's a separate piece of work. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * docs(readme): move AWS + GCP deploy buttons next to Render button * docs(readme): unify deploy button sizes and badge styles * docs(readme): bump deploy button height to 48 to match Render/Railway * docs(readme): bump AWS/GCP badge height to compensate for SVG padding * docs(readme): bump AWS/GCP badge height to 72 * docs(readme): bump AWS/GCP badge height to 84 * fix(readme): make deploy buttons same height (48px) https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc * docs(readme): flag GCP project ID substitution in image_registry * docs(readme): equalize deploy button heights and fix Cloud Shell button font GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px. Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface. * docs(readme): collapse Railway deploy anchor to a single line The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline. * Add Claude Fable 5 cost map entries as a data-only hotfix Backports only the model map changes from #30064 so deployments on released litellm versions pick up Fable 5 pricing, context window, and the adaptive thinking flag through the hosted cost map fetch without upgrading. Includes the supports_sampling_params flag on the 28 Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read by the gating that ships with the next release) and the matching one-line schema declaration so the map validation test passes. https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm * Add litellm rust workspace with mistral ocr bridge * address greptile rust ocr feedback * Simplify rust ocr entrypoint * rust(core): add Auth/Http/Network error variants * rust: add reqwest (rustls-tls) workspace dependency * rust(providers): depend on reqwest * rust(mistral): add complete_url + resolve_api_key helpers * rust(providers): end-to-end run_ocr orchestrator with shared client + timeout * rust(bridge): depend on litellm-core * rust(bridge): add GIL release accounting * rust(bridge): end-to-end ocr() + gil_stats(), GIL released for HTTP * ocr: add minimal Rust bridge (use_litellm_rust + rust_ocr) * ocr: route mistral to Rust when enabled; keep bare-str file rejection * litellm: export use_litellm_rust() * test(ocr): cover Rust OCR routing + toggle * rust: stop ignoring Cargo.lock * rust: commit Cargo.lock for reproducible builds * ci(rust): build with --locked to enforce the lockfile * Potential fix for pull request finding 'CodeQL / Module-level cyclic import' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for pull request finding 'CodeQL / Module-level cyclic import' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * ocr: lazily import rust bridge inside ocr() to break the import cycle the CodeQL autofix mangled * ocr: guard OCRResponse under TYPE_CHECKING so the annotation resolves * ocr: modernize rust_bridge typing (PEP 604, drop typing.Any/Dict) to satisfy strict-rule gate * ci: re-trigger checks * ci: re-trigger checks * Potential fix for pull request finding 'CodeQL / Cyclic import' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for pull request finding 'CodeQL / Cyclic import' Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * ocr: make rust_bridge a leaf (return raw dict, no litellm import) so the CodeQL autofix stops re-breaking it * ocr: wrap rust bridge dict into OCRResponse at the call site * test(ocr): assert rust_ocr returns the raw bridge dict * test(interactions): add budget_exceeded to expected status enum (Google updated the published spec) * ocr: resolve mistral key via get_secret_str before the rust path (secret-manager parity) * test(ocr): assert rust path resolves key via secret manager * rust(mistral): document that secret-manager resolution happens on the Python side * fix(ocr): honor timeout, logging, and missing-bridge fallback on Rust OCR path - Forward the caller's timeout into the Rust bridge so the fixed 600s client ceiling no longer overrides shorter deadlines or the library default. - Run update_from_kwargs and pre_call before invoking the Rust shortcut so observability, callbacks, and spend tracking match the Python path. - Fall back to the Python OCR path when litellm_python_bridge isn't importable instead of raising ImportError to callers. - Truncate upstream Mistral OCR error bodies before they cross the host boundary to avoid leaking document or prompt contents in CoreError::Http. * fix(ocr): log resolved api_base and headers on Rust path * refactor(ocr): inject the rust bridge via a typed seam, drop the importlib cycle dodge The rust OCR path was reached through importlib.import_module both for the bridge module and for probing the native extension, purely to keep CodeQL from flagging a cyclic import. rust_bridge has no litellm imports, so it is a leaf and main.py can import it statically without any cycle; the dance is gone Bridge selection now goes through a typed RustOcr Protocol and a load_rust_ocr() seam. use_litellm_rust() takes an optional injected bridge, so an embedder (or a test) can supply an alternative without reaching into sys.modules. The rust-path body moves into _run_rust_ocr(), which receives its dependencies (the bridge callable, the logging object, the key resolver) as arguments and is unit-tested by passing fakes in rather than monkeypatching class methods or module globals The tests are rewritten around that injection: the bridge is provided via use_litellm_rust(ocr=...), pre_call is observed through a spy logging object, and the missing-extension fallback is covered by load_rust_ocr() returning None when no wheel is built. Types were tightened along the way (a cast for the logging object, OCRResponse.model_validate for the bridge result) so no basedpyright per-rule count increases Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> * fix(ocr): preserve injected rust bridge across toggle calls use_litellm_rust() unconditionally assigned the keyword default of None to _rust_ocr_impl, so any call without ocr= silently dropped a previously injected bridge. Use a sentinel default so omission preserves the impl while ocr=None still clears it explicitly. * ci: run tests/test_litellm/ocr in the misc unit-test group The OCR test directory was not wired into any CI test group, so its coverage never uploaded to Codecov and patch coverage failed for new OCR lines. Add it to the misc group. * test(ocr): cover compiled-extension load and Python fallback paths Adds two tests so the Rust bridge module hits 100% and the ocr() fallback-to-Python branch is exercised: - load_rust_ocr() returning the compiled extension's ocr callable - ocr() degrading to the HTTP handler when no bridge is available * style(ocr): use PEP 604 X | None annotations in rust_bridge Converts Optional[X]/Union[...] to the X | None form so the new OCR code stays under the UP045 strict-rule budget gate (lint job). Safe at runtime — the module already has 'from __future__ import annotations'. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Yassin Kortam <yassin@berri.ai> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Krrish Dholakia <krrish+github@berri.ai> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
9c014716ec
|
fix(search): block server credential leak to caller-supplied api_base (#30682)
Search providers resolved the server-configured API key (e.g.
get_secret_str("SERPER_API_KEY")) in validate_environment whenever the
caller omitted api_key, while get_complete_url independently honored a
caller-supplied api_base. A caller who passes their own api_base and no
api_key therefore made the proxy send the operator's provider key to a
host they control; POST /search_tools/test_connection forwards
request-body api_base/api_key straight into asearch, so any authenticated
user could exfiltrate the server's search credentials.
Add a shared host-aware fallback in BaseSearchConfig.resolve_server_api_key
that only applies a server-managed secret when the caller-supplied
api_base is absent or resolves to a trusted host (the provider default or
the operator's own *_API_BASE env override); otherwise it refuses and asks
for an explicit api_key. The guard only triggers when a server secret
actually exists, so keyless and self-hosted providers (searxng, you.com
free tier) keep working. Every provider that carries a server secret is
migrated to the helper; dataforseo reuses the same guard for its
login:password basic-auth credentials.
This changes behavior for callers that previously passed a per-request
api_base while relying on a server-configured key: they must now pass an
explicit api_key, or the operator must configure the base via the
provider's *_API_BASE env var (which stays trusted).
|
||
|
Krrish Dholakia
|
c546b58c09
|
feat: add chat completions code interpreter loop (#31027)
* feat: add chat code interpreter loop * fix: address code interpreter pr checks * fix: satisfy strict lint budget * test: cover chat no-op interception * fix: address code interpreter review * fix: clean up agentic loop helpers * fix: preserve agentic loop controls * fix: generalize agentic loop params * fix: carry agentic state via metadata * fix: restore litellm params helpers * refactor: move chat code-interpreter loop out of provider code Dispatch the chat-completions agentic loop from a provider-agnostic helper (litellm/litellm_core_utils/chat_completion_agentic_loop.py) called from main.acompletion, instead of from OpenAI provider files. Register the agentic loop control fields in all_litellm_params so they stay LiteLLM-level and never become provider payload, removing the need for the OpenAIGPTConfig scrubber. No litellm/llms/ files are modified for this feature. * docs: explain chat agentic loop dispatch and litellm-level param registration * style: drop Any annotations and use PEP585 generics to satisfy ruff strict budget * docs: replace module docstring with one-line patch note |
||
|
Mateo Wang
|
1be957da17
|
fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint (#31053)
* fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint * fix(cloudflare): guard missing account id and migrate legacy /ai/run base Centralize the OpenAI-compatible api_base default in get_complete_url so it is built in one place instead of being duplicated in main.py. When neither api_base nor CLOUDFLARE_ACCOUNT_ID is set the call now fails fast with a clear error rather than sending a request to a URL containing the literal 'None'. An api_base still pinned to the legacy Workers AI '/ai/run' path is rewritten to the '/ai/v1' OpenAI-compatible endpoint with a deprecation warning, so users who hardcoded the previous default migrate gracefully instead of hitting a silently broken '/ai/run/chat/completions' URL. * fix(cloudflare): treat empty api_base as unset when resolving URL * fix(cloudflare): treat empty CLOUDFLARE_ACCOUNT_ID as unset An empty or whitespace-only CLOUDFLARE_ACCOUNT_ID slipped past the None guard and built .../accounts//ai/v1, producing the same confusing 404 the PR set out to prevent. Normalize the secret with normalize_nonempty_secret_str so blank values raise the explicit missing-account-id error instead. |
||
|
Mateo Wang
|
f26dbb60be
|
ci: make the basedpyright budget gate delta-vs-base (#31106)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* ci: re-run absolute basedpyright budget gate on push to long-lived branches The basedpyright budget gate counts codebase-wide errors per rule against a committed ceiling, but it only ran on pull_request against each PR's own head. Two PRs that each pass in isolation can together push a per-rule count over its ceiling once both merge, and nothing re-evaluated the budget on the merge commit, so the breach only surfaced on the next PR that happened to be checked out after the count crossed the line. Add a push trigger on the long-lived branches and a post-merge-budget job that re-runs the absolute gate on the merged tree, catching the accumulation on the merge commit itself. The existing pull_request jobs are guarded so their delta-vs-base gates don't misfire on push, where no PR base SHA exists. * ci: shallow-fetch the post-merge-budget checkout The post-merge-budget job only runs basedpyright over the working tree and the committed budget file; it never inspects git history, unlike the lint job whose delta-vs-base gates need full history. Drop its checkout from fetch-depth: 0 to fetch-depth: 1 to avoid cloning the whole repo history. * ci: scope post-merge-budget push trigger to long-lived branches On a push event the branches filter matches the branch being pushed to, not the PR target. The litellm_** glob, correct for the pull_request filter where it matches the target branch, therefore fired the post-merge-budget basedpyright job on every short-lived feature branch carrying the litellm_ prefix (litellm_dev_*, litellm_add_*, and so on), duplicating the PR lint job and burning ~10 minutes of CI per push. Restrict the push trigger to the long-lived branches PRs actually merge into (main, litellm_internal_staging, litellm_oss_branch), where budget accumulation happens. The pull_request filter keeps litellm_** so PRs targeting any long-lived branch are still linted. * ci: make the basedpyright budget gate delta-vs-base The basedpyright gate counted absolute codebase-wide errors per rule against a committed ceiling and ran only on each PR's own head. Two PRs that each pass in isolation could together push a rule past its ceiling once both merged, and because the gate had no comparison against the base, the next unrelated PR branched off the now-over-ceiling tree inherited a red it did nothing to cause. Give it the same shape as the ruff strict gate: a rule fails only when its total is both over the ceiling and higher than the count on the merge-base it merges into. Drift already in the base is never blamed on a bystander, while any change that actually grows a rule past the cap still fails. Head counts come from the existing stdin pipe; the base count is a second basedpyright pass over a detached worktree at the merge-base, reusing the head environment so import resolution matches and no second uv sync is needed. This obsoletes the push-triggered post-merge-budget job (and its event guards), which only detected accumulation after the fact; the delta check blocks it on the PR instead. Slack for reportReturnType and reportUnnecessaryComparison is raised to give real headroom under the cap. * refactor(ci): give the base ref its own name in type_check_gate cmd_check cmd_check took a parameter named base that held a git ref string, then rebound the same name to the dict of base-tree error counts returned by base_counts. Rename the parameter to base_ref so the ref and the counts each keep a single name and type, matching the no-reassignment style used elsewhere; behavior is unchanged. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Mateo Wang
|
2688f81df8
|
feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051)
* feat(cloudflare): add current Workers AI text-generation models to the cost map The Cloudflare Workers AI list in the model cost map was badly stale, holding only 4 ancient entries (llama-2-7b, mistral-7b-v0.1, codellama). This adds the 26 current text-generation models from Cloudflare's live /ai/models/search?task=Text Generation catalog (GLM 5.2, gpt-oss-120b/20b, llama 3.x/4, qwen3, deepseek-r1-distill, kimi, nemotron, and more), with pricing derived from the catalog's per-million USD rates, context windows, supports_function_calling, supports_reasoning, and cache_read_input_token_cost where Cloudflare publishes cached-input pricing. The entries are merged identically into both the root model_prices_and_context_window.json and the bundled litellm/model_prices_and_context_window_backup.json so the two maps stay in sync. A regression test pins the new entries and guards against the two files drifting for the cloudflare namespace. * fix(cloudflare): flag llama-3.2-11b-vision as vision-capable and tidy pricing precision llama-3.2-11b-vision-instruct is multimodal but was added without supports_vision, so LiteLLM capability checks would not surface it for image inputs. This sets supports_vision: true in both the root and backup cost maps It also rounds the newly added Workers AI per-token prices to their intended decimal values, dropping floating-point division artifacts like 4.839999999999999e-07 in favor of 4.84e-07, applied identically to both files so the cloudflare namespace stays in sync * test(cloudflare): pin Workers AI models against the local cost map test_glm_5_2_entry_is_present_and_well_formed and test_additional_current_models_are_present read litellm.model_cost, which defaults to the remote map fetched from main and therefore does not yet carry the entries this PR adds, so in the misc unit shard that lookup raised KeyError. The tests now load the bundled local map through an autouse fixture (LITELLM_LOCAL_MODEL_COST_MAP plus get_model_cost_map), matching the pattern used elsewhere in the suite, so they assert against the data this PR actually ships It also adds a regression test that llama-3.2-11b-vision-instruct carries supports_vision, and skips the root/backup comparison when the root file is absent so the suite stays green on wheel installs |
||
|
milan-berri
|
02f63d20bb
|
fix(router): guard num_retries=None in async_function_with_retries (#30036)
When num_retries reaches async_function_with_retries as None - e.g. a caller
passes num_retries=None explicitly (dict.get() does not fall back on an
existing None value), an auto_router/complexity_router path does not propagate
it, or Router.update_settings(num_retries=None) is used - the comparison
`if num_retries > 0:` raised:
TypeError: '>' not supported between instances of 'NoneType' and 'int'
This only surfaced when the underlying call failed with a retryable error
(rate limit / connection / 5xx), so the real upstream error was masked by a
confusing TypeError.
Normalise an explicit num_retries=None to the router default in
_update_kwargs_before_fallbacks (falling back to 0 when the router default is
itself None, and preserving an explicit 0), and keep the matching guard at the
single pop site in async_function_with_retries as the safety net for paths that
bypass the setter. Adds regression tests to
test_router_per_deployment_num_retries.py.
Relates to #23316, #25889, #23699, #28126
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
|
||
|
tin-berri
|
7020e1e5f7
|
feat(mcp): add resolve_credentials dispatch skeleton (#31056)
PR3 of the MCP v2 outbound-credential migration, stacked on the typed vocabulary. Adds resolver.py: UpstreamCredentialProvider.resolve_credentials dispatches on the declared AuthConfig variant with one arm per mode, a wildcard-free match plus an assert_never tail so a missing arm fails basedpyright's exhaustiveness gate. Every arm is a not_implemented stub returning a typed CredError; each mode's real body and seam land in follow-up PRs. Pure v2, no v1 imports, nothing wired onto a request path. |
||
|
Krrish Dholakia
|
c8a9618afd
|
feat: add opensandbox sandbox provider (#31024)
* feat: add opensandbox sandbox provider * fix: harden opensandbox sandbox startup * fix: address opensandbox review feedback * fix: address opensandbox sandbox review feedback * fix: address sandbox parser nits * fix(ci): clear opensandbox gates * fix(review): require opensandbox api base * chore(ci): rerun pass-through check |
||
|
Sameer Kankute
|
e73cbfb026
|
fix(realtime): post-tool-call function_response id omission (#30446) | ||
|
Sameer Kankute
|
80c5a84871
|
chore: litellm oss staging (#30968)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in
|
||
|
Mateo Wang
|
ec268b0d18
|
refactor(completion): extract provider dispatch into typed helpers so basedpyright can analyze it (#30813) | ||
|
tin-berri
|
69b0dd2da0
|
fix(mcp): stop auth failures on the /mcp path surfacing as cancelled tool calls (#31011)
user_api_key_auth raises ProxyException, not HTTPException, on an auth failure. The streamable-HTTP and SSE MCP handlers only re-raised HTTPException to preserve status and headers, so a ProxyException fell through to the catch-all and was flattened to a generic 500, dropping the real status (for example 401) and any WWW-Authenticate challenge. MCP clients render a 500 on the JSON-RPC POST as a cancelled or terminated session, and an OAuth client never receives the 401 it needs to re-authenticate. Because auth runs before server routing, one rejected credential fails every targeted server at once. Map ProxyException back to its real status and headers in both handlers (handle_streamable_http_mcp, handle_sse_mcp) via a small _proxy_exception_to_http_exception helper inserted before the generic except Exception. A genuine auth failure now returns its real status; a key sent without the documented Bearer prefix gets a clear 401 telling the caller to fix the header rather than a cancelled session. Regression tests assert that a ProxyException(401) raised during auth propagates as a 401 with WWW-Authenticate from both the streamable-HTTP and SSE handlers, and unit-test the converter for the 401/403/non-numeric-code cases. |
||
|
mubashir1osmani
|
21cd1d1a4f
|
fix(router): isolate all per-deployment pricing overrides from sibling deployments (#31021)
* fix(router): isolate all per-deployment pricing overrides from sibling deployments CustomPricingLiteLLMParams is the authoritative set of per-deployment pricing fields, used to strip overrides from the shared backend-alias key so one deployment cannot pollute a sibling that shares the same backend model. It had drifted from ModelInfoBase: tiered and per-unit cost fields such as input_cost_per_token_above_272k_tokens, cache_read_input_token_cost_above_*, output_vector_size, ocr_cost_per_*, and the regional uplift multipliers were absent, so a deployment overriding any of them leaked the override into litellm.model_cost under the shared key and every sibling read the wrong rate via /model/info (LIT-3897). Add the missing fields so the denylist covers every ModelInfoBase pricing field, and guard against future drift with a test asserting the two stay in sync, plus a regression test that a tiered override stays isolated to its own deployment model_id key. * chore(ui): regenerate schema.d.ts for custom pricing fields |
||
|
tin-berri
|
c18a870746
|
feat(mcp): scaffold outbound_credentials package with typed Result (#31047)
* feat(mcp): scaffold outbound_credentials package with typed Result PR1 of the MCP v2 outbound-credential migration. Adds the litellm/proxy/_experimental/mcp_server/outbound_credentials/ subpackage with a hand-rolled Ok | Error Result union (pure stdlib + typing_extensions, no new dependency) and its package surface. Nothing imports this on a live request path yet, so production behavior is unchanged; later PRs add the typed config vocabulary, the resolve_credentials dispatch, and the v1 graft. * feat(mcp): add outbound_credentials typed vocabulary (#31049) PR2 of the MCP v2 outbound-credential migration, stacked on the result.py scaffolding. Adds types.py (the AuthConfig discriminated union over seven frozen per-mode configs, CredError as an expression @tagged_union, Subject, ServerSpec, and the parse_auth_spec_kind boundary parser) and httpx_auth.py (NoOpAuth, StaticHeaderAuth). Pulls in expression>=5.6.0,<6.0 on the proxy extra for the tagged union. Construction-time tests prove illegal mode/field combinations are rejected. Nothing is wired onto a request path yet; the resolver lands next. |
||
|
Yassin Kortam
|
a9de75b1f7
|
fix(realtime): stop revalidating realtime events at the logging boundary (#31054)
Realtime websocket sessions emit events outside the OpenAIRealtimeEvents union (e.g. rate_limits.updated, response.function_call_arguments.delta, surfaced when logged_real_time_event_types="*"). Building LiteLLMRealtimeStreamLoggingObject revalidated every stored event against the 16-member union, producing thousands of ValidationErrors per session (12,670 for a ~281-event session). That synchronous work blocked the asyncio event loop, degrading realtime time-to-first-audio and dial latency and tripping readiness probes, and the raised error discarded the session usage so no cost was tracked. Type results as SkipValidation[OpenAIRealtimeStreamList] and serialize the events verbatim, so already-formed event dicts are not revalidated. The flood drops from 12,670 errors to 0 and the combined usage survives to the cost calculator. Resolves LIT-3919 Resolves LIT-3920 |
||
|
yuneng-jiang
|
6f6aec2930
|
fix(proxy): serialize team budget_limits to JSON in jsonify_team_object (#31045)
POST /team/new with any budget_limits returned 500 because jsonify_team_object serialized members_with_roles but left budget_limits as a raw Python list, which Prisma's Json column rejects. /team/update and /key/generate worked only because each json.dumps the windows itself. Serialize budget_limits in the shared helper, guarded by isinstance(list) so the pre-serialized /team/update path is unaffected. |
||
|
Yassin Kortam
|
b24b964e04
|
fix(passthrough,streaming): recover cost on interrupted and agentic Anthropic streams (#31035)
Streaming and pass-through requests could be logged with $0 cost or dropped from SpendLogs entirely while the upstream provider still billed every token. This closes the leak paths not already covered by #30160, #30787 and #30788. - Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and async). Large agentic tool-use / thinking streams can make assembly re-raise as APIError from inside the except-StopIteration handler, where the sibling except does not catch it, so it escaped __next__/__anext__ and dropped the request; recover best-effort usage from the raw chunks instead - Add a usage-only fallback for Anthropic streaming pass-through: when stream_chunk_builder returns None or raises, rebuild usage from the message_start / message_delta SSE events via AnthropicConfig.calculate_usage so cache, web-search and geo tokens are priced instead of left at $0 - Decode buffered pass-through bytes with errors="replace" so a stream cut mid-multibyte-sequence still logs the usage events already received - Record response_cost into model_call_details on the pass-through success path (it is read from there, not from kwargs), matching the gemini/cohere/openai handlers - Name the key (alias + masked key) in the virtual-key BudgetExceededError so operators don't have to reverse-map spend back to a key |
||
|
ryan-crabbe-berri
|
3615049071
|
feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools (#31031)
* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools GET /v1/mcp/tools returns the MCP tools available to the calling key, the same data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"]) got a 403 listing tools one way but not the other. Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no management write counterpart, so it does not need the method-aware carve-out used for server discovery. * test(proxy): parametrize MCP inference route check over the full endpoint set |
||
|
ryan-crabbe-berri
|
26da56fbb6
|
feat(ui): add Amazon Bedrock Mantle to the Add Model provider dropdown (#31034)
The Add Model provider dropdown is driven by provider_create_fields.json (served at /public/providers/fields), and Bedrock Mantle had no entry, so it could not be selected even though the backend provider, its models, and the UI enum/logo mappings already existed. Add a bedrock_mantle entry exposing the credential fields the provider actually honors: an optional bearer api_key for BYOK, the AWS SigV4 chain, a region, and an api_base override. Selecting it now populates the bedrock_mantle models from the cost map via the existing getProviderModels filter. Also resolve the provider logo when getProviderLogoAndName is given the enum key (e.g. BedrockMantle) rather than the slug, which the dropdown passes; previously only slugs that lowercase-matched their key (like bedrock) resolved a logo. |
||
|
ryan-crabbe-berri
|
19a29e0579
|
feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel (#31029)
* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel
A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.
* refactor(ui): centralize no-mcp-servers sentinel in a shared constant
The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.
* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes
Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.
|
||
|
yucheng-berri
|
4ef7d0815b
|
fix(bedrock): only expand config-sourced AWS credential references (#30867)
AWS auth parameters in the Bedrock and SageMaker path could be expanded against the process environment when credentials were built. Config-sourced references are already expanded at load time, so restrict expansion to that path: a reference still present at request time is treated as caller-supplied input and is left as-is, and the web-identity helper rejects environment-variable references before resolving the token. Also rework the ambient AWS_* fallback as a single pass that pairs each value with its own env-var name, fixing a latent index misalignment that left AWS_EXTERNAL_ID unresolved. Adds regression tests covering the resolution behavior. |
||
|
ryan-crabbe-berri
|
ce4111b800
|
fix(proxy): scope team BYOK models by key team_id in /model/info (#31009)
GET /model/info returned an empty list for a team key whose team only has team-scoped BYOK deployments, even though /v1/models and a master key both returned them. _get_caller_byok_team_scope resolved the caller's allowed teams only from user_api_key_dict.user_id and the bound user's team memberships. A team or service key has user_id=None, so the helper returned an empty set and _byok_row_outside_caller_teams then dropped every team BYOK row This includes the key's own team_id in the allowed-team set across every non-admin branch, since a team key is authoritatively scoped to its team regardless of whether a bound user is resolvable or a formal member of that team Completes the work in #30025, which aligned /v1/model/info with router deployments but missed the team-key case in the scope helper it introduced |
||
|
Mateo Wang
|
8bc18388e3
|
fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body (#30985)
* fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body * test: cover bedrock key-tag litellm_metadata pre-seed in common_checks Add a regression test asserting key-level tags on a bedrock passthrough request land in litellm_metadata and never leak into the provider-facing metadata field, which closes the codecov/patch gap on the auth_checks pre-seed line. Also drop the now-stale comment that hardcoded metadata["headers"]; the headers are written under whichever metadata field _get_metadata_variable_name selects. * refactor(auth): pre-seed litellm_metadata from LITELLM_METADATA_ROUTES The auth-time pre-seed in common_checks hardcoded "bedrock", so any other route later added to LITELLM_METADATA_ROUTES would reintroduce GH#30629 (key tags leaking into the provider-facing metadata field) without a matching update here. Key off the shared constant instead, and extend the regression test to cover a non-bedrock metadata route so the route-agnostic behavior is locked in. * fix(auth): pre-seed litellm_metadata before header-tag merge apply_client_tag_policy_pre_auth runs in user_api_key_auth.py before common_checks, so it resolved get_metadata_variable_name_from_kwargs to 'metadata' (litellm_metadata was not yet present). common_checks then pre-seeded litellm_metadata on LITELLM_METADATA_ROUTES, after which apply_key_tags_pre_auth and _tag_max_budget_check both targeted litellm_metadata, leaving header tags stranded in metadata and invisible to per-tag budget enforcement on Bedrock and other matching routes. Extract the pre-seed into LiteLLMProxyRequestSetup.pre_seed_litellm_metadata_for_route and invoke it before apply_client_tag_policy_pre_auth so all tag merges and the budget-check read agree on the same metadata key. * test(auth): guard early litellm_metadata pre-seed wiring for header tags Bugbot's autofix added a pre-seed of litellm_metadata in _run_centralized_common_checks before apply_client_tag_policy_pre_auth, so x-litellm-tags header tags land in litellm_metadata and stay visible to _tag_max_budget_check on LITELLM_METADATA_ROUTES. Its test replayed that call order in the test body, so removing the production call site still passed. Add a wiring-level regression that drives the real _run_centralized_common_checks and asserts header tags land in litellm_metadata (not metadata) for bedrock and /v1/messages. Dropping the pre-seed call site now fails the test. --------- Co-authored-by: Zang Peiyu <166481866+factnn@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Yassin Kortam
|
e70f7e2d7a
|
fix(ui): resolve user_id to email in Spend Per User usage chart (#30992)
The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and default_user_id) instead of emails. /user/daily/activity passed entity_metadata_field=None, so every user entity in the breakdown carried empty metadata; the chart could only fall back to the user_id. The frontend resolved labels from a separately paginated user list, so any spender not on a loaded page showed as a UUID. Resolve the email/alias for the user_ids actually on the page (mirroring how api key metadata is already resolved) and attach it to the entity metadata, so the chart labels each spender with their email and falls back to the UUID only when no email is on file. get_daily_activity gains an optional resolve_entity_metadata hook so the user endpoint can do this page-scoped lookup without loading the whole user table. Resolves LIT-3889 |
||
|
Yassin Kortam
|
fd377eece8
|
feat(scim): ingest enterprise extension attributes into user metadata (#30893)
Map the SCIM enterprise extension block (urn:ietf:params:scim:schemas:extension:enterprise:2.0:User) onto SCIMUser so create and PUT persist employeeNumber, costCenter, organization, division, department, and manager into LiteLLM_UserTable.metadata under scim_enterprise, and round-trip them back out on read. This lets financial reporting group spend by fields like cost center and department. The enterprise block holds directory-only HR attributes, so it is kept out of the generic user management responses (/user/info, /v2/user/info, and /user/list), which non-proxy-admin callers such as team and org admins can use to read other users. The data still lands in metadata for reporting and still round-trips through the SCIM read endpoints, which build their response from the user row directly. Resolves LIT-3617 |
||
|
Yassin Kortam
|
2be0183cc7
|
feat(scim): drive global proxy role from a SCIM admin group (#30895)
Adds an optional litellm_settings.scim_admin_group. When configured, the global proxy role is recomputed from a user's resulting groups on every SCIM write that can change membership: user create/PUT/PATCH and group create/PUT/PATCH/DELETE, plus the existing-email upsert path. Membership in the admin group grants PROXY_ADMIN and its absence demotes to the non-admin default, enabling just-in-time elevation and automatic demotion without a re-login. When the setting is unset the role is never touched, so current behavior is preserved and a misconfigured IdP can never unexpectedly grant admin. |
||
|
yucheng-berri
|
1667b8f740
|
fix: reject model_list in proxy body and gate advisor client credentials (#30585)
* fix: validate proxy request body and nested fields
Ensure caller-supplied request fields cannot override server-side deployment
configuration, and apply request-body validation consistently to nested
structures. Adjusts router kwarg handling and client-side credential handling
for base-url overrides
* test: cover router strip ordering and advisor clientside credential gate
* fix: clear deployment credentials on client base-url override
When a request overrides api_base/base_url, recompute the deployment's
litellm_params (clearing the deployment's own api_key) and drop the cached
client built for the original endpoint, so the deployment credential is not
reused for the client-supplied endpoint. Adds regression tests that assert the
credentials actually forwarded to litellm.completion/acompletion.
* fix(proxy): require api_key alongside api_base override
A request that overrides api_base/base_url but supplies no api_key still
left the proxy carrying a server credential: once the override clears the
banned-param opt-in, the provider re-resolves a key from the environment
(api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in
main.py) and forwards it to the caller-controlled URL. Popping the
deployment api_key only changed which server key leaked.
Gate is_request_body_safe so a permitted api_base/base_url override must
also carry a non-empty caller api_key; reject otherwise. The env
resolution in main.py is left as the provider boundary.
* fix(proxy): extend request-body banlist with five additional credential and session targeting fields
Yuneng's review found five deployment-owned request-body params still missing
from the denylist and the router strip set. Each lets a caller reach the
operator's provider credentials or retarget the outbound request:
aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region
retarget the OCI request, litellm_credential_name selects any server-loaded
credential by name with no ownership check, and runtimeSessionId resumes a
Bedrock AgentCore runtime session (AWS does not enforce session-to-user
mapping, so this is a cross-tenant session-resume vector).
Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to
_DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and
SDK direct calls are unaffected: the banlist gates the request body only, and
the router strip drops caller kwargs, never deployment["litellm_params"].
* test: rename arbitrary canary values in security tests to neutral placeholders
* fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip
P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS
(litellm_embedding_config, extra_body) for the banned-param check but not for
the api_key co-presence check, so a base override smuggled into one of those
nested dicts cleared the client-side-credentials opt-in without a paired
api_key and let the provider re-resolve a server credential from the
environment. Run _check_base_override_has_api_key on each nested config dict
too, so the requirement applies wherever a base override is permitted.
P1-B: the deployment-owned credential strip in the Router runs unconditionally
on every _completion/_acompletion, which is security-correct but silently
drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a
single warning (key names only, never values) when the strip removes a
non-empty value, so the backwards-incompatible behavior is visible without
gating the strip on a context flag that does not exist.
* fix(proxy): apply api_key co-presence to tool-entry base override
is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and
tools[]); the previous commit extended the api_key co-presence rule to root
and nested config dicts but not to tool entries. With
allow_client_side_credentials enabled, a tool entry carrying api_base/base_url
and no paired api_key cleared the gate, letting a provider interceptor fall
back to a server-side credential for a caller-controlled URL. Add the same
_check_base_override_has_api_key call to each tool dict and its nested function
dict, mirroring the symmetry already applied to the nested config keys. The
rule is unchanged: api_key must live in the same dict as the base override it
accompanies.
* test(proxy/auth): require paired api_key under extra_body opt-in
* fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running
* fix(advisor): narrow proxy-import guard to ImportError-family
* fix(router): gate api_key clear on base override behind litellm.proxy_is_running
* test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture
* style: use built-in generics in PR-added type annotations
* revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate
* revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes
* style: black-format advisor orchestration test
|
||
|
Yassin Kortam
|
1322ad7224
|
perf(otel): resolve LITELLM_OTEL_V2 flag once instead of rebuilding settings per call (#30989)
is_otel_v2_enabled() constructed a pydantic-settings model (_OTelV2Flag) on every call, which re-scans the process environment and costs ~28us. The flag is read multiple times along the proxy request hot path (auth, logging-callback setup, proxy_server), so the cost compounded into a measurable per-request CPU overhead and a throughput regression visible from v1.87.3 onward. The flag is a process-level setting that is fixed at startup, so resolve it once with lru_cache. Caching it alone restores throughput to the pre-regression baseline in load tests. Tests that toggle the env now call cache_clear(). |
||
|
tin-berri
|
963816c00e
|
fix(mcp): stop exposing MCP server URLs on the AI Hub and public hub API (#30902)
The AI Hub MCP Hub listed each MCP server's upstream URL in a table column and in the server Details modal, on both the authenticated dashboard and the public hub. The unauthenticated GET /public/mcp_hub endpoint also returned the url field via MCPPublicServer, so the upstream address was readable by any client even with the column gone. These surfaces are for end users discovering available servers, so the gateway-internal endpoint should not be exposed there. Drop the URL column from both MCP Hub tables and the URL field from both detail modals, and remove url from MCPPublicServer so /public/mcp_hub no longer serialises it; schema.d.ts is updated to match. The public hub MCPServerData interface no longer declares url since the response omits it. Admin surfaces that configure the endpoint (the MCP server management page, the submissions review tab, the make-public form) and the authenticated /v1/mcp/server endpoint are untouched. publicMCPHubColumns is lifted to a module-level export so both hub column sets get a mutation-killing regression test, public_model_hub.test.tsx covers the details modal hiding the url, and test_public_endpoints.py asserts /public/mcp_hub never returns url even when the server has one. |
||
|
tin-berri
|
f2b823ce1a
|
test(interactions): drop role from Interaction output fields to match Google spec (#30986)
Google moved the role field off the top-level Interaction response schema onto the per-turn Turn schema, so the OpenAPI compliance canary test_interaction_response_fields started failing on main with "Output field 'role' not in spec". role on Turn is already asserted by test_turn_schema, so dropping it from the Interaction output_fields list realigns the test with the live spec without losing coverage. |
||
|
Mateo Wang
|
6437b812be
|
refactor(streaming): extract chunk_creator dispatch so basedpyright can analyze it (#30793) | ||
|
Mateo Wang
|
1ea91bf296
|
refactor(exceptions): extract exception_type provider dispatch so basedpyright can analyze it (#30802) | ||
|
Krrish Dholakia
|
84c1414aef
|
feat(sandbox): code interpreter interceptor on the Responses API (#30905)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(sandbox): code interpreter interceptor on the Responses API
Route OpenAI's code interpreter to a configured sandbox (e2b) instead of
OpenAI's container, with no client change. A client calls /v1/responses with
a code_interpreter tool; the interceptor converts it to a function tool so the
model emits the code, runs that code in the sandbox via the phase 1 primitive,
feeds the result back, and lets the agentic loop continue.
Reuses the existing agentic-loop hooks (no new hook methods). The anthropic
agentic caller _call_agentic_completion_hooks gains an api_surface argument and
a responses execute path (_execute_responses_agentic_plan re-calls aresponses);
the responses handler invokes it after transforming the response. Web search and
compression interceptors are untouched.
Adds an api_base passthrough to the sandbox SDK and a sandbox_tools registry the
proxy parses, so the interceptor resolves a named tool to provider/key/base.
v0 limitation: no file upload or download yet; stdout and inline results flow
back, attaching input files and downloading produced files do not.
* feat(sandbox): re-inject code_interpreter_call so the response matches OpenAI
The native OpenAI Responses code interpreter returns a code_interpreter_call
output item (id, type, status, code, container_id, outputs) alongside the
message. The interceptor now re-injects an equivalent item via
async_post_agentic_loop_response_hook so a client gets the same response shape
whether the code ran in OpenAI's container or the sandbox: build_plan records
the executed code and the container id per call, and the post hook inserts the
code_interpreter_call before the message in the final response output.
* feat(sandbox): support streaming for the code interpreter interceptor
A stream:true /v1/responses request with code_interpreter previously broke,
because the agentic loop only runs on the non-streaming responses path. The
interceptor now forces stream=False in the pre-call hook (so the loop runs in
the sandbox) and the responses handler wraps the completed response back into a
synthetic stream via MockResponsesAPIStreamingIterator, so the caller still gets
SSE. The follow-up call and nested wrapping are guarded by stripping the
converted-stream flag from the follow-up request and only wrapping at the
outermost call (agentic loop depth 0).
* fix(lint): use builtin generics in code interpreter interceptor to satisfy UP006 budget
* fix(code-interpreter): gate sandbox execution, delete sandboxes, harden registry
Gate the agentic loop on a server-set interception marker and re-check
provider scope so an authenticated caller cannot trigger sandbox code
execution by naming their own function tool litellm_code_execution; the
marker is stripped from client requests at the proxy boundary and only
set when the pre-call hook actually converts a native code_interpreter
tool. Delete the sandbox once the final response is assembled instead of
leaking it until its own timeout, and prune expired cache entries by
deleting their containers too. Resolve sandbox params once at create time
and reuse them for run and delete. Clear the sandbox-tool registry before
re-registering so stale tools do not survive a config reload.
* fix(lint): use PEP 604 X | None unions to satisfy UP045 budget
* test(code-interpreter): cover execution-error and unparseable-argument tool-call paths
* fix(code-interpreter): rewrite forced code_interpreter tool_choice to the function tool
* test(sandbox): cover sandbox-tool registry resolution, reload clearing, and secret lookup
* test(code-interpreter): cover dict-shaped responses and object-attribute tool-call detection
* fix(proxy): strip client-supplied _code_interpreter_interception_converted_stream
A client could inject the converted-stream marker to force the completed
response to be re-wrapped as a synthetic SSE stream it never requested.
Add it to the untrusted root control fields alongside the other agentic
loop markers so the proxy strips it at the request boundary.
* fix(code-interpreter): isolate sandboxes by server-minted key and clear registry on tool removal
Key the per-request sandbox cache on a server-minted random token instead
of the caller-controlled litellm_call_id (sourced from the x-litellm-call-id
header). Two concurrent requests that send a colliding call id can no longer
share a sandbox container and read each other's code or files. The token is
minted in the pre-call hook when interception activates, stripped from client
requests at the proxy boundary, and survives the server-driven followups so a
single request still reuses one sandbox across the agentic loop.
Register sandbox tools unconditionally with an empty-list fallback so a config
reload that removes sandbox_tools clears the previously registered credentials
instead of leaving them resolvable in the process.
* refactor(sandbox): swap the tool registry atomically on reload
Build the new registry and rebind it in one assignment instead of clearing
then repopulating in place, so a concurrent resolve_sandbox_tool can never
observe a transiently empty or half-populated registry during a config
reload. clear_sandbox_tools now delegates to register_sandbox_tools([]).
* fix(code-interpreter): cap caller loop limit and emit OpenAI-shaped outputs
Strip max_agentic_loops at the proxy request boundary so an authenticated
caller cannot raise the agentic-loop ceiling to drive many upstream model
calls and sandbox executions from a single request; the loop stays bounded
by the server default.
Populate the re-injected code_interpreter_call.outputs with an OpenAI-shaped
logs array ([{"type": "logs", "logs": stdout}], or [] when there is no
stdout) instead of None, so clients that iterate over outputs or validate the
response through the OpenAI SDK's Pydantic model do not break.
|
||
|
Krrish Dholakia
|
accbd7e587
|
feat: litellm plugin architecture v2 (#30688)
* feat: plugin architecture — toggle between AI Gateway and external plugins
Adds a generic plugin system so any external service can register with
litellm and appear as a mode in the UI alongside the AI Gateway.
Backend (litellm/proxy/plugin_routes.py — new):
- GET /api/plugins: returns registered plugins from config; returns
plugin_key only to authenticated requests
- ANY /plugin-proxy/{name}/{path}: reverse proxies API calls to plugin
Config:
general_settings:
plugins:
- name: my-plugin
display_name: My Plugin
url: https://my-plugin.example.com
plugin_key: sk-... # plugin auth key, passed to iframe
UI:
- PluginModeContext.tsx: fetches /api/plugins, persists mode to localStorage
- leftnav.tsx: mode switcher dropdown at top of sidebar; plugin mode shows
plugin-specific nav items
- layout.tsx: renders iframe to plugin URL in plugin mode; passes plugin_key
as ?token= for auto sign-in
Plugin contract: expose GET /api/plugin-manifest returning
{ name, display_name, nav_items[], capabilities[] }. No litellm changes
needed to add new plugins — config only.
Reference implementation: LiteLLM-Labs/litellm-agent-control-plane
* feat: add Plugins tab to Admin Settings UI
Allows admins to add/edit/delete plugin registrations directly in the
litellm UI under Admin Settings > Plugins, instead of editing config.yaml.
Uses existing /config/field/update API to persist to general_settings.plugins.
Each plugin entry has: name (identifier), display_name, url, plugin_key.
* fix(ci): black, prettier, eslint, async-client violations
- Black: format plugin_routes.py and proxy_server.py
- Prettier: format PluginModeContext.tsx and PluginSettings.tsx
- ESLint: replace raw fetch() with createApiClient in PluginModeContext
- ESLint: use lazy useState initializer to read localStorage instead of
calling setModeState inside useEffect (react-hooks/set-state-in-effect)
- code-quality: replace httpx.AsyncClient per-request with
get_async_httpx_client() shared client (avoids +500ms overhead)
* fix(ci): schema.d.ts regen, Black proxy_server.py, ApiClientConfig fix
- Regenerate schema.d.ts for new /api/plugins routes
- Re-run Black 26.3.1 on proxy_server.py (matches CI version)
- Fix PluginModeContext: createApiClient requires getBaseUrl field
* fix: security hardening + CI fixes
Security (Greptile 1/5 → addressing all 3 findings):
- plugin_routes.py: add Depends(user_api_key_auth) to both /api/plugins
and /plugin-proxy/{name}/{path} — was an unauthenticated open relay
- plugin_routes.py: /api/plugins now returns plugin_key only to callers
with a valid litellm token (enforced by user_api_key_auth), not just
any header presence
- layout.tsx: replace ?token= URL param with postMessage(targetOrigin)
— token no longer exposed in browser history / logs / Referer headers
CI:
- backend/routes/allowlist.py: add /api/plugins and /plugin-proxy/ to
fix test_gateway_plus_backend_covers_full_app
- schema.d.ts: regenerated with enterprise routes included
- Black + Prettier formatting
* fix: regenerate schema.d.ts with enterprise routes included
Install litellm-enterprise workspace member before gen:api so audit and
other enterprise routes appear in the generated types, matching what CI
produces with uv sync --extra proxy.
* fix: exclude plugin routes from OpenAPI schema, restore upstream schema.d.ts
Both /api/plugins and /plugin-proxy/ are internal infrastructure routes,
not part of the public litellm API surface. Marking include_in_schema=False
prevents Python-version-dependent schema diffs from breaking the schema
sync check across different environments.
* fix: schema.d.ts - passing schema base + exact plugin route types from openapi-typescript
Use the CI-correct schema from a recently passing branch as base, then
inject plugin route entries (paths + operations) generated by
openapi-typescript from the plugin routes' OpenAPI spec. This avoids
Python-version-dependent formatting differences that made local gen:api
produce incorrect output.
* fix: schema.d.ts - insert plugin ops at correct route registration position
Plugin operations belong after delete_memory_v1_memory__key__delete
(memory_router is included immediately before plugin_router in proxy_server.py),
not after list_organization which is alphabetically but not registration-order.
* fix: schema.d.ts - correct op positions from hunk analysis
list_plugins_api_plugins_get goes after event_logging_batch op (hunk 1: line 33583).
plugin_proxy ops go after create_policy_policies_post (hunk 2: line 44634).
Previous location after delete_memory_v1_memory__key__delete was wrong.
* fix: schema.d.ts - proxy ops go before create_policy (after otel_spans)
* fix(security): restrict plugin_key to proxy_admin role only
Veria finding: plugin_key was returned to any authenticated caller.
Now only proxy_admin users receive plugin credentials in /api/plugins
response — regular internal users see plugin name/url but not the key.
* fix: update schema.d.ts docstring for list_plugins
* fix: clear plugin registry on config reload (Greptile medium)
register_plugins_from_config now replaces the registry instead of
merging, so plugins removed from config are unreachable immediately
without requiring a process restart.
* fix(security): encrypted token exchange for plugin iframe — no raw litellm credential exposure
The dashboard was sending the user's litellm bearer token to the plugin
iframe via postMessage, allowing a compromised plugin to act as that user.
Fix:
- GET /api/plugins/auth-token: proxy encrypts caller token with Fernet
keyed from LITELLM_SALT_KEY, returns ciphertext only
- UI postMessages the ciphertext (not raw token) to the iframe
- Plugin decrypts server-side with same LITELLM_SALT_KEY via POST /api/plugin-auth
- Raw litellm credential never leaves the proxy in plaintext
Additional hardening already in place:
- /plugin-proxy/* strips Authorization header, injects plugin_key instead
- plugin_key only returned to proxy_admin role via /api/plugins
- Plugin registry cleared (not merged) on config reload
Adds docs/plugin_architecture.md with plugin integration guide.
* fix(code-quality): use get_async_httpx_client in plugin_proxy
* fix: add /api/plugins/auth-token to schema.d.ts
* fix: use apiClient for auth-token fetch, copy correct layout.tsx and PluginModeContext
- Replace raw fetch() with createApiClient (fixes no-restricted-syntax ESLint rule)
- Copy correct layout.tsx with encrypted token + postMessage approach
- Copy correct PluginModeContext.tsx with accessToken prop injection
- Update schema.d.ts with auth-token path and operation entries
* fix: add plugin_auth_token operation to schema.d.ts
* fix(security): strip cookie/set-cookie + fix compressed response headers
Veria High: cookie header was forwarded to plugin backends allowing
capture of litellm JWT session cookies. Strip cookie on requests.
Strip set-cookie from responses so plugins cannot overwrite litellm
session cookies.
Greptile P1: httpx decompresses responses but resp.headers still
contained Content-Encoding/Transfer-Encoding/Content-Length from the
wire. Forwarding these caused double-decompression and length errors.
Now filtered via _RESPONSE_STRIP before returning to the browser.
* fix: update plugin_key help text — no more ?token= reference
* fix(security): disable follow_redirects to prevent SSRF
follow_redirects=True allowed a plugin backend to return a 3xx to an
internal URL, causing the proxy to fetch that internal service and relay
the response. Disabled: clients handle their own redirects.
* fix: forward user identity headers to plugin to address confused deputy
Plugins receive X-LiteLLM-User-Id and X-LiteLLM-User-Role so they can
enforce their own per-user access control before acting on requests that
arrive with the shared plugin_key credential.
* fix(security): restrict /plugin-proxy/* to proxy_admin role
Closes the confused deputy gap: regular users could invoke any plugin
endpoint using the shared plugin_key as a bearer credential. Now only
proxy_admin callers can use the plugin proxy route.
Plugin UIs communicate with the plugin service directly via the iframe
(using the encrypted token exchange); this proxy route is for
administrative/server-to-server access only.
* fix: update schema.d.ts for admin-only proxy route docstring
* fix(bug): use PassThroughEndpoint instead of None for get_async_httpx_client
get_async_httpx_client(llm_provider=None) raises TypeError — the function
concatenates the provider string and None is not a str. Use
httpxSpecialProvider.PassThroughEndpoint, the enum value used by other
internal proxy pass-through routes.
* fix(security): add 30s TTL to encrypted plugin auth tokens
Veria medium: encrypted tokens had no expiry, allowing indefinite replay.
Fernet embeds a timestamp; decrypt_token now passes ttl=30 so tokens
older than 30 seconds are rejected even with a valid HMAC.
Plugin's /api/plugin-auth must call litellm within 30s of the iframe
receiving the postMessage — normal browser behavior, tight enough to
close the replay window.
* feat(ui): topnav plugin switcher, embed plugins at their root
Builds on the plugin architecture already on this branch (encrypted-token
postMessage handshake, /api/plugins, PluginSettings) and removes the parts of the
embed that assumed a specific plugin's shape.
The mode switcher moves out of the sidebar into the topnav and lists AI Gateway
plus each registered plugin by its display_name. Selecting a plugin hides
litellm's sidebar entirely and renders the plugin full-bleed at its root url; the
plugin draws its own navigation inside the iframe. This drops the hardcoded
"Agent Control Plane" label and the hardcoded Sessions/Agents/Routines/... nav
groups (agentControlPlaneMenuGroups / acpPagePaths) that only matched the agent
platform and 404'd for a plugin that serves only / (e.g. the chat UI). The
encrypted-token postMessage flow is unchanged.
Note: embedding at root means a plugin must route internally from /; plugins that
previously relied on the /sessions entrypoint should redirect from their root.
* fix(security): audience-scoped identity claim replaces litellm token
Veria: shared LITELLM_SALT_KEY with plugins + encrypting user bearer token
created delegation/impersonation risk.
Architecture change:
- /api/plugins/auth-token now issues a plugin-scoped identity CLAIM
{user_id, user_role, plugin, exp} encrypted with HMAC(LITELLM_SALT_KEY, plugin_name)
- Each plugin holds only its own HMAC-derived key; cannot forge claims for
other plugins or recover LITELLM_SALT_KEY
- Claim contains NO litellm bearer token — compromised plugin learns caller
identity only, cannot act as that user against the proxy
- 30s TTL enforced in both Fernet header and explicit exp field
- LAP /api/plugin-auth verifies claim, returns its own master key to browser
(LAP key never exposed without valid claim)
* fix(plugins): allow registering plugins from the admin UI
Adding a plugin in the UI POSTs general_settings.plugins to /config/field/update,
which rejected it with "Invalid field=plugins passed in." because `plugins` was
not a field on ConfigGeneralSettings. Add a typed PluginConfig model and a
`plugins` field so the update validates and persists.
The in-memory plugin registry only refreshed at startup, so a plugin added via
the UI did not appear in /api/plugins (the view switcher) until a restart. Refresh
the registry from the new general_settings whenever the plugins field is updated.
While here, type the registry as dict[str, PluginConfig] instead of raw dicts so
list_plugins and plugin_proxy access typed attributes.
Fix the Plugin Key field copy: it is optional and only used to authenticate
litellm's server-side reverse proxy to a plugin's own backend
(/plugin-proxy/<name>/*). It is not involved in iframe auth, which forwards the
user's litellm token. Plugins that use the forwarded token leave it blank.
* fix: regenerate schema.d.ts with PluginConfig type and updated auth-token endpoint
* fix: use CI-compatible schema base for plugin entries
* fix(plugins): load DB-persisted plugins on startup
Plugins added through the admin UI are saved to DB general_settings, but the
registry only initialised from the YAML config at boot, so UI-added plugins
disappeared from the view switcher after a restart (the Plugins table still
listed them since it reads the DB directly). Refresh the registry from the DB
general_settings when it is merged in at startup.
* fix: add PluginConfig schema, plugins field, fix list_plugins return type
* fix: correct PluginConfig and plugins field positions in schema
* fix: correct plugins field position in schema (after pass_through_endpoints)
* fix: update PluginConfig.plugin_key description to match _types.py source
* fix: move plugins field after pass_through_request_timeout (correct alphabetical position)
* fix: redact plugin_key in config/field/info response
Veria medium: proxy_admin_viewer could read plugin_key via
GET /config/field/info?field_name=plugins. Now plugin_key is
replaced with *** in the response regardless of caller role.
The credential is only usable server-side.
* fix(security): correct plugin docs salt-key guidance, drop iframe clipboard-read
Address the two open Veria findings on the plugin architecture.
The plugin docs told external services to decrypt the iframe auth payload
with the proxy's LITELLM_SALT_KEY directly. That is both insecure and wrong:
the running code derives a per-plugin key as HMAC-SHA256(LITELLM_SALT_KEY,
plugin_name) and ships only a short-lived identity claim with no litellm
bearer token. Sharing the master salt would let a compromised plugin decrypt
any litellm secret recovered from a dump or backup. Rewrite the doc to match
the implementation: the proxy computes the per-plugin key once and provisions
it as a dedicated secret, the plugin validates the claim's audience and 30s
TTL, and LITELLM_SALT_KEY never leaves the proxy. Also refresh the now-stale
module and UI comments that still described the old shared-key token flow.
Drop clipboard-read from the plugin iframe's allow attribute so an untrusted
plugin can no longer read the user's clipboard; clipboard-write is retained.
* fix(ci): modernize PluginConfig typing, refresh budget baselines via merge
* fix(plugins): close iframe auth race and empty-plugins mode fallback
Address the two open Greptile behavioral findings.
The iframe auth handshake only posted the encrypted claim on the iframe's
`load` event. When the auth-token fetch resolved after the iframe had already
loaded, that listener never fired again and the plugin never received the
claim. Send the claim immediately as well as on subsequent loads so both
orderings are covered.
The plugin mode fallback guarded on a non-empty plugins list, so removing all
plugins left a user stranded on a stale mode with a blank iframe instead of
returning to the AI Gateway. Track a loaded flag and fall back to ai-gateway
once plugins have loaded whenever the stored mode is no longer registered,
including the empty-list case.
Add a PluginModeContext regression test covering the empty-list fallback and
the still-registered path.
* chore: re-trigger CI (GH Actions missed the prior head; re-run flaky live-API suites)
* fix(plugins): scope iframe auth claim to the active plugin
The iframe auth-token fetch omitted plugin_name, so the proxy always issued a
claim encrypted under the default plugin's per-plugin key. For any other active
plugin the iframe received a claim it could not decrypt and sign-in silently
broke, and because the cached claim was posted to whichever plugin was mounted,
a compromised iframe could replay the default plugin's claim. The active
plugin's name was also missing from the fetch effect's dependencies, so
switching plugins never refreshed the claim.
Request the claim with the active plugin's name, re-fetch when the active
plugin changes, and only deliver a claim while it still matches the mounted
plugin so one plugin's claim is never replayed to another.
* fix(plugins): never overwrite a stored plugin_key with its redaction placeholder
/config/field/info redacts every plugin_key to "***", so an admin editing a
plugin in the settings UI posted that placeholder straight back and the update
handler persisted "***" as the real credential, permanently destroying the key.
Preserve the stored credential on update: a blank or redacted plugin_key now
sources the existing key from the saved config, only a real value replaces it,
and a placeholder with no stored key is dropped rather than written. The edit
modal also starts the key field blank so an untouched save keeps the current
key, with the field labelled accordingly.
* fix(security): sandbox proxied plugin responses on the dashboard origin
The /plugin-proxy reverse proxy returned the plugin's body and content-type on
the litellm dashboard origin, so a compromised plugin could serve an HTML/JS
document that a proxy_admin navigates to and have it execute with the admin's
session against same-origin management APIs.
Force every proxied response inert: set Content-Security-Policy: sandbox (opaque
origin, scripts disabled) and X-Content-Type-Options: nosniff, applied after the
plugin's own headers so they cannot be overridden. The header construction moves
to a pure helper with a unit test covering the sandbox enforcement and the
existing wire/cookie header stripping.
* fix(plugins): recover to ai-gateway when the plugins fetch fails
The loaded flag was only set on a successful /api/plugins response, so when the
fetch failed a user with a plugin mode stored in localStorage stayed on the
blank plugin placeholder with no switcher to escape. Mark loaded in a finally
so the stored mode still falls back to ai-gateway on failure, and add a
regression test for the failed-fetch path.
* fix(security): never return plugin_key from /api/plugins
The plugin list endpoint returned the plaintext plugin_key to proxy_admin
callers, and the dashboard fetches /api/plugins on every load into React state,
so the credential was exposed to DevTools, memory snapshots, and any same-origin
script. The browser never uses the key; the proxy injects it server-side from
the registry and admin key management runs through the redacted
/config/field/info path. Drop plugin_key from the response for every caller and
update the regression test to assert it is never returned.
* chore(ui): regenerate schema.d.ts for updated list_plugins docstring
* fix(security): strip every litellm auth header before forwarding to plugins
The plugin reverse proxy only removed Authorization and x-api-key, but
user_api_key_auth also authenticates a caller via API-Key, x-goog-api-key,
Ocp-Apim-Subscription-Key, x-litellm-api-key, and any configured custom key
header. A malicious plugin could lure a proxy_admin into calling
/plugin-proxy/... with the litellm key in one of those headers; the request
authenticated locally and then forwarded the same key to the plugin, letting it
impersonate the admin.
Add a canonical SpecialHeaders.litellm_credential_header_names() that the auth
header enum is the single source for, and strip that whole set plus the live
general_settings.litellm_key_header_name from every forwarded request. New auth
headers added to SpecialHeaders are now stripped automatically. Regression tests
cover each credential header, the custom configured header, and the canonical
list's contents.
|