* fix(docker): restore litellm-proxy-extras source dir in runtime images
#30243 narrowed the runtime stage to an allowlist COPY, which dropped
/app/litellm-proxy-extras from the published images. Downstream
migration jobs point prisma migrate deploy at that path; with the
schema gone (or a schema with no adjacent migrations dir, where prisma
exits 0 without applying anything) those jobs went green while never
migrating the database. Restore the folder in all three runtime stages
and assert in image-scan that the schema and a non-empty migrations dir
ship at the source path
* chore(ci): drop image-scan migration-assets assertion
(cherry picked from commit 111d447e1b)
* fix(guardrails): walk Responses-API text taxonomy in shared content helpers
Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently
drops all text on the /v1/responses path. AIM turns it into a loud 422 (
{"error":"No messages in the request"}); every other guardrail (Lakera v2,
Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret
detection) scans an empty payload and lets the request through unscanned.
Three defects, all in _content_utils.py:
1. _iter_text_parts_in_content recognised only part.type == "text", but the
Responses API uses input_text (request) and output_text (assistant).
2. _coerce_input_to_messages gated on "every item has a role key"; any
Responses input list containing a function_call or function_call_output
item failed the check and was wrapped as one opaque blob.
3. build_inspection_messages forwarded any role through, including a bare
tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze
reject with a schema error.
Fix walks the actual Responses item taxonomy (message, function_call,
function_call_output, bare content parts and strings), recognises
{text, input_text, output_text} everywhere, and coerces any role outside
{system, user, assistant} to user in the outbound inspection payload.
* style: ruff-format changed guardrail files
* test(guardrails): cover function_call_output string form; drop em-dash in new docstring
* fix(guardrails): map function_call_output straight to user role
Avoids ever materialising a schema-invalid bare tool message. The
downstream role-safety coercion in build_inspection_messages still
guards genuinely caller-supplied non-standard roles (developer,
function, custom values); add a regression test covering that path
so the coercion has real coverage after this simplification.
* test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages
* docs(test): soften AIM-specific claims in LIT-4294 test docstrings
Ryan's review flagged that several test docstrings assert AIM's
/fw/v1/analyze validates + rejects specific schema violations. That
behavior is customer-reported in the LIT-4294 writeup, not directly
verified by us. Rephrase to attribute the AIM 422 to the customer's
writeup and describe the underlying constraint as the OpenAI chat
schema; any downstream API that validates against that schema rejects
the same shape.
* refactor(guardrails): move unsupported-role coercion into AIM only
The generic coercion in build_inspection_messages collapsed any role
outside {system, user, assistant} to user for every caller of the
helper. Combined with the pre-existing apply_redacted_messages_back
write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400
on chat-completions tool-message masking into a silent semantic
corruption of the outbound request (role tool with tool_call_id got
rewritten to bare role user, dropping the assistant + tool_calls
sibling).
AIM specifically requires the coercion because its /fw/v1/analyze
validates the payload against the OpenAI chat schema; other guardrails
either do not validate roles or do their own reconstruction. Move the
coercion to AimGuardrail._build_aim_inspection_messages so the shared
helper keeps caller roles intact and no new cross-guardrail role
corruption is introduced. The pre-existing apply_redacted_messages_back
structural flatten remains as separate follow-up work.
function_call_output items still synthesise role user in the shared
helper because they have no natural role field, which is a different
concern from coercing a caller-supplied role.
* refactor(guardrails): preserve role fidelity in shared _content_utils
Shared inspection helpers should extract text and preserve semantic
role signals; role coercion for third-party schema safety stays inside
the guardrail that needs it (AIM).
Three shared-helper changes:
- Bare content-part dicts (input_text/output_text) with an explicit role
keep it; only role-less parts default to user.
- Responses message items already had their role preserved; the
behavior is now covered by an explicit test.
- function_call_output items default to role tool (semantic equivalent
of the chat-completions tool message shape) instead of role user, so
Responses and chat completions produce symmetric inspection payloads.
A caller-supplied role on the item is still preserved.
AIM's schema-safe coercion in _build_aim_inspection_messages already
handles the resulting role tool: it collapses to user before the POST
to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the
bare tool message (no tool_call_id can survive the flatten). Added a
regression test in test_aim.py covering that path.
(cherry picked from commit e84a19acd5)
Backport of #31393 to stable/1.90.x.
Cherry-picked from 7acc0157df (litellm_internal_staging).
The test-file conflict hunk also carried the staging-only TestMCPClientResolvedAuth
class from an unrelated commit that never reached this line; it was dropped and only
the additions belonging to #31393 were kept.
Backport of #31929 to stable/1.90.x.
Cherry-picked from 1543725916 (litellm_internal_staging).
Deviations from the upstream squash, needed because the line predates some
staging-only state:
- model_prices JSON churn reduced to its semantic content: drop the stale
1h-TTL pricing keys from the two Claude 3.5 Sonnet Bedrock entries and add
supports_parallel_tool_use_config to the flagged entries. 6 of the 58
flagged entries (*.anthropic.claude-sonnet-5) do not exist on this line and
were skipped
- ported the local_model_cost_map fixture into
test_anthropic_claude3_transformation.py (added upstream by an earlier
staging commit that never reached this line)
- test_parallel_tool_calls_config_kept_for_sonnet_5 renamed and pointed at
anthropic.claude-sonnet-4-6, since sonnet-5 has no entry on this line
- test_parallel_tool_calls_newer_model_adds_disable_flag now forces the
bundled cost map so it does not depend on the remote map having synced the
new flag
- ruff-strict-budget.json kept at the line's schema (no change needed)
RealTimeStreaming.log_messages dispatched the success handler with a bare
asyncio.create_task, bypassing GLOBAL_LOGGING_WORKER (which gives a per-coroutine
timeout and a concurrency cap). On a long-lived realtime websocket a slow logging
callback left one suspended task per logged turn, each pinning that turn's
assembled response, accumulating without bound (~12-15k in-flight under load in a
repro) until OOM. Route realtime success logging through the bounded worker so
in-flight logging is capped and a hung callback is cancelled at the worker
timeout.
The chat and responses streaming success-logging paths are intentionally left
unchanged: their success callbacks must complete within the call's event-loop run
(the non-streaming path pairs the worker with a synchronous callback; the
streaming path has no such companion), so deferring them through the worker would
drop logs for one-shot SDK calls and breaks test_async_custom_handler_stream.
Bounding those paths needs a load-shedding approach and is left to a follow-up.
(cherry picked from commit d4c33b2b59)
* fix(realtime): stop sending a second Gemini Live setup on follow-up session.update
Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client
message; a second setup closes the socket with 1007 Request contains an invalid
argument. The AI Studio Gemini path forwarded every client session.update after
the first as a follow-up setup, and GA clients (pipecat) send several while
configuring the session, so the second one tore the session down before the
first turn. Callers saw silence after the first response, exponential per-turn
latency from reconnect/retry churn, and intermittent 1011 errors.
Drop subsequent session.updates instead of resending setup, matching what the
Vertex subclass already does. Tools and instructions must ride on the first
session.update before any conversation content.
Adds regression tests covering the plain follow-up, a follow-up that adds tools
(the case the previous identical-only dedup still forwarded), and the guardrail
create_response=False warning path.
* fix(realtime): retry the backend open handshake instead of failing with 1011
The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs;
waiting longer never recovers a hung attempt, but a fresh attempt almost always
connects in ~1s. The proxy opened the backend websocket once with the default
open_timeout and no retry, so a single slow handshake surfaced to the caller as
a fatal 1011 internal error and dropped the call.
Bound each open attempt with a short open_timeout and retry; a bounded attempt
that already timed out spaces out the next try, so no backoff is needed.
Deterministic handshake-status rejections (auth/4xx) are not retried, and the
retry only ever wraps the open, never a live session.
Adds tests for retry-then-succeed, raise-after-max-attempts, and
no-retry-on-auth-failure.
* fix(realtime): close guardrail bypass + surface handshake status; drop obsolete tests
Three review fixes on the Gemini Live realtime path.
Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once
the initial setup is sent the guardrail's automaticActivityDetection.disabled=true
can no longer be delivered as a follow-up session.update. With that follow-up now
dropped, the model's auto-response stayed enabled and a realtime_input_transcription
guardrail was bypassed (the model answered before the proxy could gate the turn).
Fold the disable into the one-and-only setup instead: the handler injects it into
the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it
into the deferred first setup. OpenAI sessions accept follow-up updates and are left
untouched.
Backend handshake status: the open-retry treated only InvalidStatusCode as
deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake,
so a 401/403 fell into the broad WebSocketException branch and was retried before
the caller closed the client with 1011 instead of the upstream status. Treat both as
non-retryable.
Obsolete tests: the four tests asserting a follow-up session.update is merged and
re-sent as a second setup asserted behavior that crashes Gemini Live with 1007
(verified directly against the API). Removed; the drop is covered by new regression
tests.
* style(realtime): reformat changed files to ruff line-length 120
Post-merge with litellm_internal_staging, which unified ruff format width to 120
(#31518). The realtime change set was formatted at 88, so the changed lines
tripped the whole-file ruff format check. Reformat with ruff 0.15.3 at the repo's
120 width; no logic changes.
(cherry picked from commit ef5d05f137)
* fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads
PR #31036 switched the vertex batch file upload from a single GCS media
upload to a chunked resumable session. The resumable path sends the body as
many sequential PUTs, each waiting a full round-trip to GCS before the next,
so a multi-GB upload accumulates hundreds of round-trips and overruns the
client/load-balancer request timeout, surfacing as 499s (client closed
connection) on files as small as 500MB. This was a regression from the
last-known-good commit, where the upload completed as one continuous request.
Revert the batch upload to a single uploadType=media request, but stage the
transformed payload to a temp file first so peak memory stays bounded (the
goal of the resumable rewrite) without the per-chunk round-trips. The temp
file is closed deterministically (TemporaryFile unlinks on close), not left
to the GC. The now-unused resumable chunked-upload plumbing is removed.
Also swap the per-row transform's stdlib json for orjson (parse + serialize),
which is ~4x faster on this hot path; the streaming body now emits compact
orjson bytes.
The request stays synchronous, so the returned file object is real and
POST /v1/batches keeps working immediately against the uploaded object.
Tests: single media request carries the whole payload with a real
Content-Length (no chunked transfer-encoding); failed upload raises; the
staged temp file is closed deterministically; byte-for-byte transform parity.
* test(vertex_ai/files): mock single media upload POST instead of removed resumable method
test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type.
* fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports
Forward the per-request timeout through _stage_and_upload_media /
_astage_and_upload_media to the GCS POST. Every other upload branch forwards
it; the new media path was dropping it, so a caller-provided timeout was
silently ignored (the files path passes 600s by default, but a custom
request_timeout would not have reached this upload). Regression test asserts
the resolved timeout reaches the request (mutation-verified).
Revert the orjson swap in the batch transform: importing orjson at module load
in this core-path file broke `import litellm` on environments without orjson
(the Windows import test). Back to stdlib json; the upload leg dominates large
uploads anyway, so the transform-side win was marginal.
Fix import ordering in llm_http_handler.py (I001) introduced by the new imports.
* fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file
Addresses a disk-exhaustion concern: staging the full transformed batch body to
a local temp file before the GCS request meant an authenticated user could fill
the proxy's temp volume with large concurrent uploads (on top of Starlette's
input spool).
GCS's simple/media upload accepts chunked transfer-encoding, so stream the
transform straight to the single media request instead. Each block is produced
on a worker thread (the transform never runs on the event loop) and sent
chunked, so the body is neither buffered in memory nor written to disk, and the
upload is still one continuous request (no per-chunk round-trips, no 499). Drops
the temp-file staging, the tempfile/IO imports, and Content-Length computation.
Regression test asserts the upload streams (chunked transfer-encoding, no
Content-Length) and creates no temp file; mutation-verified that reintroducing
staging fails it.
(cherry picked from commit 85840aef51)
* fix(proxy/client): redact api key from key/info client error messages
The keys management client builds GET /key/info?key=<key> and lets the
requests HTTPError propagate. str(HTTPError) renders the failing request URL
verbatim ("... for url: .../key/info?key=sk-..."), so any caller that logs the
exception leaks the full key; the 401 branch leaked the same way through
UnauthorizedError(str(orig_exception))
Redact both branches with the existing redact_secrets helper so the
secret-bearing query param is scrubbed to ?REDACTED while the status code,
reason, and response object are preserved. Server-side responses already mask
the key, so this closes the remaining client-side surface
* fix: preserve key info unauthorized response
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 71ee1a852a)
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 56825926af)
An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the
user to sign in. On an unauthenticated initialize the gateway answered locally
(200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop
either connected empty or hit "OAuth probe timeout after 10000ms".
#30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers
to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth
servers, expecting the upstream to emit its own challenge. On initialize the
gateway never probes upstream, so no challenge ever reached the client.
Replace the `continue` with a preemptive 401 carrying the proxied
resource_metadata (RFC 9728) challenge, the same form passthrough servers and
MCPUpstreamAuthError already use. This keeps #29770 fixed (still no
authorization_uri) while restoring the upstream PKCE sign-in prompt.
The App Router migration moved pages to deeper path segments and the proxy
can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local
logo asset paths were emitted without the server root prefix, so they resolved
off the origin root and 404'd. Route every local logo src through a single
resolver that prefixes the live server root path and leaves external URLs
untouched, fixing provider, guardrail, vector store, callback, MCP and
audit-log logos at any route depth and root path.
The migrated /ui/api-keys route gates rendering on useAuthorized() but read userID from the AuthContext (useAuth), which hydrates asynchronously. On a hard refresh or deep link the route could render UserDashboard before AuthContext had populated userID, so UserDashboard hit its `userID == null` guard and showed "User ID is not set". The legacy index page avoided this by gating on AuthContext's own authLoading; the migration switched the gate to useAuthorized without aligning the identity source.
Read identity from useAuthorized (a synchronous cookie decode) so userID is populated whenever the route is authorized. useAuth is kept only for the backfill setters UserDashboard still expects, until the planned AuthContext consolidation removes them.
Refs LIT-3687
* feat(sandbox): add e2b code execution primitive
Add a provider-agnostic code execution primitive that runs model-generated
code in an isolated sandbox and returns the output, with e2b as the first
backend over raw httpx (no SDK dependency).
Public API: litellm.acode_interpreter_tool (ephemeral create -> run -> delete)
plus the low-level lifecycle litellm.acreate_sandbox / arun_code /
adelete_sandbox. Each is @client-decorated so operations are logged like
litellm.asearch. Backends implement BaseSandboxConfig; resolved via
ProviderConfigManager.get_provider_sandbox_config.
* fix(sandbox): address review feedback and CI gates
- document e2b provider in provider_endpoints_support.json and add a sandbox endpoint definition
- regenerate dashboard CallTypes after the sandbox call-type additions
- guard explicit timeout=0 instead of coercing it to the default
- require a ContainerHandle access token before running code; reject bare-id runs
- return False on a 404 delete now that the shared http handler raises for status
- skip non-JSON NDJSON lines and cap streamed output to bound memory
- move the real-network integration tests out of tests/test_litellm into tests/integration/sandbox
* fix(sandbox): satisfy strict ruff gate and scope star-exports
- modernize annotations in the new sandbox modules to PEP 585/604 (list/dict,
X | None) and drop the now-unnecessary quoted forward refs so the strict-rule
budget delta for UP006/UP037/UP045 returns to zero
- add __all__ to litellm/sandbox/main.py so 'import *' only re-exports the four
public entrypoints instead of leaking module-level imports
* fix(sandbox): drop quotes on sandbox config return annotation
utils.py uses 'from __future__ import annotations', so the quoted forward ref
tripped UP037; the unquoted union is lazily evaluated and keeps the strict-rule
delta at zero
* chore(sandbox): re-trigger automated review after addressing feedback
* test: point router/completion/triton tests at the local fake OpenAI endpoint
The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app)
takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted
proxy configs onto a job-local fake server but left these in-Python api_base
literals pointing at the dead host, so litellm_router_testing, local_testing_part1,
local_testing_part2 and llm_translation_testing still fail with a 404
"Application not found" when Railway is down
Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190)
through a shared helper, auto-start the canned server from the local_testing and
llm_translation conftests when nothing is already serving, and extend the server
with a Triton embeddings route and a slow-endpoint delay so the triton and
latency-timeout tests run fully offline. The deliberately broken fallback URL is
left as-is so fallback handling still has a failing upstream
* fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI
* fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target
* fix(tests): keep fake OpenAI mock alive across xdist workers
ensure_fake_openai_endpoint registered atexit on the worker that spawned
the subprocess, so under -n 4 the first worker to drain its queue would
terminate the shared mock while siblings were still hitting it. Detach
the child via start_new_session and drop the per-worker teardown; reuse
on /health handles re-runs and CI containers clean up themselves
* fix(watsonx): wrap string embedding input in array for WatsonX API
WatsonX text/embeddings expects inputs as []string; OpenAI clients often send a single string.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(watsonx): format watsonx embed transformation for black
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(watsonx): avoid UP006 in embed transformation strict lint gate
Use list[str] and branch-based input normalization instead of List and cast
so the watsonx embedding change does not add strict ruff UP006 violations.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add CI-parity mode and truncation-proof summary to strict ruff gate
* refactor: tolerant worktree cleanup and concrete GateInputs types
* fix: clean up temp dir when git worktree add fails
* fix: align lint-gate with CI by dropping unused --ci-parity path
The lint-gate Makefile target invoked ruff_strict_gate.py with --ci-parity,
which counted violations on a throwaway merge of base into HEAD against base
counts at the base tip. CI in test-linting.yml runs the same script without
--ci-parity on a PR-head checkout, taking the gather_fast path that counts on
the live tree against base counts at the merge-base. A local pass could
therefore disagree with CI.
Drop --ci-parity from the Makefile and remove the now-unused gather_ci_parity
branch and flag so there is one code path that both local and CI exercise.
The docstring claim that CI runs against the synthetic merge ref was also
wrong; the workflow checks out github.event.pull_request.head.sha.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The deprecated OldTeams component takes only accessToken, userID, userRole
and premiumUser; it ignores the teams prop these tests passed and instead
populates its table from the mocked teamListCall. The delete-warning block
never set teamListCall, and vi.clearAllMocks clears call history but not
implementations, so the table rendered the "Legacy Team" (keys.length 2)
left behind by the previous block's last test. Both delete tests therefore
ran against that leaked team: the keys-present case passed only because the
leaked count happened to be 2, and the no-keys case rendered the same warning
it asserted should be absent, so it failed.
Seed the team through the channel the component actually reads (teamListCall)
and drop the props it never consumes, so each test renders exactly the team
it declares. The keys-present case now uses a distinctive count so it can no
longer pass on a coincidental leak
Adds a "valkey-semantic" cache type so semantic prompt caching can run
against Valkey clusters (for example AWS ElastiCache for Valkey) using the
valkey-search module.
The existing "redis-semantic" backend cannot drive valkey-search. RedisVL
gates the connection on a RediSearch module version that valkey-search does
not report, and its SemanticCache index declares the prompt as a TEXT field,
which valkey-search does not implement. ValkeySemanticCache therefore talks to
valkey-search directly over redis-py: it builds a vector index from the field
types valkey-search supports (TAG for caller scope, VECTOR for the prompt
embedding) and runs KNN queries for retrieval. Prompt extraction, embedding
generation, and cached-response parsing are reused from RedisSemanticCache
since those are backend agnostic. The redis dependency is imported lazily in
the cache dispatch so importing litellm without redis installed still works.
It also fixes semantic-cache scope keys so similarity matching works across
reworded prompts. get_cache_key() hashed messages / prompt / input into the
litellm_cache_key that every semantic backend filters its KNN search on, so a
paraphrase landed in a different bucket and never matched, even far above the
similarity threshold. For semantic cache types the prompt-bearing params are
now excluded from the scope key and the server-set tenant identity
(user_api_key, team, org) is appended instead, restoring embedding matching
within a tenant while keeping cache entries scoped to the authenticated
key / team / org. The three semantic backends share this key, so the same
change fixes redis-semantic and qdrant-semantic.
Connections resolve from VALKEY_HOST / VALKEY_PORT / VALKEY_PASSWORD, falling
back to REDIS_* for drop-in compatibility, and passwordless clusters (IAM or
no-auth) are supported.
Resolves#29121Fixes#29086
The delete-team confirmation modal warned that a team's keys would be
deleted but said nothing about models. #29977 made team deletion also
delete the team's BYOK models, so the modal copy was understating what
gets removed.
The warning banner now mentions models alongside keys, and the
always-shown confirmation message does too so a team that has models but
no keys (the banner only renders when keys exist) still gets warned.
The "View Usage Guide" button on the legacy Usage page (shown when
DISABLE_EXPENSIVE_DB_QUERIES is set, i.e. SpendLogs has 1M+ rows) linked
to docs/proxy/spending_monitoring, which was removed from the docs and
now returns 404. Point it at docs/proxy/cost_tracking, which is live.
Fixes LIT-2724
A streaming request that breaks mid-flight, for example on a mid-stream read
timeout, still bills the provider for the chunks already delivered, yet the proxy
recorded that interrupted request as a zero-spend failure. An earlier revision
logged the recovered partial usage through the success path, which mislabeled a
failed request as a success and produced a misleading spend row
This recovers the partial usage where the failure is actually logged. The
streaming handler assembles the usage from the chunks seen so far and stashes it,
with its cost, on the logging object before firing the failure handlers. The
proxy failure hook lifts that usage and cost onto request_data before the
non-serialisable logging object is popped, and the spend-log writer records the
real partial spend on the failure row instead of a hardcoded zero;
get_logging_payload honors the recovered usage for the token columns and
_failure_handler_helper_fn preserves the recovered cost so the non-DB failure
loggers stay consistent
A request that recovers via a successful fallback is unaffected: the failure hook
only fires when the whole request fails, so the fallback's combined-usage success
row stays the single source of truth and there is no double counting
Resolves LIT-3825
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(otel): one v2 logger owns the global provider; scope tenant creds per exporter
The proxy published the OTel global TracerProvider before callbacks were
initialized, so no preset logger existed yet and a second generic logger was
built that won the global provider. Server spans then exported through a
different provider than the preset's gen-ai spans, orphaning the LLM span on
the preset backend. Publish after callback init and reuse the already-built
logger instead.
Separately, per-request tenant OTLP credentials were stamped onto every OTLP
exporter, leaking one backend's key onto a co-configured backend. Tag each
exporter with the preset that contributed it and apply dynamic credentials
only to the matching owner.
* fix(otel): satisfy Any-discipline on changed lines
Type the logger-selection parameter as Sequence[object] (isinstance narrows
it), cast the list[Any] global at the single call site, and pass model_copy a
typed dict[str, str] update so no changed line carries an Any value.
* fix(otel): annotate the untyped-global boundary with any-ok
select_global_otel_v2_logger consumes litellm._in_memory_loggers, a shared
List[Any] global this change does not own. A cast doesn't satisfy the
Any-discipline checker (it inspects the inner expression), and re-annotating the
global is out of scope, so mark the single boundary line any-ok.
* test(otel): cover the startup global-provider publish via injectable helper
The publish step lived inline in proxy_startup_event (a FastAPI lifespan unit
tests do not execute), so its lines were uncovered though the selection logic
was tested. Extract publish_global_otel_v2_provider, which selects the single v2
logger and publishes its provider through an injected setter, and unit-test that
the published provider is the selected logger's. proxy_server delegates to it.
* refactor(otel): select global provider from the registered owner, not a list scan
The startup publish picked the global TracerProvider by scanning
_in_memory_loggers for the first OpenTelemetryV2, re-deriving an answer the
factory already settled: the first logger built registers itself as
proxy_server.open_telemetry_logger, and every other v2 path (guardrail, identity
seeding, phase spans) routes through that owner via _registered_v2_logger. Pass
that owner into select_global_otel_v2_logger so the global provider reuses the
same logger instead of an independent, order-dependent guess; the list scan
remains the SDK-path fallback. The owner is injected at the proxy call site to
keep the helper free of hidden global reads.
* refactor(otel): type ExporterSpec.owner as an ExporterOwner enum
The owner field carried free-form strings that had to match preset callback
names. Introduce a str-based ExporterOwner enum (values equal to the callback
names, so per-request credential routing's owner==callback_name comparison still
holds) and have each preset tag its exporter with the enum member.
* refactor(otel): rename ExporterOwner.ARIZE to ARIZE_AX
Distinguish the hosted Arize AX backend from Arize Phoenix at the member level
while keeping the value 'arize' (the public callback name routing compares
against). Add a comment noting AX and Phoenix are separate backends.
* fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough
When a guardrail blocks a streaming request pre-call by raising
ModifyResponseException (or RejectedRequestError), chat_completion streams the
violation message back as a 200 by building a CustomStreamWrapper. It read the
logging object from the outer request body (`data.get("litellm_logging_obj")`),
but that dict never carries litellm_logging_obj -- it diverges from the
processor's data at function_setup, and only the processor copy (exposed as
e.request_data, already bound to `_data` here) gets the logging object
attached. CustomStreamWrapper.__init__ then dereferences
`logging_obj.model_call_details` on None and 500s the request with
"AttributeError: 'NoneType' object has no attribute 'model_call_details'".
Read logging_obj from `_data` (= e.request_data) in both streaming
passthrough handlers so the refusal streams correctly. The non-streaming and
the anthropic/responses passthrough paths were unaffected.
Adds a regression test asserting the wrapper receives the logging object from
e.request_data rather than None.
* test(proxy): cover RejectedRequestError streaming passthrough
The streaming logging_obj fix was applied to both the ModifyResponseException
and RejectedRequestError handlers, but only the former had a regression test.
Extract a shared helper and add a parallel test for the RejectedRequestError
streaming path so both handlers stay guarded against the None-logging_obj crash.
---------
Co-authored-by: Joseph Barker <joseph.barker@rubrik.com>
Switch the zizmor check to fail on any finding at medium severity or
above (advanced-security off, min-severity medium, annotations on) so it
can be promoted to a required check, and pin the engine to zizmor 1.24.1
through zizmor-action v0.5.6 for deterministic runs.
Clear the findings that were outstanding so the check passes: correct
mismatched action pin version comments, scope the proxy endpoint
workflow's id-token and pull-requests permissions to the jobs that use
them, and mark the server-root-path docker build as non-publishing while
dropping its shared gha build cache.
* feat(proxy): add configurable response headers middleware
Adds a small ASGI middleware that sets standard response headers
(X-Frame-Options, Content-Security-Policy frame-ancestors, X-Content-Type-Options)
on proxy and UI responses. Strict-Transport-Security is optional and gated
behind LITELLM_ENABLE_HSTS for HTTPS deployments. Values use setdefault so a
route that sets its own header is preserved.
* feat(proxy/ui): make login page credentials hint configurable
build_ui_login_form accepts a hide_default_credentials_hint parameter and
google_login reads LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (or general_settings)
so the legacy login page behaves consistently with the new UI. Also collapses
a duplicated branch and removes an unused variable and module-level constant.
* fix(proxy/ui): apply credentials hint flag on /fallback/login
The /fallback/login handler still rendered the default-credentials hint
regardless of LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT. Collapse its duplicate
branch and forward the flag, matching google_login, so all login surfaces
behave consistently. Adds regression tests for /fallback/login and makes the
ui_sso test helper restore os.environ so env vars do not leak across tests.
Cut the default "Virtual Keys" landing (?page=api-keys) over to a path route at
(dashboard)/api-keys. The dashboard is extracted into a shared ApiKeysDashboard
component used by both the new route and the index's inline render, so there's
no duplication. Adding the MIGRATED_PAGES entry repoints the sidebar item and
redirects ?page=api-keys to /ui/api-keys.
The index is the post-login landing and still hosts the legacy switch for the
not-yet-migrated pages (models, pass-through, usage) plus the invitation flow,
so it stays. The auto-redirect now fires only for an explicit ?page= param,
leaving the bare /ui/ landing to render inline; this keeps the return-URL
handling and the invitation_id flow (both of which run at the bare landing)
intact, where a blanket redirect would have dropped them. The new route uses
useAuthorized for the login gate, matching every other migrated route.
