Root cause: PR #30867 removed request-time os.environ/ expansion in
BaseAWSLLM.get_credentials. That is only safe if config-load pre-resolves
os.environ/ refs so the value reaching get_credentials is already the real
secret. The YAML config path has always done this. The DB-load path
(ProxyConfig._resolve_db_litellm_param) only re-expanded keys in a hardcoded
whitelist (_DB_LITELLM_PARAM_ENV_REF_KEYS) plus short-circuited env-ref
resolution entirely for team-scoped rows. PR #32256 extended that whitelist
to 18 keys to unblock a customer whose Bedrock model with aws_role_name:
os.environ/BEDROCK_ASSUME_ROLE_ARN broke on v1.90+, but the whitelist is
structurally fragile: every future auth field breaks the same way until
someone remembers to add it
Fix: remove the whitelist and the team-scope short-circuit. The DB-load
resolver now expands os.environ/ on every string field, matching the YAML
path. Trust boundary stays on the write side: only PROXY_ADMIN can create
team_id=None rows, only team admins of a team can create rows scoped to
that team, and the request-body vector is still blocked by
_BANNED_REQUEST_BODY_PARAMS. Team-scoped rows now resolve env refs — this
is a deliberate LIT-3831 threat-model expansion trusting team admins for
env-var reads
Regression tests in tests/test_litellm/proxy/proxy_server/test_proxy_config.py:
- test_ProxyConfig__add_deployment_resolves_env_refs_after_db_decrypt pins
admin-scoped rows resolve every field (previously api_base stayed literal)
- test_ProxyConfig__add_deployment_resolves_team_env_refs pins team rows
resolve env refs (previously stayed literal)
- test_ProxyConfig__add_deployment_resolves_env_refs_on_arbitrary_field pins
the no-whitelist invariant against a made-up field name
- test_ProxyConfig__add_deployment_resolves_env_refs_for_aws_bedrock_auth_params
(from #32256) still passes
- Path B counterparts (decrypt_model_list_from_db) mirror the above
Left as followups (not fixed here):
- /model/info and /v2/model/info still echo resolved values for fields not
in the current pop-list (aws_role_name, aws_sts_endpoint, api_base, etc.).
Fix is to extend remove_sensitive_info_from_deployment; separate PR
- Master-key rotation reads DB rows via decrypt_model_list_from_db which
now resolves universally, so rotation collapses env-refs into hardcoded
values. Pre-existing bug for the 6 previously-whitelisted fields; wider
surface after this PR. Separate PR
(cherry picked from commit 5862be3e79)
* fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool
The advisor_20260301 interceptor honored a caller-supplied api_base once
allow_client_side_credentials was enabled, even without a caller-supplied
api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's
own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials
plus the conversation history got sent to a caller-chosen destination
_resolve_advisor_credentials() now only honors api_base alongside a
non-empty caller-supplied api_key, requires the https scheme, and validates
api_base via validate_url() before use, mirroring check_complete_credentials
in auth_utils.py. https is required because validate_url only DNS-pins the
connection for http; for https with TLS verification on it returns the URL
unchanged and relies on certificate validation to block DNS rebinding
* fix(anthropic): also reject advisor api_base when ssl_verify is disabled
validate_url only DNS-pins the connection for http, or for https with
litellm.ssl_verify disabled; the previous https-only check missed the
ssl_verify=False case, where validate_url's rewritten URL was still being
discarded, per Greptile's review of this PR. Reject api_base outright when
ssl_verify is False so the discarded rewrite can no longer matter
(cherry picked from commit 07b9ea8c3b)
The security fix in 34e9be1ba7 removed turn_off_message_logging from
_supported_callback_params to stop callers bypassing global redaction via
the request body. That also killed the documented admin-only per-key or
per-team override because both flows resolve through the same allowlist
in initialize_standard_callback_dynamic_params.
Put turn_off_message_logging back in _supported_callback_params so an
admin-configured metadata.logging[].callback_vars.turn_off_message_logging
survives into StandardCallbackDynamicParams and can override the global
setting for that key or team, as documented at
docs/proxy/team_logging#disableenable-message-redaction.
Consolidate the metadata traversal so the extractor and the proxy strip
walk the same set of client-controllable slots. iter_client_callback_metadata_dicts
in litellm_core_utils/initialize_dynamic_callback_params.py is the single
source of truth for metadata, litellm_metadata, and litellm_params.metadata;
_strip_client_message_redaction_opt_out imports it so a future addition
to one side automatically reaches the other. The extractor iterates the
helper in reversed order so litellm_params.metadata keeps overriding
metadata, matching the pre-refactor merge precedence.
Client bypass stays blocked. Restoring the field re-enrolls it in the
auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from
_supported_callback_params via _build_banned_observability_params), so
client submissions at the top level, inside metadata, or inside a
JSON-string litellm_metadata all 401 at ingress. is_request_body_safe
also now descends into litellm_params.metadata for the same 401 defense
against the nested-body attack vector, matching how the metadata and
litellm_metadata slots are handled. _strip_client_message_redaction_opt_out
runs after the litellm_metadata JSON parse and before the admin callback_vars
unpack, so admin values survive while any leftover client-supplied
opt-out is dropped when global redaction is on and the key or team
lacks allow_client_message_redaction_opt_out.
Flip the two dynamic-param e2e tests added by the security fix to
reflect the restored override behavior, keeping the invariant that
proxy client bypass is stopped by the auth layer 401 above.
Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 8e6098adc3)
AWS auth parameters in the Bedrock and SageMaker path could be expanded against
the process environment when credentials were built. Config-sourced references
are already expanded at load time, so restrict expansion to that path: a
reference still present at request time is treated as caller-supplied input and
is left as-is, and the web-identity helper rejects environment-variable
references before resolving the token.
Also rework the ambient AWS_* fallback as a single pass that pairs each value
with its own env-var name, fixing a latent index misalignment that left
AWS_EXTERNAL_ID unresolved.
Adds regression tests covering the resolution behavior.
(cherry picked from commit 4ef7d0815b)
* fix: validate proxy request body and nested fields
Ensure caller-supplied request fields cannot override server-side deployment
configuration, and apply request-body validation consistently to nested
structures. Adjusts router kwarg handling and client-side credential handling
for base-url overrides
* test: cover router strip ordering and advisor clientside credential gate
* fix: clear deployment credentials on client base-url override
When a request overrides api_base/base_url, recompute the deployment's
litellm_params (clearing the deployment's own api_key) and drop the cached
client built for the original endpoint, so the deployment credential is not
reused for the client-supplied endpoint. Adds regression tests that assert the
credentials actually forwarded to litellm.completion/acompletion.
* fix(proxy): require api_key alongside api_base override
A request that overrides api_base/base_url but supplies no api_key still
left the proxy carrying a server credential: once the override clears the
banned-param opt-in, the provider re-resolves a key from the environment
(api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in
main.py) and forwards it to the caller-controlled URL. Popping the
deployment api_key only changed which server key leaked.
Gate is_request_body_safe so a permitted api_base/base_url override must
also carry a non-empty caller api_key; reject otherwise. The env
resolution in main.py is left as the provider boundary.
* fix(proxy): extend request-body banlist with five additional credential and session targeting fields
Yuneng's review found five deployment-owned request-body params still missing
from the denylist and the router strip set. Each lets a caller reach the
operator's provider credentials or retarget the outbound request:
aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region
retarget the OCI request, litellm_credential_name selects any server-loaded
credential by name with no ownership check, and runtimeSessionId resumes a
Bedrock AgentCore runtime session (AWS does not enforce session-to-user
mapping, so this is a cross-tenant session-resume vector).
Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to
_DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and
SDK direct calls are unaffected: the banlist gates the request body only, and
the router strip drops caller kwargs, never deployment["litellm_params"].
* test: rename arbitrary canary values in security tests to neutral placeholders
* fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip
P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS
(litellm_embedding_config, extra_body) for the banned-param check but not for
the api_key co-presence check, so a base override smuggled into one of those
nested dicts cleared the client-side-credentials opt-in without a paired
api_key and let the provider re-resolve a server credential from the
environment. Run _check_base_override_has_api_key on each nested config dict
too, so the requirement applies wherever a base override is permitted.
P1-B: the deployment-owned credential strip in the Router runs unconditionally
on every _completion/_acompletion, which is security-correct but silently
drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a
single warning (key names only, never values) when the strip removes a
non-empty value, so the backwards-incompatible behavior is visible without
gating the strip on a context flag that does not exist.
* fix(proxy): apply api_key co-presence to tool-entry base override
is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and
tools[]); the previous commit extended the api_key co-presence rule to root
and nested config dicts but not to tool entries. With
allow_client_side_credentials enabled, a tool entry carrying api_base/base_url
and no paired api_key cleared the gate, letting a provider interceptor fall
back to a server-side credential for a caller-controlled URL. Add the same
_check_base_override_has_api_key call to each tool dict and its nested function
dict, mirroring the symmetry already applied to the nested config keys. The
rule is unchanged: api_key must live in the same dict as the base override it
accompanies.
* test(proxy/auth): require paired api_key under extra_body opt-in
* fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running
* fix(advisor): narrow proxy-import guard to ImportError-family
* fix(router): gate api_key clear on base override behind litellm.proxy_is_running
* test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture
* style: use built-in generics in PR-added type annotations
* revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate
* revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes
* style: black-format advisor orchestration test
(cherry picked from commit 1667b8f740)
* fix(docker): bake prisma CLI and engines at a fixed path so fresh-DB migrations work for any uid offline
The runtime image shipped the prisma CLI and engines under /root/.cache, the
default HOME-derived prisma-python cache location. Any deployment whose
runtime HOME is not /root (kubernetes runAsUser, docker --user, HOME
overrides) missed that cache on a fresh database, fell back to a nodeenv
Node download that crashes on Wolfi (libatomic.so.1), and started the proxy
with zero tables while every DB-backed endpoint returned 500
The bake now lives at /opt/prisma, a path no HOME resolution or cache
volume mount can shadow. The builder records the engine paths there at
generate time, and the runtime stage pins PRISMA_BINARY_CACHE_DIR,
PRISMA_CLI_PATH, PRISMA_CLI_QUERY_ENGINE_TYPE=binary and
PRISMA_OFFLINE_MODE so both litellm-proxy-extras and prisma-python resolve
the baked CLI and engines directly. prisma migrate deploy on a fresh
database now needs no npm and no network access for any runtime uid,
including readOnlyRootFilesystem deployments
Verified against live containers: fresh and existing databases as root,
uid 12345, HOME overridden, on an internal-only docker network, and with
a read-only root filesystem all migrate and serve /team/new successfully
Fixes#33650, #24554
* chore(docker): fail the image build if the baked prisma CLI layout drifts
Asserts the baked CLI shim is executable and its entrypoint exists in the
runtime stage after the COPY and chmod, so a layout change in a future
prisma-python release breaks the image build loudly instead of silently
degrading the migration path at container startup
(cherry picked from commit 567ebcb3e9)
* fix(docker): restore litellm-proxy-extras source dir in runtime images
#30243 narrowed the runtime stage to an allowlist COPY, which dropped
/app/litellm-proxy-extras from the published images. Downstream
migration jobs point prisma migrate deploy at that path; with the
schema gone (or a schema with no adjacent migrations dir, where prisma
exits 0 without applying anything) those jobs went green while never
migrating the database. Restore the folder in all three runtime stages
and assert in image-scan that the schema and a non-empty migrations dir
ship at the source path
* chore(ci): drop image-scan migration-assets assertion
(cherry picked from commit 111d447e1b)
* fix(guardrails): walk Responses-API text taxonomy in shared content helpers
Every guardrail sharing litellm/proxy/guardrails/_content_utils.py silently
drops all text on the /v1/responses path. AIM turns it into a loud 422 (
{"error":"No messages in the request"}); every other guardrail (Lakera v2,
Cato, Lasso, Repello, IBM, Azure Content Safety, enterprise secret
detection) scans an empty payload and lets the request through unscanned.
Three defects, all in _content_utils.py:
1. _iter_text_parts_in_content recognised only part.type == "text", but the
Responses API uses input_text (request) and output_text (assistant).
2. _coerce_input_to_messages gated on "every item has a role key"; any
Responses input list containing a function_call or function_call_output
item failed the check and was wrapped as one opaque blob.
3. build_inspection_messages forwarded any role through, including a bare
tool role missing tool_call_id, which validators like AIM's /fw/v1/analyze
reject with a schema error.
Fix walks the actual Responses item taxonomy (message, function_call,
function_call_output, bare content parts and strings), recognises
{text, input_text, output_text} everywhere, and coerces any role outside
{system, user, assistant} to user in the outbound inspection payload.
* style: ruff-format changed guardrail files
* test(guardrails): cover function_call_output string form; drop em-dash in new docstring
* fix(guardrails): map function_call_output straight to user role
Avoids ever materialising a schema-invalid bare tool message. The
downstream role-safety coercion in build_inspection_messages still
guards genuinely caller-supplied non-standard roles (developer,
function, custom values); add a regression test covering that path
so the coercion has real coverage after this simplification.
* test(guardrails): pin chat-completions tool-role coercion in build_inspection_messages
* docs(test): soften AIM-specific claims in LIT-4294 test docstrings
Ryan's review flagged that several test docstrings assert AIM's
/fw/v1/analyze validates + rejects specific schema violations. That
behavior is customer-reported in the LIT-4294 writeup, not directly
verified by us. Rephrase to attribute the AIM 422 to the customer's
writeup and describe the underlying constraint as the OpenAI chat
schema; any downstream API that validates against that schema rejects
the same shape.
* refactor(guardrails): move unsupported-role coercion into AIM only
The generic coercion in build_inspection_messages collapsed any role
outside {system, user, assistant} to user for every caller of the
helper. Combined with the pre-existing apply_redacted_messages_back
write-back behavior in Lakera/AIM/Cato, that turned a loud OpenAI 400
on chat-completions tool-message masking into a silent semantic
corruption of the outbound request (role tool with tool_call_id got
rewritten to bare role user, dropping the assistant + tool_calls
sibling).
AIM specifically requires the coercion because its /fw/v1/analyze
validates the payload against the OpenAI chat schema; other guardrails
either do not validate roles or do their own reconstruction. Move the
coercion to AimGuardrail._build_aim_inspection_messages so the shared
helper keeps caller roles intact and no new cross-guardrail role
corruption is introduced. The pre-existing apply_redacted_messages_back
structural flatten remains as separate follow-up work.
function_call_output items still synthesise role user in the shared
helper because they have no natural role field, which is a different
concern from coercing a caller-supplied role.
* refactor(guardrails): preserve role fidelity in shared _content_utils
Shared inspection helpers should extract text and preserve semantic
role signals; role coercion for third-party schema safety stays inside
the guardrail that needs it (AIM).
Three shared-helper changes:
- Bare content-part dicts (input_text/output_text) with an explicit role
keep it; only role-less parts default to user.
- Responses message items already had their role preserved; the
behavior is now covered by an explicit test.
- function_call_output items default to role tool (semantic equivalent
of the chat-completions tool message shape) instead of role user, so
Responses and chat completions produce symmetric inspection payloads.
A caller-supplied role on the item is still preserved.
AIM's schema-safe coercion in _build_aim_inspection_messages already
handles the resulting role tool: it collapses to user before the POST
to /fw/v1/analyze so AIM's OpenAI-schema validator does not reject the
bare tool message (no tool_call_id can survive the flatten). Added a
regression test in test_aim.py covering that path.
(cherry picked from commit e84a19acd5)
Backport of #31393 to stable/1.90.x.
Cherry-picked from 7acc0157df (litellm_internal_staging).
The test-file conflict hunk also carried the staging-only TestMCPClientResolvedAuth
class from an unrelated commit that never reached this line; it was dropped and only
the additions belonging to #31393 were kept.
Backport of #31929 to stable/1.90.x.
Cherry-picked from 1543725916 (litellm_internal_staging).
Deviations from the upstream squash, needed because the line predates some
staging-only state:
- model_prices JSON churn reduced to its semantic content: drop the stale
1h-TTL pricing keys from the two Claude 3.5 Sonnet Bedrock entries and add
supports_parallel_tool_use_config to the flagged entries. 6 of the 58
flagged entries (*.anthropic.claude-sonnet-5) do not exist on this line and
were skipped
- ported the local_model_cost_map fixture into
test_anthropic_claude3_transformation.py (added upstream by an earlier
staging commit that never reached this line)
- test_parallel_tool_calls_config_kept_for_sonnet_5 renamed and pointed at
anthropic.claude-sonnet-4-6, since sonnet-5 has no entry on this line
- test_parallel_tool_calls_newer_model_adds_disable_flag now forces the
bundled cost map so it does not depend on the remote map having synced the
new flag
- ruff-strict-budget.json kept at the line's schema (no change needed)
RealTimeStreaming.log_messages dispatched the success handler with a bare
asyncio.create_task, bypassing GLOBAL_LOGGING_WORKER (which gives a per-coroutine
timeout and a concurrency cap). On a long-lived realtime websocket a slow logging
callback left one suspended task per logged turn, each pinning that turn's
assembled response, accumulating without bound (~12-15k in-flight under load in a
repro) until OOM. Route realtime success logging through the bounded worker so
in-flight logging is capped and a hung callback is cancelled at the worker
timeout.
The chat and responses streaming success-logging paths are intentionally left
unchanged: their success callbacks must complete within the call's event-loop run
(the non-streaming path pairs the worker with a synchronous callback; the
streaming path has no such companion), so deferring them through the worker would
drop logs for one-shot SDK calls and breaks test_async_custom_handler_stream.
Bounding those paths needs a load-shedding approach and is left to a follow-up.
(cherry picked from commit d4c33b2b59)
* fix(realtime): stop sending a second Gemini Live setup on follow-up session.update
Gemini Live (BidiGenerateContent) accepts setup as the first-and-only client
message; a second setup closes the socket with 1007 Request contains an invalid
argument. The AI Studio Gemini path forwarded every client session.update after
the first as a follow-up setup, and GA clients (pipecat) send several while
configuring the session, so the second one tore the session down before the
first turn. Callers saw silence after the first response, exponential per-turn
latency from reconnect/retry churn, and intermittent 1011 errors.
Drop subsequent session.updates instead of resending setup, matching what the
Vertex subclass already does. Tools and instructions must ride on the first
session.update before any conversation content.
Adds regression tests covering the plain follow-up, a follow-up that adds tools
(the case the previous identical-only dedup still forwarded), and the guardrail
create_response=False warning path.
* fix(realtime): retry the backend open handshake instead of failing with 1011
The upstream Live API open handshake (e.g. Gemini Live) intermittently hangs;
waiting longer never recovers a hung attempt, but a fresh attempt almost always
connects in ~1s. The proxy opened the backend websocket once with the default
open_timeout and no retry, so a single slow handshake surfaced to the caller as
a fatal 1011 internal error and dropped the call.
Bound each open attempt with a short open_timeout and retry; a bounded attempt
that already timed out spaces out the next try, so no backoff is needed.
Deterministic handshake-status rejections (auth/4xx) are not retried, and the
retry only ever wraps the open, never a live session.
Adds tests for retry-then-succeed, raise-after-max-attempts, and
no-retry-on-auth-failure.
* fix(realtime): close guardrail bypass + surface handshake status; drop obsolete tests
Three review fixes on the Gemini Live realtime path.
Transcription-guardrail bypass: Gemini Live rejects a second setup (1007), so once
the initial setup is sent the guardrail's automaticActivityDetection.disabled=true
can no longer be delivered as a follow-up session.update. With that follow-up now
dropped, the model's auto-response stayed enabled and a realtime_input_transcription
guardrail was bypassed (the model answered before the proxy could gate the turn).
Fold the disable into the one-and-only setup instead: the handler injects it into
the auto-sent setup (gemini_live_defer_setup false) and _send_to_backend injects it
into the deferred first setup. OpenAI sessions accept follow-up updates and are left
untouched.
Backend handshake status: the open-retry treated only InvalidStatusCode as
deterministic; websockets>=15 raises InvalidStatus for a rejected client handshake,
so a 401/403 fell into the broad WebSocketException branch and was retried before
the caller closed the client with 1011 instead of the upstream status. Treat both as
non-retryable.
Obsolete tests: the four tests asserting a follow-up session.update is merged and
re-sent as a second setup asserted behavior that crashes Gemini Live with 1007
(verified directly against the API). Removed; the drop is covered by new regression
tests.
* style(realtime): reformat changed files to ruff line-length 120
Post-merge with litellm_internal_staging, which unified ruff format width to 120
(#31518). The realtime change set was formatted at 88, so the changed lines
tripped the whole-file ruff format check. Reformat with ruff 0.15.3 at the repo's
120 width; no logic changes.
(cherry picked from commit ef5d05f137)
* fix(vertex_ai/files): upload batch files in a single media request to fix 499s on large uploads
PR #31036 switched the vertex batch file upload from a single GCS media
upload to a chunked resumable session. The resumable path sends the body as
many sequential PUTs, each waiting a full round-trip to GCS before the next,
so a multi-GB upload accumulates hundreds of round-trips and overruns the
client/load-balancer request timeout, surfacing as 499s (client closed
connection) on files as small as 500MB. This was a regression from the
last-known-good commit, where the upload completed as one continuous request.
Revert the batch upload to a single uploadType=media request, but stage the
transformed payload to a temp file first so peak memory stays bounded (the
goal of the resumable rewrite) without the per-chunk round-trips. The temp
file is closed deterministically (TemporaryFile unlinks on close), not left
to the GC. The now-unused resumable chunked-upload plumbing is removed.
Also swap the per-row transform's stdlib json for orjson (parse + serialize),
which is ~4x faster on this hot path; the streaming body now emits compact
orjson bytes.
The request stays synchronous, so the returned file object is real and
POST /v1/batches keeps working immediately against the uploaded object.
Tests: single media request carries the whole payload with a real
Content-Length (no chunked transfer-encoding); failed upload raises; the
staged temp file is closed deterministically; byte-for-byte transform parity.
* test(vertex_ai/files): mock single media upload POST instead of removed resumable method
test_avertex_batch_prediction patched BaseLLMHTTPHandler._aresumable_chunked_upload, which was removed when the batch jsonl upload moved from a chunked resumable GCS session to a single uploadType=media request. Patch the raw httpx.AsyncClient.post that _astage_and_upload_media issues so the real staging, upload and response transform run while the GCS object response is mocked, and assert the media URL and Content-Type.
* fix(vertex_ai/files): forward request timeout to media upload, drop orjson, sort imports
Forward the per-request timeout through _stage_and_upload_media /
_astage_and_upload_media to the GCS POST. Every other upload branch forwards
it; the new media path was dropping it, so a caller-provided timeout was
silently ignored (the files path passes 600s by default, but a custom
request_timeout would not have reached this upload). Regression test asserts
the resolved timeout reaches the request (mutation-verified).
Revert the orjson swap in the batch transform: importing orjson at module load
in this core-path file broke `import litellm` on environments without orjson
(the Windows import test). Back to stdlib json; the upload leg dominates large
uploads anyway, so the transform-side win was marginal.
Fix import ordering in llm_http_handler.py (I001) introduced by the new imports.
* fix(vertex_ai/files): stream batch upload to GCS instead of staging to a temp file
Addresses a disk-exhaustion concern: staging the full transformed batch body to
a local temp file before the GCS request meant an authenticated user could fill
the proxy's temp volume with large concurrent uploads (on top of Starlette's
input spool).
GCS's simple/media upload accepts chunked transfer-encoding, so stream the
transform straight to the single media request instead. Each block is produced
on a worker thread (the transform never runs on the event loop) and sent
chunked, so the body is neither buffered in memory nor written to disk, and the
upload is still one continuous request (no per-chunk round-trips, no 499). Drops
the temp-file staging, the tempfile/IO imports, and Content-Length computation.
Regression test asserts the upload streams (chunked transfer-encoding, no
Content-Length) and creates no temp file; mutation-verified that reintroducing
staging fails it.
(cherry picked from commit 85840aef51)
* fix(proxy/client): redact api key from key/info client error messages
The keys management client builds GET /key/info?key=<key> and lets the
requests HTTPError propagate. str(HTTPError) renders the failing request URL
verbatim ("... for url: .../key/info?key=sk-..."), so any caller that logs the
exception leaks the full key; the 401 branch leaked the same way through
UnauthorizedError(str(orig_exception))
Redact both branches with the existing redact_secrets helper so the
secret-bearing query param is scrubbed to ?REDACTED while the status code,
reason, and response object are preserved. Server-side responses already mask
the key, so this closes the remaining client-side surface
* fix: preserve key info unauthorized response
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 71ee1a852a)
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 56825926af)
An oauth2 MCP server with delegate_auth_to_upstream=true never prompted the
user to sign in. On an unauthenticated initialize the gateway answered locally
(200, no tools) and emitted no WWW-Authenticate, so clients like Claude Desktop
either connected empty or hit "OAuth probe timeout after 10000ms".
#30124 added a bare `continue` in _raise_preemptive_401_for_unauthenticated_servers
to stop sending LiteLLM's gateway authorization_uri challenge for delegate-auth
servers, expecting the upstream to emit its own challenge. On initialize the
gateway never probes upstream, so no challenge ever reached the client.
Replace the `continue` with a preemptive 401 carrying the proxied
resource_metadata (RFC 9728) challenge, the same form passthrough servers and
MCPUpstreamAuthError already use. This keeps #29770 fixed (still no
authorization_uri) while restoring the upstream PKCE sign-in prompt.
The App Router migration moved pages to deeper path segments and the proxy
can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local
logo asset paths were emitted without the server root prefix, so they resolved
off the origin root and 404'd. Route every local logo src through a single
resolver that prefixes the live server root path and leaves external URLs
untouched, fixing provider, guardrail, vector store, callback, MCP and
audit-log logos at any route depth and root path.
The migrated /ui/api-keys route gates rendering on useAuthorized() but read userID from the AuthContext (useAuth), which hydrates asynchronously. On a hard refresh or deep link the route could render UserDashboard before AuthContext had populated userID, so UserDashboard hit its `userID == null` guard and showed "User ID is not set". The legacy index page avoided this by gating on AuthContext's own authLoading; the migration switched the gate to useAuthorized without aligning the identity source.
Read identity from useAuthorized (a synchronous cookie decode) so userID is populated whenever the route is authorized. useAuth is kept only for the backfill setters UserDashboard still expects, until the planned AuthContext consolidation removes them.
Refs LIT-3687
* feat(sandbox): add e2b code execution primitive
Add a provider-agnostic code execution primitive that runs model-generated
code in an isolated sandbox and returns the output, with e2b as the first
backend over raw httpx (no SDK dependency).
Public API: litellm.acode_interpreter_tool (ephemeral create -> run -> delete)
plus the low-level lifecycle litellm.acreate_sandbox / arun_code /
adelete_sandbox. Each is @client-decorated so operations are logged like
litellm.asearch. Backends implement BaseSandboxConfig; resolved via
ProviderConfigManager.get_provider_sandbox_config.
* fix(sandbox): address review feedback and CI gates
- document e2b provider in provider_endpoints_support.json and add a sandbox endpoint definition
- regenerate dashboard CallTypes after the sandbox call-type additions
- guard explicit timeout=0 instead of coercing it to the default
- require a ContainerHandle access token before running code; reject bare-id runs
- return False on a 404 delete now that the shared http handler raises for status
- skip non-JSON NDJSON lines and cap streamed output to bound memory
- move the real-network integration tests out of tests/test_litellm into tests/integration/sandbox
* fix(sandbox): satisfy strict ruff gate and scope star-exports
- modernize annotations in the new sandbox modules to PEP 585/604 (list/dict,
X | None) and drop the now-unnecessary quoted forward refs so the strict-rule
budget delta for UP006/UP037/UP045 returns to zero
- add __all__ to litellm/sandbox/main.py so 'import *' only re-exports the four
public entrypoints instead of leaking module-level imports
* fix(sandbox): drop quotes on sandbox config return annotation
utils.py uses 'from __future__ import annotations', so the quoted forward ref
tripped UP037; the unquoted union is lazily evaluated and keeps the strict-rule
delta at zero
* chore(sandbox): re-trigger automated review after addressing feedback
* test: point router/completion/triton tests at the local fake OpenAI endpoint
The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app)
takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted
proxy configs onto a job-local fake server but left these in-Python api_base
literals pointing at the dead host, so litellm_router_testing, local_testing_part1,
local_testing_part2 and llm_translation_testing still fail with a 404
"Application not found" when Railway is down
Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190)
through a shared helper, auto-start the canned server from the local_testing and
llm_translation conftests when nothing is already serving, and extend the server
with a Triton embeddings route and a slow-endpoint delay so the triton and
latency-timeout tests run fully offline. The deliberately broken fallback URL is
left as-is so fallback handling still has a failing upstream
* fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI
* fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target
* fix(tests): keep fake OpenAI mock alive across xdist workers
ensure_fake_openai_endpoint registered atexit on the worker that spawned
the subprocess, so under -n 4 the first worker to drain its queue would
terminate the shared mock while siblings were still hitting it. Detach
the child via start_new_session and drop the per-worker teardown; reuse
on /health handles re-runs and CI containers clean up themselves
* fix(watsonx): wrap string embedding input in array for WatsonX API
WatsonX text/embeddings expects inputs as []string; OpenAI clients often send a single string.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style(watsonx): format watsonx embed transformation for black
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(watsonx): avoid UP006 in embed transformation strict lint gate
Use list[str] and branch-based input normalization instead of List and cast
so the watsonx embedding change does not add strict ruff UP006 violations.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
