Root cause: PR #30867 removed request-time os.environ/ expansion in
BaseAWSLLM.get_credentials. That is only safe if config-load pre-resolves
os.environ/ refs so the value reaching get_credentials is already the real
secret. The YAML config path has always done this. The DB-load path
(ProxyConfig._resolve_db_litellm_param) only re-expanded keys in a hardcoded
whitelist (_DB_LITELLM_PARAM_ENV_REF_KEYS) plus short-circuited env-ref
resolution entirely for team-scoped rows. PR #32256 extended that whitelist
to 18 keys to unblock a customer whose Bedrock model with aws_role_name:
os.environ/BEDROCK_ASSUME_ROLE_ARN broke on v1.90+, but the whitelist is
structurally fragile: every future auth field breaks the same way until
someone remembers to add it
Fix: remove the whitelist and the team-scope short-circuit. The DB-load
resolver now expands os.environ/ on every string field, matching the YAML
path. Trust boundary stays on the write side: only PROXY_ADMIN can create
team_id=None rows, only team admins of a team can create rows scoped to
that team, and the request-body vector is still blocked by
_BANNED_REQUEST_BODY_PARAMS. Team-scoped rows now resolve env refs — this
is a deliberate LIT-3831 threat-model expansion trusting team admins for
env-var reads
Regression tests in tests/test_litellm/proxy/proxy_server/test_proxy_config.py:
- test_ProxyConfig__add_deployment_resolves_env_refs_after_db_decrypt pins
admin-scoped rows resolve every field (previously api_base stayed literal)
- test_ProxyConfig__add_deployment_resolves_team_env_refs pins team rows
resolve env refs (previously stayed literal)
- test_ProxyConfig__add_deployment_resolves_env_refs_on_arbitrary_field pins
the no-whitelist invariant against a made-up field name
- test_ProxyConfig__add_deployment_resolves_env_refs_for_aws_bedrock_auth_params
(from #32256) still passes
- Path B counterparts (decrypt_model_list_from_db) mirror the above
Left as followups (not fixed here):
- /model/info and /v2/model/info still echo resolved values for fields not
in the current pop-list (aws_role_name, aws_sts_endpoint, api_base, etc.).
Fix is to extend remove_sensitive_info_from_deployment; separate PR
- Master-key rotation reads DB rows via decrypt_model_list_from_db which
now resolves universally, so rotation collapses env-refs into hardcoded
values. Pre-existing bug for the 6 previously-whitelisted fields; wider
surface after this PR. Separate PR
(cherry picked from commit 5862be3e79)
* fix(anthropic): require caller api_key and SSRF-validate api_base in advisor tool
The advisor_20260301 interceptor honored a caller-supplied api_base once
allow_client_side_credentials was enabled, even without a caller-supplied
api_key. AnthropicModelInfo.get_auth_header() then fell back to the proxy's
own ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN, so the server's real credentials
plus the conversation history got sent to a caller-chosen destination
_resolve_advisor_credentials() now only honors api_base alongside a
non-empty caller-supplied api_key, requires the https scheme, and validates
api_base via validate_url() before use, mirroring check_complete_credentials
in auth_utils.py. https is required because validate_url only DNS-pins the
connection for http; for https with TLS verification on it returns the URL
unchanged and relies on certificate validation to block DNS rebinding
* fix(anthropic): also reject advisor api_base when ssl_verify is disabled
validate_url only DNS-pins the connection for http, or for https with
litellm.ssl_verify disabled; the previous https-only check missed the
ssl_verify=False case, where validate_url's rewritten URL was still being
discarded, per Greptile's review of this PR. Reject api_base outright when
ssl_verify is False so the discarded rewrite can no longer matter
(cherry picked from commit 07b9ea8c3b)
The security fix in 34e9be1ba7 removed turn_off_message_logging from
_supported_callback_params to stop callers bypassing global redaction via
the request body. That also killed the documented admin-only per-key or
per-team override because both flows resolve through the same allowlist
in initialize_standard_callback_dynamic_params.
Put turn_off_message_logging back in _supported_callback_params so an
admin-configured metadata.logging[].callback_vars.turn_off_message_logging
survives into StandardCallbackDynamicParams and can override the global
setting for that key or team, as documented at
docs/proxy/team_logging#disableenable-message-redaction.
Consolidate the metadata traversal so the extractor and the proxy strip
walk the same set of client-controllable slots. iter_client_callback_metadata_dicts
in litellm_core_utils/initialize_dynamic_callback_params.py is the single
source of truth for metadata, litellm_metadata, and litellm_params.metadata;
_strip_client_message_redaction_opt_out imports it so a future addition
to one side automatically reaches the other. The extractor iterates the
helper in reversed order so litellm_params.metadata keeps overriding
metadata, matching the pre-refactor merge precedence.
Client bypass stays blocked. Restoring the field re-enrolls it in the
auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from
_supported_callback_params via _build_banned_observability_params), so
client submissions at the top level, inside metadata, or inside a
JSON-string litellm_metadata all 401 at ingress. is_request_body_safe
also now descends into litellm_params.metadata for the same 401 defense
against the nested-body attack vector, matching how the metadata and
litellm_metadata slots are handled. _strip_client_message_redaction_opt_out
runs after the litellm_metadata JSON parse and before the admin callback_vars
unpack, so admin values survive while any leftover client-supplied
opt-out is dropped when global redaction is on and the key or team
lacks allow_client_message_redaction_opt_out.
Flip the two dynamic-param e2e tests added by the security fix to
reflect the restored override behavior, keeping the invariant that
proxy client bypass is stopped by the auth layer 401 above.
Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 8e6098adc3)
AWS auth parameters in the Bedrock and SageMaker path could be expanded against
the process environment when credentials were built. Config-sourced references
are already expanded at load time, so restrict expansion to that path: a
reference still present at request time is treated as caller-supplied input and
is left as-is, and the web-identity helper rejects environment-variable
references before resolving the token.
Also rework the ambient AWS_* fallback as a single pass that pairs each value
with its own env-var name, fixing a latent index misalignment that left
AWS_EXTERNAL_ID unresolved.
Adds regression tests covering the resolution behavior.
(cherry picked from commit 4ef7d0815b)
* fix: validate proxy request body and nested fields
Ensure caller-supplied request fields cannot override server-side deployment
configuration, and apply request-body validation consistently to nested
structures. Adjusts router kwarg handling and client-side credential handling
for base-url overrides
* test: cover router strip ordering and advisor clientside credential gate
* fix: clear deployment credentials on client base-url override
When a request overrides api_base/base_url, recompute the deployment's
litellm_params (clearing the deployment's own api_key) and drop the cached
client built for the original endpoint, so the deployment credential is not
reused for the client-supplied endpoint. Adds regression tests that assert the
credentials actually forwarded to litellm.completion/acompletion.
* fix(proxy): require api_key alongside api_base override
A request that overrides api_base/base_url but supplies no api_key still
left the proxy carrying a server credential: once the override clears the
banned-param opt-in, the provider re-resolves a key from the environment
(api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in
main.py) and forwards it to the caller-controlled URL. Popping the
deployment api_key only changed which server key leaked.
Gate is_request_body_safe so a permitted api_base/base_url override must
also carry a non-empty caller api_key; reject otherwise. The env
resolution in main.py is left as the provider boundary.
* fix(proxy): extend request-body banlist with five additional credential and session targeting fields
Yuneng's review found five deployment-owned request-body params still missing
from the denylist and the router strip set. Each lets a caller reach the
operator's provider credentials or retarget the outbound request:
aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region
retarget the OCI request, litellm_credential_name selects any server-loaded
credential by name with no ownership check, and runtimeSessionId resumes a
Bedrock AgentCore runtime session (AWS does not enforce session-to-user
mapping, so this is a cross-tenant session-resume vector).
Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to
_DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and
SDK direct calls are unaffected: the banlist gates the request body only, and
the router strip drops caller kwargs, never deployment["litellm_params"].
* test: rename arbitrary canary values in security tests to neutral placeholders
* fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip
P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS
(litellm_embedding_config, extra_body) for the banned-param check but not for
the api_key co-presence check, so a base override smuggled into one of those
nested dicts cleared the client-side-credentials opt-in without a paired
api_key and let the provider re-resolve a server credential from the
environment. Run _check_base_override_has_api_key on each nested config dict
too, so the requirement applies wherever a base override is permitted.
P1-B: the deployment-owned credential strip in the Router runs unconditionally
on every _completion/_acompletion, which is security-correct but silently
drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a
single warning (key names only, never values) when the strip removes a
non-empty value, so the backwards-incompatible behavior is visible without
gating the strip on a context flag that does not exist.
* fix(proxy): apply api_key co-presence to tool-entry base override
is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and
tools[]); the previous commit extended the api_key co-presence rule to root
and nested config dicts but not to tool entries. With
allow_client_side_credentials enabled, a tool entry carrying api_base/base_url
and no paired api_key cleared the gate, letting a provider interceptor fall
back to a server-side credential for a caller-controlled URL. Add the same
_check_base_override_has_api_key call to each tool dict and its nested function
dict, mirroring the symmetry already applied to the nested config keys. The
rule is unchanged: api_key must live in the same dict as the base override it
accompanies.
* test(proxy/auth): require paired api_key under extra_body opt-in
* fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running
* fix(advisor): narrow proxy-import guard to ImportError-family
* fix(router): gate api_key clear on base override behind litellm.proxy_is_running
* test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture
* style: use built-in generics in PR-added type annotations
* revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate
* revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes
* style: black-format advisor orchestration test
(cherry picked from commit 1667b8f740)
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 56825926af)
Moves the runtime dependencies forward on this stable line: cryptography
46.0.7 to 48.0.1 and aiohttp 3.13.5 to 3.14.1, with vcrpy moved to 8.2.1
alongside aiohttp so the test harness can still import aiohttp 3.14. The
relock also brings starlette to 1.3.1, python-multipart to 0.0.32,
pydantic-settings to 2.14.2 and pypdf to 6.13.3. mlflow moves to 3.14.0
because mlflow 3.11.1 pins cryptography below 47, so it cannot coexist with
cryptography 48; it stays within the existing mlflow<4.0 range.
The lock change is confined to these ten packages; nothing else in the graph
moved.
Re-pins LITELLM_BUILD_IMAGE and LITELLM_RUNTIME_IMAGE across all 6 Dockerfiles
from the prior digests (openssl 3.6.2-r3) to the current chainguard wolfi-base
digest c61ac691 (openssl 3.6.3-r2, >= the fixed 3.6.3-r0). The runtime stage is
the shipped image, so the runtime digest is what actually resolves the
customer-facing CVE; the build image is bumped too for hygiene. Two Dockerfiles
tracked a second equally-stale digest; both are unified onto the patched one.
(cherry picked from commit fda08dd727)
Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.
- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
async). Large agentic tool-use / thinking streams can make assembly re-raise
as APIError from inside the except-StopIteration handler, where the sibling
except does not catch it, so it escaped __next__/__anext__ and dropped the
request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
stream_chunk_builder returns None or raises, rebuild usage from the
message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
(it is read from there, not from kwargs), matching the gemini/cohere/openai
handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
operators don't have to reverse-map spend back to a key
(cherry picked from commit b24b964e04)
A streaming request that breaks mid-flight, for example on a mid-stream read
timeout, still bills the provider for the chunks already delivered, yet the proxy
recorded that interrupted request as a zero-spend failure. An earlier revision
logged the recovered partial usage through the success path, which mislabeled a
failed request as a success and produced a misleading spend row
This recovers the partial usage where the failure is actually logged. The
streaming handler assembles the usage from the chunks seen so far and stashes it,
with its cost, on the logging object before firing the failure handlers. The
proxy failure hook lifts that usage and cost onto request_data before the
non-serialisable logging object is popped, and the spend-log writer records the
real partial spend on the failure row instead of a hardcoded zero;
get_logging_payload honors the recovered usage for the token columns and
_failure_handler_helper_fn preserves the recovered cost so the non-DB failure
loggers stay consistent
A request that recovers via a successful fallback is unaffected: the failure hook
only fires when the whole request fails, so the fallback's combined-usage success
row stays the single source of truth and there is no double counting
Resolves LIT-3825
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
(cherry picked from commit 4847fa5dd5)
* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel
A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.
* refactor(ui): centralize no-mcp-servers sentinel in a shared constant
The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.
* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes
Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.
(cherry picked from commit 19a29e0579)
* fix(guardrails): return 400 not 500 when AIM blocks a request
AIM guardrail blocks raised a bare HTTPException whose type and param
serialized as the literal string "None", which broke OpenAI-SDK error
parsing for downstream consumers. Switching AIM to raise a ProxyException
surfaced a second bug: the shared error funnel re-derived the HTTP status
from a nonexistent status_code attribute and downgraded the 400 to a 500.
The funnel now honors an already-normalized ProxyException rather than
rebuilding it, and ProxyException is excluded from llm_exceptions alerting
so a content-policy block no longer pages on-call as an LLM API failure
Resolves LIT-3751
* fix(guardrails): route all AIM rejection paths through ProxyException
The block-action fix left two AIM rejection paths raising a bare
HTTPException: the multimodal anonymize rejection and the output-side
block. Both serialized type and param as the literal string "None", the
same malformed shape the block fix removed. Funnel all three through a
shared _rejection helper so they return a conformant OpenAI error body.
The output block carries content_policy_violation; the multimodal
rejection stays a plain invalid_request_error because it is a usage
error, not a policy violation
Resolves LIT-3751
* fix(guardrails): record AIM ProxyException blocks in failure logs
Switching AIM blocks from HTTPException to ProxyException made
_is_proxy_only_llm_api_error return False for them, so
_handle_logging_proxy_only_error was skipped and the blocked prompt was
dropped from the configured failure loggers. Classify ProxyException as a
proxy-only error alongside HTTPException so guardrail blocks are recorded
again, matching the prior behavior. The llm_exceptions alert suppression
is a separate check and stays in place
Resolves LIT-3751
* style(guardrails): use str | None over Optional[str] in AIM _rejection
* style(guardrails): collapse AIM _rejection signature per black
(cherry picked from commit b5fcd859be)
* fix(integrations): cap Anthropic cache_control injection at 4 blocks
Respect Anthropic's 4 cache_control breakpoint limit by counting client-supplied blocks, skipping messages that already carry cache_control, and stopping further auto-injection once the limit is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(integrations): reserve cache slot for tool_config and short-circuit cap
Address review feedback on the cache_control cap: break out of the injection loop before resolving target indices once the limit is reached, and reserve one of the four breakpoint slots when a tool_config injection point is present so the cachePoint appended by the Bedrock transform does not push the total past Anthropic's limit.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit fc9d789d24)
* fix: stop use_chat_completions_api flag from leaking into provider request body
use_chat_completions_api is a LiteLLM control flag that forces the
/responses -> /chat/completions bridge. It was missing from
all_litellm_params, so get_non_default_completion_params treated it as a
model-specific param and forwarded it to the upstream provider. A
model-level "use_chat_completions_api: true" in the proxy config therefore
reached the chat-completions path and was rejected by strict providers
(OpenAI/Anthropic) with HTTP 400 for an unknown body field.
Register it as a known internal param so it is stripped on every path
(completion, the responses bridge that calls litellm.completion, and
filter_out_litellm_params).
Adds a regression test driving litellm.completion() with a mocked OpenAI
client that asserts the flag never reaches the request body.
* test: clarify extra_body assertion in use_chat_completions_api leak test
Replace the misleading 'not in ... or {}' precedence idiom with an explicit
parenthesized guard that also handles extra_body being None.
(cherry picked from commit 65b6e04da6)
* fix(guardrails): stop re-initializing DB guardrails on every poll
InMemoryGuardrailHandler._has_guardrail_params_changed compared the
in-memory LitellmParams against the raw dict loaded from the DB. The
in-memory side carries every field default and coerces enums via
model_dump(), while the DB side only holds the keys originally stored,
so the two shapes never compared equal and the guardrail was rebuilt on
every poll cycle.
Each rebuild created a fresh instance, but delete_in_memory_guardrail
only removed the old callback from litellm.callbacks. Request handling
promotes guardrail callbacks into the success/failure/async lists, so
the previous instance stayed referenced there and instances accumulated.
Normalize both sides through LitellmParams(...).model_dump() before
diffing, and purge the callback from every callback list on delete.
* refactor(guardrails): narrow params-normalization fallback to ValidationError
The comparison normalizer caught a bare Exception and silently fell back
to the raw dict, which hid the cause and quietly degraded the affected
guardrail back to re-initializing on every poll. Catch only the
ValidationError that LitellmParams construction can raise, log a warning
so the offending row is diagnosable, and let any other error surface
instead of being swallowed.
* refactor(callbacks): add remove_callback_from_all_lists helper to manager
Move the knowledge of which callback lists a callback can be promoted
into out of the guardrail registry and into LoggingCallbackManager, where
the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail
now delegates to the new helper instead of iterating the lists itself.
(cherry picked from commit 9fa74ad8b4)
* fix(guardrails): run pre_call hook once for model-level guardrails
A CustomGuardrail attached to a deployment via litellm_params.guardrails
gets its async_pre_call_hook invoked twice per request: once by the proxy
pre-call loop and again by async_pre_call_deployment_hook after the router
spreads the model-level guardrails into the top-level request kwargs.
Record in request metadata that the proxy pre-call loop already ran a given
guardrail, and have the deployment hook skip it when the marker is present.
Direct-SDK usage never runs the proxy loop, so the deployment hook stays the
sole invocation there and still fires exactly once.
The marker key is stripped from untrusted caller metadata so a request body
cannot suppress a model-only guardrail by pre-seeding it.
* fix(guardrails): mark pre_call dedup on the post-hook request data
Record the exactly-once marker after async_pre_call_hook runs, on the data
object that flows downstream, rather than before it. A guardrail whose hook
returns a brand-new request dict (instead of mutating or spreading the one it
received) would otherwise discard the marker, letting the deployment hook
re-run the guardrail a second time.
(cherry picked from commit 4faeabc254)
* fix(proxy): populate access_via_team_ids on /v1/model/info
Team metadata enrichment previously only ran on /v2/model/info with
include_team_models=true, leaving /v1/model/info without
access_via_team_ids for project model-picker flows.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(dashboard): sync OpenAPI schema for /v1/model/info query params
Add include_team_models and teamId to the generated schema for /model/info
and /v1/model/info after the proxy endpoint gained team-access filtering.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): always return direct_access on /v1/model/info
Set direct_access to true or false on every enriched model so clients
can filter without treating a missing field as ambiguous.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(proxy): fail fast when teamId is set without a connected DB on /v1/model/info
Raise the db_not_connected error before building, enriching, and translating the model list instead of after, so a teamId query against a proxy with no database no longer wastes the full enrichment pipeline.
* fix(proxy): fail fast when include_team_models is set without a database
include_team_models=True relies on _populate_team_access_on_models to set
direct_access/access_via_team_ids, which only runs when a database is connected.
Without one, _filter_models_to_user_accessible discarded every model and the
endpoint returned an empty list with HTTP 200. Mirror the teamId guard so the
request fails fast with a clear db_not_connected error before any model-list work.
* fix(proxy): populate direct_access on single-model /model/info lookup
The /v1/model/info list path populates model_info.direct_access (and
access_via_team_ids) when a database is connected, but the
litellm_model_id single-model lookup returned early without it. This
made the two endpoints disagree, breaking the parity assertion in
test_get_specific_model. Run the same population on the single-model
path so both responses match.
* fix(proxy): apply no-DB fast-fail before litellm_model_id branch
The teamId/include_team_models no-DB guard sat after the litellm_model_id
early return, so ?litellm_model_id=X&teamId=Y with no DB returned 200 with
unpopulated access fields instead of the 500 raised on every other path.
Move the guard ahead of the branch so the fast-fail is uniform.
* fix(proxy): apply teamId/include_team_models filters on single-model lookup
The litellm_model_id early-return branch in model_info_v1 populated the
team access fields but returned before the teamId and include_team_models
filters ran, so a single-model lookup surfaced the deployment regardless
of team access when the DB was connected. Run both filters on the
single-model list before returning so the documented query params behave
the same with and without litellm_model_id.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adaptation: the generated ui schema.d.ts hunk is dropped (absent on this line,
regenerated at UI build, no Python dependency). proxy_server.py and the test apply
verbatim once the model_info_v1 BYOK pipeline is present from #29731 and #30025.
(cherry picked from commit 7d1f68e72a)
* fix(proxy): align /v1/model/info with router deployments
Return router model_list entries (including team-scoped models) with team
access metadata instead of wildcard-expanded names from get_complete_model_list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): gate v1 team filter and honor key allowlists
Only apply get_all_team_and_direct_access_models for admin or user-bound
keys, then intersect with key/team model restrictions to avoid empty lists
for service tokens and metadata leaks for restricted keys.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): skip v1 team filter when user row is missing
Require a DB-backed user before applying team-access filtering on
/v1/model/info, and skip the trailing filter in get_all_team_and_direct_access_models
when user context cannot be resolved.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Revert "fix(proxy): skip v1 team filter when user row is missing"
This reverts commit 74e1fbd77a.
* fix(proxy): restore legacy v1 model access filtering
Keep /v1/model/info on key/team allowlists instead of DB team-membership
filtering, while still listing router deployments for team-scoped models.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): drop A2A agent entries from public /v1/model/info list
* fix(proxy): scope team BYOK rows on /v1/model/info to caller's teams
Listing the full router model_list let any authenticated key without
explicit model restrictions enumerate other teams' BYOK deployments
(public name, team_id, api_base) via /v1/model/info. Reuse the existing
_get_caller_byok_team_scope check so non-admin callers only see global
deployments plus their own team's BYOK rows; admins keep the full view.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 6068bb7781)
* fix(proxy): stop team model name corruption on edit (#28382) (#29001)
Team-scoped ("Team-BYOK") models store an internal routing key
model_name_{team_id}_{uuid} in the model_name column and the user-facing
name in model_info.team_public_model_name. The internal name leaked into
/v1, /v2, and /model/info responses; the dashboard bound its edit form to
it, so any non-rename save (e.g. a TPM tweak) PATCHed the internal name
back. The update path then treated it as a rename, overwriting
team_public_model_name and rewriting the team's models[] ACL with the
mangled string -- breaking team key calls with team_model_access_denied.
Two-layer fix:
- Read path (root cause): add _translate_model_name_for_response and apply
it in model_info_v2 and _get_proxy_model_info so /v1, /v2, and
/model/info surface the public name for team-scoped rows. The DB column
and router index keep the internal name as the routing key; this is a
presentation-layer swap on a shallow copy (never mutates input).
- Write path (defense in depth): harden _get_public_model_name so a value
matching the internal shape, or a no-op against the current DB column,
is never treated as a rename -- for both the top-level model_name and an
explicit model_info.team_public_model_name.
Tests: regression for the reported scenario, full branch coverage of
_get_public_model_name, two internal-shape guard cases, an end-to-end
PATCH through _update_team_model_in_db (asserts the team ACL is untouched),
and four response-translation cases. 60 passed (model management),
181 passed (proxy server).
* fix(ui): key Agent Builder agent selection on model_info.id (#29729)
* fix(ui): key Agent Builder agent selection on model_info.id
Once team-scoped BYOK models can share a public name (the backend now
returns the public name on /model/info instead of the internal routing
key), selecting agents by model_name collides. Key selection, create,
update and delete on the stable model_info.id instead, falling back to
model_name only for config-defined agents that have no id.
* fix(ui): add name-match fallback to post-create agent selection
If the just-created agent's id is not yet present in the re-fetched
list, try matching by name before falling back to the first agent.
Addresses greptile review on #29729.
---------
Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com>
(cherry picked from commit 56aa55b991)
* chore(deps): bump aiohttp to 3.14.1 and vitest to 3.2.6
Lockfile-only bump for aiohttp (3.13.5 -> 3.14.1, within the existing
pyproject constraint) and dashboard devDependency bumps for vitest,
@vitest/coverage-v8, @vitest/ui (3.2.4 -> 3.2.6) plus transitive
brace-expansion (5.0.5 -> 5.0.6). Clears the currently published
advisories flagged by osv.dev against uv.lock and the dashboard
lockfile. Verified: 154 custom_httpx unit tests and all 3943 dashboard
vitest tests pass; live proxy completion and streaming calls succeed on
the bumped venv
* chore(deps): raise aiohttp floor to 3.14.0
The lockfile bump alone only protects environments built from uv.lock.
Raising the pyproject floor extends the same minimum to package
consumers installing litellm from PyPI, and prevents a future lockfile
regeneration from resolving below 3.14.0
* Revert "chore(deps): raise aiohttp floor to 3.14.0"
This reverts commit d6c1c9dc0c.
* revert(deps): roll back aiohttp to 3.13.5
vcrpy is incompatible with aiohttp >= 3.14 (the aiohttp_stubs module
imports a symbol removed in 3.14) and the upstream fix is merged but
unreleased, so every cassette-based test suite fails on 3.14. Hold
aiohttp at 3.13.5 until a vcrpy release ships; the vitest and
brace-expansion bumps stay
* chore(deps): bump pypdf to 6.13.1 and tornado to 6.5.7
Lockfile-only bumps clearing the advisories published for both since
this branch was opened
* chore(deps): add regression guards for the bumped versions
Raise the pypdf floor to 6.12.0 (direct dependency, applies to package
consumers too) and add uv constraint-dependencies for the transitive
pins: tornado >= 6.5.6, and aiohttp held in [3.13.5, 3.14) so a lockfile
regeneration can neither fall back below the current version nor move
onto 3.14 while vcrpy is incompatible. Constraints live in [tool.uv]
and only affect this repo's resolution, not published metadata.
Verified: uv lock -P with each out-of-range version fails to resolve;
in-range resolutions unchanged (pypdf 6.13.1, tornado 6.5.7,
aiohttp 3.13.5)
Backport handling (stable/1.88.x): the manifest hunks (pyproject pypdf floor 6.10.2->6.12.0
plus the [tool.uv] tornado/aiohttp constraints, and the package.json vitest 3.2.4->3.2.6 bumps)
are taken as-is; the staging lockfiles are not. uv.lock and package-lock.json are regenerated on
this line so the closure stays minimal: uv.lock moves only pypdf 6.10.2->6.13.2 and tornado
6.5.5->6.5.7 (aiohttp held at 3.13.5 by the new constraint), and package-lock.json moves vitest
and @vitest to 3.2.6. brace-expansion reached 5.0.6 transitively on staging but resolves to 5.0.5
on this line's tree, so a brace-expansion 5.0.6 override is added to clear the same advisory. No
version bump: 1.88.2 is bumped but unreleased, so these ride the pending 1.88.2.
(cherry picked from commit d96ab467f1)
The grace-period branch assigned the recursive get_data result (a
finished LiteLLM_VerificationTokenView) back into the variable that the
combined-view dict normalization then subscripts, raising TypeError on
every request made with a rotated key inside its grace window; auth
surfaced that as a 401. Return the recursive result directly instead.
Regression test drives the full get_data flow: old hash misses the view,
deprecated table resolves to the active token, and the call must return
the view object
(cherry picked from commit 5047eaf7f0)
Targeted subset of staging commit cfcdf8714a (#30202): only the
anthropic_passthrough_logging_handler.py hardening hunks and their four
tests are taken; the rest of that staging batch is intentionally excluded.
Backport adaptation (stable/1.88.x): the new TestBuildCompleteStreamingResponseRobustness
class is unioned with this line's existing test_parity_*/test_collapse_* tests, which
the 1.84.x base of this commit did not have. Source handler applies verbatim.
(cherry picked from commit cfcdf8714a)
(cherry picked from commit 973c7eb8d6)
Adaptation: in the test file, #30160's six own tests (cost-calculation model
resolution + extract_model) are taken; the trailing
test_passthrough_logging_sets_response_cost_with_server_tool_use_dict, which is
diff context present at the PR's base but not on this line, is dropped. Source
handler applies verbatim.
(cherry picked from commit 1828a7c6f0)
Adaptation: this line's auth_exception_handler.py does not import
seed_request_identity. The OTEL request-identity feature has its call site in
user_api_key_auth.py here; the second call site staging added in the exception
handler is not on this line. The picked tests mock
auth_exception_handler.seed_request_identity to keep the failure path off OTEL,
so those four patch entries (three in test_auth_exception_handler.py, one in
the test_user_api_key_auth.py builder helper) are removed because the symbol is
absent and the handler never calls it here. The 503/401 behavioral assertions
are unchanged and pass. Separately, the one-line context hunk that reformats an
assert inside test_auto_register_passes_validated_org_context_to_generated_key
is dropped; that test is not present on this line. Source files apply verbatim.
(cherry picked from commit da9d64b4de)
