Root cause: PR #30867 removed request-time os.environ/ expansion in
BaseAWSLLM.get_credentials. That is only safe if config-load pre-resolves
os.environ/ refs so the value reaching get_credentials is already the real
secret. The YAML config path has always done this. The DB-load path
(ProxyConfig._resolve_db_litellm_param) only re-expanded keys in a hardcoded
whitelist (_DB_LITELLM_PARAM_ENV_REF_KEYS) plus short-circuited env-ref
resolution entirely for team-scoped rows. PR #32256 extended that whitelist
to 18 keys to unblock a customer whose Bedrock model with aws_role_name:
os.environ/BEDROCK_ASSUME_ROLE_ARN broke on v1.90+, but the whitelist is
structurally fragile: every future auth field breaks the same way until
someone remembers to add it
Fix: remove the whitelist and the team-scope short-circuit. The DB-load
resolver now expands os.environ/ on every string field, matching the YAML
path. Trust boundary stays on the write side: only PROXY_ADMIN can create
team_id=None rows, only team admins of a team can create rows scoped to
that team, and the request-body vector is still blocked by
_BANNED_REQUEST_BODY_PARAMS. Team-scoped rows now resolve env refs — this
is a deliberate LIT-3831 threat-model expansion trusting team admins for
env-var reads
Regression tests in tests/test_litellm/proxy/proxy_server/test_proxy_config.py:
- test_ProxyConfig__add_deployment_resolves_env_refs_after_db_decrypt pins
admin-scoped rows resolve every field (previously api_base stayed literal)
- test_ProxyConfig__add_deployment_resolves_team_env_refs pins team rows
resolve env refs (previously stayed literal)
- test_ProxyConfig__add_deployment_resolves_env_refs_on_arbitrary_field pins
the no-whitelist invariant against a made-up field name
- test_ProxyConfig__add_deployment_resolves_env_refs_for_aws_bedrock_auth_params
(from #32256) still passes
- Path B counterparts (decrypt_model_list_from_db) mirror the above
Left as followups (not fixed here):
- /model/info and /v2/model/info still echo resolved values for fields not
in the current pop-list (aws_role_name, aws_sts_endpoint, api_base, etc.).
Fix is to extend remove_sensitive_info_from_deployment; separate PR
- Master-key rotation reads DB rows via decrypt_model_list_from_db which
now resolves universally, so rotation collapses env-refs into hardcoded
values. Pre-existing bug for the 6 previously-whitelisted fields; wider
surface after this PR. Separate PR
(cherry picked from commit 5862be3e79)
The security fix in 34e9be1ba7 removed turn_off_message_logging from
_supported_callback_params to stop callers bypassing global redaction via
the request body. That also killed the documented admin-only per-key or
per-team override because both flows resolve through the same allowlist
in initialize_standard_callback_dynamic_params.
Put turn_off_message_logging back in _supported_callback_params so an
admin-configured metadata.logging[].callback_vars.turn_off_message_logging
survives into StandardCallbackDynamicParams and can override the global
setting for that key or team, as documented at
docs/proxy/team_logging#disableenable-message-redaction.
Consolidate the metadata traversal so the extractor and the proxy strip
walk the same set of client-controllable slots. iter_client_callback_metadata_dicts
in litellm_core_utils/initialize_dynamic_callback_params.py is the single
source of truth for metadata, litellm_metadata, and litellm_params.metadata;
_strip_client_message_redaction_opt_out imports it so a future addition
to one side automatically reaches the other. The extractor iterates the
helper in reversed order so litellm_params.metadata keeps overriding
metadata, matching the pre-refactor merge precedence.
Client bypass stays blocked. Restoring the field re-enrolls it in the
auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from
_supported_callback_params via _build_banned_observability_params), so
client submissions at the top level, inside metadata, or inside a
JSON-string litellm_metadata all 401 at ingress. is_request_body_safe
also now descends into litellm_params.metadata for the same 401 defense
against the nested-body attack vector, matching how the metadata and
litellm_metadata slots are handled. _strip_client_message_redaction_opt_out
runs after the litellm_metadata JSON parse and before the admin callback_vars
unpack, so admin values survive while any leftover client-supplied
opt-out is dropped when global redaction is on and the key or team
lacks allow_client_message_redaction_opt_out.
Flip the two dynamic-param e2e tests added by the security fix to
reflect the restored override behavior, keeping the invariant that
proxy client bypass is stopped by the auth layer 401 above.
Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 8e6098adc3)
* fix: validate proxy request body and nested fields
Ensure caller-supplied request fields cannot override server-side deployment
configuration, and apply request-body validation consistently to nested
structures. Adjusts router kwarg handling and client-side credential handling
for base-url overrides
* test: cover router strip ordering and advisor clientside credential gate
* fix: clear deployment credentials on client base-url override
When a request overrides api_base/base_url, recompute the deployment's
litellm_params (clearing the deployment's own api_key) and drop the cached
client built for the original endpoint, so the deployment credential is not
reused for the client-supplied endpoint. Adds regression tests that assert the
credentials actually forwarded to litellm.completion/acompletion.
* fix(proxy): require api_key alongside api_base override
A request that overrides api_base/base_url but supplies no api_key still
left the proxy carrying a server credential: once the override clears the
banned-param opt-in, the provider re-resolves a key from the environment
(api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in
main.py) and forwards it to the caller-controlled URL. Popping the
deployment api_key only changed which server key leaked.
Gate is_request_body_safe so a permitted api_base/base_url override must
also carry a non-empty caller api_key; reject otherwise. The env
resolution in main.py is left as the provider boundary.
* fix(proxy): extend request-body banlist with five additional credential and session targeting fields
Yuneng's review found five deployment-owned request-body params still missing
from the denylist and the router strip set. Each lets a caller reach the
operator's provider credentials or retarget the outbound request:
aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region
retarget the OCI request, litellm_credential_name selects any server-loaded
credential by name with no ownership check, and runtimeSessionId resumes a
Bedrock AgentCore runtime session (AWS does not enforce session-to-user
mapping, so this is a cross-tenant session-resume vector).
Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to
_DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and
SDK direct calls are unaffected: the banlist gates the request body only, and
the router strip drops caller kwargs, never deployment["litellm_params"].
* test: rename arbitrary canary values in security tests to neutral placeholders
* fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip
P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS
(litellm_embedding_config, extra_body) for the banned-param check but not for
the api_key co-presence check, so a base override smuggled into one of those
nested dicts cleared the client-side-credentials opt-in without a paired
api_key and let the provider re-resolve a server credential from the
environment. Run _check_base_override_has_api_key on each nested config dict
too, so the requirement applies wherever a base override is permitted.
P1-B: the deployment-owned credential strip in the Router runs unconditionally
on every _completion/_acompletion, which is security-correct but silently
drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a
single warning (key names only, never values) when the strip removes a
non-empty value, so the backwards-incompatible behavior is visible without
gating the strip on a context flag that does not exist.
* fix(proxy): apply api_key co-presence to tool-entry base override
is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and
tools[]); the previous commit extended the api_key co-presence rule to root
and nested config dicts but not to tool entries. With
allow_client_side_credentials enabled, a tool entry carrying api_base/base_url
and no paired api_key cleared the gate, letting a provider interceptor fall
back to a server-side credential for a caller-controlled URL. Add the same
_check_base_override_has_api_key call to each tool dict and its nested function
dict, mirroring the symmetry already applied to the nested config keys. The
rule is unchanged: api_key must live in the same dict as the base override it
accompanies.
* test(proxy/auth): require paired api_key under extra_body opt-in
* fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running
* fix(advisor): narrow proxy-import guard to ImportError-family
* fix(router): gate api_key clear on base override behind litellm.proxy_is_running
* test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture
* style: use built-in generics in PR-added type annotations
* revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate
* revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes
* style: black-format advisor orchestration test
(cherry picked from commit 1667b8f740)
* fix(proxy/client): redact api key from key/info client error messages
The keys management client builds GET /key/info?key=<key> and lets the
requests HTTPError propagate. str(HTTPError) renders the failing request URL
verbatim ("... for url: .../key/info?key=sk-..."), so any caller that logs the
exception leaks the full key; the 401 branch leaked the same way through
UnauthorizedError(str(orig_exception))
Redact both branches with the existing redact_secrets helper so the
secret-bearing query param is scrubbed to ?REDACTED while the status code,
reason, and response object are preserved. Server-side responses already mask
the key, so this closes the remaining client-side surface
* fix: preserve key info unauthorized response
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 71ee1a852a)
* fix(vertex/files): stream OpenAI->Vertex batch JSONL uploads to fix OOM on large files
Large (1GB+) batch JSONL uploads to Vertex AI / GCS caused OOM or killed the worker
because the request body was buffered and multiplied 2-3x in size. The create-file
path is now streaming end-to-end: transform_create_file_request returns a
ResumableChunkedUploadConfig carrying a lazy _OpenAIToVertexBatchUploadStream, and the
HTTP handler opens a GCS resumable session and PUTs the body in bounded 8 MiB chunks
(Content-Range, 308 between chunks) so the transformed payload is never held in full.
The proxy /v1/files endpoint streams from Starlette's spooled upload handle instead of
reading the whole body, and batch rate limiting counts tokens and models in a single
streaming pass.
Only gcs_bucket_name is supported for the GCS target; the legacy bucket_name key is
intentionally not read.
Also removes the unreachable VertexAIFilesHandler create path and everything only it
kept alive (VertexAIJsonlFilesTransformation, _stream_openai_jsonl_to_vertex, the legacy
transform helpers), plus the orphaned batch_utils helpers the streaming rewrite replaced.
* fix(batches): return original JSONL on unparseable row to avoid silent batch truncation
The streaming rewrite of replace_model_in_jsonl accumulated physical lines and
skipped a row on JSONDecodeError to support multi-line objects, but a genuinely
malformed or truncated row never completes: it poisons the buffer, swallows every
following row, and the function still returned the partial rewrite (the rows before
the bad one, already model-rewritten) as if the batch were complete. That turned the
pre-rewrite behavior of returning the original file unchanged (so the provider rejects
the bad batch loudly) into a silent partial submission.
Restore the original-content fallback: when an unparseable remainder is left after the
loop, return the original file_content (rewinding a consumed seekable source) instead of
the truncated output. The multi-line happy path is unchanged.
* test(batches): mock resumable GCS upload in vertex batch prediction test
The vertex batch file-create path now streams to a GCS resumable session via
_aresumable_chunked_upload (httpx send) instead of AsyncHTTPHandler.post, so the
existing test's post mock no longer intercepted the upload and a real request hit
GCS (401). Mock _aresumable_chunked_upload to return the GCS object response; the
resumable protocol itself is covered in test_vertex_ai_files_streaming.py.
* fix(batches): resilient per-row token accounting; no hard-block on count failure
The batch input-file pass iterated a generator whose json.loads raised on a
malformed line; the outer except caught it and stopped the loop, so any body.model
on rows after a bad line was never collected and the model allowlist check ran
against a partial set. It also hard-blocked the batch with a 400 whenever token
counting raised, a backwards-incompatible change from the prior swallow-and-proceed
behavior that breaks legitimate rows the token counter cannot measure (e.g. some
multimodal content).
Iterate the JSONL line-by-line and account each row independently. A malformed line
is skipped (its request cannot run upstream anyway) and a row the counter cannot
measure falls back to a conservative size-based estimate. The loop never aborts, so
the allowlist check always sees every parseable model, and the token total is never
zeroed, so a crafted uncountable row still cannot evade the TPM limit, without
hard-rejecting a legitimate batch.
* perf(vertex/files): unblock async upload; drop empty finalize; widen batch MIME types
Three review follow-ups on the resumable batch upload:
- _aresumable_chunked_upload pulled chunks from a synchronous generator that runs
the per-row transform inline on the event loop thread, blocking other requests
between PUTs on large uploads. Each chunk is now produced via asyncio.to_thread.
- _iter_resumable_chunks no longer yields a trailing empty chunk, so an exactly
chunk-aligned upload finalizes on its last data chunk instead of an extra
zero-byte PUT; a 0-byte stream still finalizes via the caller's empty request.
- valid_content_type now accepts the MIME types clients label .jsonl batch uploads
with (text/plain, application/json, ndjson, ...), so such a batch file no longer
silently bypasses the streaming path into the buffered media upload.
* fix(vertex/files): keep legacy bucket_name as GCS bucket fallback
The rename to gcs_bucket_name dropped the legacy bucket_name key entirely, so an SDK caller passing bucket_name to a Vertex AI file create/retrieve/content call with GCS_BUCKET_NAME unset got ValueError("GCS bucket_name is required") where it previously resolved the bucket. _get_configured_bucket_name now reads gcs_bucket_name, then bucket_name, then the env var, and bucket_name is restored to OPTIONAL_KWARGS_KEYS so it survives get_litellm_params on the retrieve and content paths. gcs_bucket_name keeps precedence when both are present
* style: sort imports in llm_http_handler to satisfy I001 budget
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 56825926af)
Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.
- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
async). Large agentic tool-use / thinking streams can make assembly re-raise
as APIError from inside the except-StopIteration handler, where the sibling
except does not catch it, so it escaped __next__/__anext__ and dropped the
request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
stream_chunk_builder returns None or raises, rebuild usage from the
message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
(it is read from there, not from kwargs), matching the gemini/cohere/openai
handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
operators don't have to reverse-map spend back to a key
(cherry picked from commit b24b964e04)
A streaming request that breaks mid-flight, for example on a mid-stream read
timeout, still bills the provider for the chunks already delivered, yet the proxy
recorded that interrupted request as a zero-spend failure. An earlier revision
logged the recovered partial usage through the success path, which mislabeled a
failed request as a success and produced a misleading spend row
This recovers the partial usage where the failure is actually logged. The
streaming handler assembles the usage from the chunks seen so far and stashes it,
with its cost, on the logging object before firing the failure handlers. The
proxy failure hook lifts that usage and cost onto request_data before the
non-serialisable logging object is popped, and the spend-log writer records the
real partial spend on the failure row instead of a hardcoded zero;
get_logging_payload honors the recovered usage for the token columns and
_failure_handler_helper_fn preserves the recovered cost so the non-DB failure
loggers stay consistent
A request that recovers via a successful fallback is unaffected: the failure hook
only fires when the whole request fails, so the fallback's combined-usage success
row stays the single source of truth and there is no double counting
Resolves LIT-3825
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
(cherry picked from commit 4847fa5dd5)
* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel
A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.
* refactor(ui): centralize no-mcp-servers sentinel in a shared constant
The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.
* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes
Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.
(cherry picked from commit 19a29e0579)
* fix(guardrails): return 400 not 500 when AIM blocks a request
AIM guardrail blocks raised a bare HTTPException whose type and param
serialized as the literal string "None", which broke OpenAI-SDK error
parsing for downstream consumers. Switching AIM to raise a ProxyException
surfaced a second bug: the shared error funnel re-derived the HTTP status
from a nonexistent status_code attribute and downgraded the 400 to a 500.
The funnel now honors an already-normalized ProxyException rather than
rebuilding it, and ProxyException is excluded from llm_exceptions alerting
so a content-policy block no longer pages on-call as an LLM API failure
Resolves LIT-3751
* fix(guardrails): route all AIM rejection paths through ProxyException
The block-action fix left two AIM rejection paths raising a bare
HTTPException: the multimodal anonymize rejection and the output-side
block. Both serialized type and param as the literal string "None", the
same malformed shape the block fix removed. Funnel all three through a
shared _rejection helper so they return a conformant OpenAI error body.
The output block carries content_policy_violation; the multimodal
rejection stays a plain invalid_request_error because it is a usage
error, not a policy violation
Resolves LIT-3751
* fix(guardrails): record AIM ProxyException blocks in failure logs
Switching AIM blocks from HTTPException to ProxyException made
_is_proxy_only_llm_api_error return False for them, so
_handle_logging_proxy_only_error was skipped and the blocked prompt was
dropped from the configured failure loggers. Classify ProxyException as a
proxy-only error alongside HTTPException so guardrail blocks are recorded
again, matching the prior behavior. The llm_exceptions alert suppression
is a separate check and stays in place
Resolves LIT-3751
* style(guardrails): use str | None over Optional[str] in AIM _rejection
* style(guardrails): collapse AIM _rejection signature per black
(cherry picked from commit b5fcd859be)
* fix(guardrails): stop re-initializing DB guardrails on every poll
InMemoryGuardrailHandler._has_guardrail_params_changed compared the
in-memory LitellmParams against the raw dict loaded from the DB. The
in-memory side carries every field default and coerces enums via
model_dump(), while the DB side only holds the keys originally stored,
so the two shapes never compared equal and the guardrail was rebuilt on
every poll cycle.
Each rebuild created a fresh instance, but delete_in_memory_guardrail
only removed the old callback from litellm.callbacks. Request handling
promotes guardrail callbacks into the success/failure/async lists, so
the previous instance stayed referenced there and instances accumulated.
Normalize both sides through LitellmParams(...).model_dump() before
diffing, and purge the callback from every callback list on delete.
* refactor(guardrails): narrow params-normalization fallback to ValidationError
The comparison normalizer caught a bare Exception and silently fell back
to the raw dict, which hid the cause and quietly degraded the affected
guardrail back to re-initializing on every poll. Catch only the
ValidationError that LitellmParams construction can raise, log a warning
so the offending row is diagnosable, and let any other error surface
instead of being swallowed.
* refactor(callbacks): add remove_callback_from_all_lists helper to manager
Move the knowledge of which callback lists a callback can be promoted
into out of the guardrail registry and into LoggingCallbackManager, where
the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail
now delegates to the new helper instead of iterating the lists itself.
(cherry picked from commit 9fa74ad8b4)
* fix(guardrails): run pre_call hook once for model-level guardrails
A CustomGuardrail attached to a deployment via litellm_params.guardrails
gets its async_pre_call_hook invoked twice per request: once by the proxy
pre-call loop and again by async_pre_call_deployment_hook after the router
spreads the model-level guardrails into the top-level request kwargs.
Record in request metadata that the proxy pre-call loop already ran a given
guardrail, and have the deployment hook skip it when the marker is present.
Direct-SDK usage never runs the proxy loop, so the deployment hook stays the
sole invocation there and still fires exactly once.
The marker key is stripped from untrusted caller metadata so a request body
cannot suppress a model-only guardrail by pre-seeding it.
* fix(guardrails): mark pre_call dedup on the post-hook request data
Record the exactly-once marker after async_pre_call_hook runs, on the data
object that flows downstream, rather than before it. A guardrail whose hook
returns a brand-new request dict (instead of mutating or spreading the one it
received) would otherwise discard the marker, letting the deployment hook
re-run the guardrail a second time.
(cherry picked from commit 4faeabc254)
* fix(proxy): resolve list files credentials from team BYOK deployments
GET /v1/files without target_model_names now prefers the team's own
deployment (model_info.team_id) over shared global provider keys, so JWT
team auth lists files against the correct upstream account.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): scope list files credential lookup to team allowlist
Remove the unrestricted deployment scan that could leak global provider
keys to teams without access, normalize all-proxy-models to the team-scoped
model list, and fix TID251 violations by using dict instead of Dict/Any.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 6c8b60d50d)
* fix(proxy): optionally surface public team model name in /v1/models
Behind general_settings.use_team_public_model_name (default False). When
enabled, /v1/models and /models surface the public team_public_model_name
for team-scoped (BYOK) models instead of the internal routing key
model_name_{team_id}_{uuid} -- consistent with /v1/model/info and
OpenAI-compatible. Off by default so the listing's model ids stay
backward-compatible for callers that scripted against the internal name;
routing by the internal name is unchanged regardless of the flag.
Presentation-layer only: access-group, auth, and routing semantics are
unchanged; non-team models are pass-through.
* fix(proxy): default team model listings to public names
* test(proxy): cover team model listing metadata
* test(proxy): cover empty team listing deployments
* refactor(proxy): simplify team model listing translation
* fix(proxy): resolve public team model name on GET /v1/models/{id}
The listing endpoints advertise team_public_model_name, but the retrieve
endpoint validated and looked up by the raw id, so a public name 404'd.
Resolve the public name back to the internal routing key (scoped to the
caller's accessible models so colliding names never cross teams), look up
by it, and echo the public name back as the response id.
* test(proxy): cover public-name resolution on model retrieve
* refactor(proxy): extract team model-name translation into TeamModelNameTranslator
Move the team-scoped (BYOK) listing/retrieve name translation out of
proxy_server.py into a dedicated common_utils module. Static methods with
general_settings injected so the logic is unit-testable without globals and
proxy_server.py stays thin.
* refactor(proxy): use TeamModelNameTranslator in model_list and model_info
* test(proxy): target TeamModelNameTranslator for model-name translation
* fix(proxy): type create_model_info_response return as dict[str, object]
* fix(proxy): keep internal routing key for team model listing metadata lookup
Add listing_entries returning (public response id, internal lookup id) so
include_metadata=true resolves fallbacks against the routing key the router
indexes by, instead of the translated public name (which never matches).
* fix(proxy): build /v1/models metadata from internal key, show public id
* test(proxy): cover team listing fallback metadata via internal key
* fix(proxy): use builtin dict generics in create_model_info_response (UP006)
---------
Co-authored-by: Tushar More <tusharmore8408@gmail.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
(cherry picked from commit 60f4c01b74)
* feat: add opt-in healthy_only filter to GET /v1/models
Adds an opt-in `healthy_only=true` query parameter to GET /v1/models and
GET /models that hides models whose backing deployments are all marked
unhealthy by background health checks.
- Add Router.async_get_fully_unhealthy_model_names(), mirroring the
semantics of get_fully_blocked_model_names(): a model is hidden only
when every backing deployment is unhealthy and the health state is
not stale (fail open otherwise).
- Reuses the existing DeploymentHealthCache populated by
_run_background_health_check(), so no new health state is introduced.
- No-op when allowed_fails_policy is set, mirroring
_async_filter_health_check_unhealthy_deployments semantics.
- team_public_model_name aliases are aggregated alongside model_name.
- Hiding is presentation-only; default behavior is unchanged.
Fixes#30128
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: address Greptile review notes
- Note team-alias asymmetry vs get_fully_blocked_model_names
- Debug-log when healthy_only is set but no health state is available
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 9dd9d2322a)
Add bare /v1/vector_stores/{vector_store_id} to openai_routes so retrieve, update, and delete classify as LLM API routes for internal user and internal viewer roles.
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit 902122a06b)
The OAuth2 passthrough ran user_api_key_auth on the client's upstream bearer
first and only recovered after the failed validation had already logged a 401
auth event to the tracer, so successful tool calls to a delegated server each
carried a phantom 401 span. Check delegate_auth_to_upstream before validating:
a delegated server skips the doomed call entirely so nothing is logged, and a
non-delegated server validates normally and surfaces a real 401 rather than
being exchanged for an anonymous upstream-passthrough session.
(cherry picked from commit 039a2d8bf5)
The grace-period branch assigned the recursive get_data result (a
finished LiteLLM_VerificationTokenView) back into the variable that the
combined-view dict normalization then subscripts, raising TypeError on
every request made with a rotated key inside its grace window; auth
surfaced that as a 401. Return the recursive result directly instead.
Regression test drives the full get_data flow: old hash misses the view,
deprecated table resolves to the active token, and the call must return
the view object
(cherry picked from commit 5047eaf7f0)
Targeted subset of staging commit cfcdf8714a (#30202): only the
anthropic_passthrough_logging_handler.py hardening hunks and their four
tests are taken; the rest of that staging batch is intentionally excluded.
(cherry picked from commit cfcdf8714a)
(cherry picked from commit 973c7eb8d6)
* fix(proxy): populate access_via_team_ids on /v1/model/info
Team metadata enrichment previously only ran on /v2/model/info with
include_team_models=true, leaving /v1/model/info without
access_via_team_ids for project model-picker flows.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(dashboard): sync OpenAPI schema for /v1/model/info query params
Add include_team_models and teamId to the generated schema for /model/info
and /v1/model/info after the proxy endpoint gained team-access filtering.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): always return direct_access on /v1/model/info
Set direct_access to true or false on every enriched model so clients
can filter without treating a missing field as ambiguous.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(proxy): fail fast when teamId is set without a connected DB on /v1/model/info
Raise the db_not_connected error before building, enriching, and translating the model list instead of after, so a teamId query against a proxy with no database no longer wastes the full enrichment pipeline.
* fix(proxy): fail fast when include_team_models is set without a database
include_team_models=True relies on _populate_team_access_on_models to set
direct_access/access_via_team_ids, which only runs when a database is connected.
Without one, _filter_models_to_user_accessible discarded every model and the
endpoint returned an empty list with HTTP 200. Mirror the teamId guard so the
request fails fast with a clear db_not_connected error before any model-list work.
* fix(proxy): populate direct_access on single-model /model/info lookup
The /v1/model/info list path populates model_info.direct_access (and
access_via_team_ids) when a database is connected, but the
litellm_model_id single-model lookup returned early without it. This
made the two endpoints disagree, breaking the parity assertion in
test_get_specific_model. Run the same population on the single-model
path so both responses match.
* fix(proxy): apply no-DB fast-fail before litellm_model_id branch
The teamId/include_team_models no-DB guard sat after the litellm_model_id
early return, so ?litellm_model_id=X&teamId=Y with no DB returned 200 with
unpopulated access fields instead of the 500 raised on every other path.
Move the guard ahead of the branch so the fast-fail is uniform.
* fix(proxy): apply teamId/include_team_models filters on single-model lookup
The litellm_model_id early-return branch in model_info_v1 populated the
team access fields but returned before the teamId and include_team_models
filters ran, so a single-model lookup surfaced the deployment regardless
of team access when the DB was connected. Run both filters on the
single-model list before returning so the documented query params behave
the same with and without litellm_model_id.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 7d1f68e72a)
* fix(proxy): align /v1/model/info with router deployments
Return router model_list entries (including team-scoped models) with team
access metadata instead of wildcard-expanded names from get_complete_model_list.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): gate v1 team filter and honor key allowlists
Only apply get_all_team_and_direct_access_models for admin or user-bound
keys, then intersect with key/team model restrictions to avoid empty lists
for service tokens and metadata leaks for restricted keys.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): skip v1 team filter when user row is missing
Require a DB-backed user before applying team-access filtering on
/v1/model/info, and skip the trailing filter in get_all_team_and_direct_access_models
when user context cannot be resolved.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Revert "fix(proxy): skip v1 team filter when user row is missing"
This reverts commit 74e1fbd77a.
* fix(proxy): restore legacy v1 model access filtering
Keep /v1/model/info on key/team allowlists instead of DB team-membership
filtering, while still listing router deployments for team-scoped models.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): drop A2A agent entries from public /v1/model/info list
* fix(proxy): scope team BYOK rows on /v1/model/info to caller's teams
Listing the full router model_list let any authenticated key without
explicit model restrictions enumerate other teams' BYOK deployments
(public name, team_id, api_base) via /v1/model/info. Reuse the existing
_get_caller_byok_team_scope check so non-admin callers only see global
deployments plus their own team's BYOK rows; admins keep the full view.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 6068bb7781)
* fix(proxy): authorize batch files using upload target_model_names (LIT-3593)
After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)
Restores the reverse-lookup for the JSONL body.model fallback path so that
legacy/pre-target_model_names managed files still map stripped provider IDs
back to proxy aliases before auth. Also cleans up redundant `or None`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)"
This reverts commit 30d2e96f77.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit 2cd7e87485)
* fix(ui): load MCP tool configuration tools via the OBO/passthrough-aware GET path
* fix(mcp): admin-only include_disabled_tools so the settings UI shows toggled-off tools
* fix(ui): repopulate MCP server edit form when server data loads after mount (OAuth return)
* fix(ui): persist MCP OAuth token on save and return to the Settings tab after authorize
* fix(ui): scope MCP OAuth callback to the initiating form so create and edit flows don't cross-talk
* fix(ui): derive OAuth-return Settings tab via lazy state init instead of setState-in-effect
* Fix MCP OAuth edit token handling
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
(cherry picked from commit 51ba6e39cd)
Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first
(cherry picked from commit 1bbaf1c39d)
* feat(proxy): publish /v2/model/info in Swagger OpenAPI spec
Expose the v2 model info endpoint in /docs by removing include_in_schema=False
and documenting query parameters used by the admin UI and proxy CLI consumers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(ui): regenerate schema.d.ts for /v2/model/info OpenAPI docs
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit f5b11b72a6)
* fix(proxy): resolve vector store file list credentials from team deployments
GET /v1/vector_stores/{id}/files now uses the same router credential routing as POST, including JWT team model hints and wildcard model selectors, so list requests no longer call OpenAI with Bearer None.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): authorize model hints and fix credential routing for vector store file list
Resolves three review findings on the vector store file list path.
Authorize user-controlled model hints (?model= query param and the
x-litellm-model header) against the key's and team's allowed models via
can_key_call_model / _can_object_call_model before any deployment
credentials are resolved, closing a model access bypass where a normal
key could file-list using a restricted deployment's provider credentials.
Run the managed vector store registry resolution before the model routing
hint so the managed store sets the routing model first; the hint resolver
then selects credentials matching that model instead of a team fallback
deployment, avoiding a credential/model mismatch across deployments.
Skip team-fallback deployments whose provider cannot be determined instead
of treating them as OpenAI, so a deployment without an explicit
custom_llm_provider or "openai/" prefix no longer has its credentials
injected.
* fix(proxy): enforce vector store file model auth
Ensure vector store file listing routes authorize explicit and inferred model routing before resolving deployment credentials.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): type guard vector store model hints
Keep vector store model hint authorization typed to string-only values so static checks pass.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(proxy): hot-reload .env in dev when running with --reload
The --reload watcher already restarts the worker on *.py and --config YAML
edits, but .env was unwatched, so changing a key there did nothing until a
manual restart. Add .env to the uvicorn reload_includes (and to the
StatReload monkeypatch, which ignores reload_includes) so an edit triggers a
worker restart.
A reloaded worker is a fresh process that inherits the reloader's
environment, so load_dotenv(override=False) would keep serving the stale
inherited value for any key already in the environment. The CLI now exports
LITELLM_DEV_ENV_HOT_RELOAD when --reload is set, and litellm/__init__.py
reads it to load .env with override=True only on that dev path, leaving
normal startup precedence untouched.
* feat(proxy): warn that --reload makes .env override shell env vars
When --reload is active, worker processes re-read .env with override=True, so
.env values win over shell-exported environment variables. Surface this dotenv
precedence change with a startup warning so a developer who relies on a
shell-exported override is not silently surprised.
* fix(proxy): type reload helper paths as Optional[str] to satisfy mypy
* fix(proxy): watch the cwd .env in both reload backends for parity
WatchFiles only watches cwd (and the --config dir) for .env, while the
StatReload fallback used find_dotenv(usecwd=True), which walks up to a
parent-dir .env that WatchFiles never sees. Point StatReload at the same
cwd .env so the two reload backends react to the same file.
Deleting a team-scoped BYOK model left its public name in team.models, so /models
with a team key kept listing the now-deleted "ghost" model. delete_model stripped
team.models using only litellm_modeltable alias lookups, but models added via
/model/new with a team_id never create an alias row; their public name lives only
in team.models and model_info.team_public_model_name, so it was never removed. The
team cache was also left stale because the delete path skipped _refresh_cached_team.
The cleanup now keys off team_public_model_name (falling back to alias keys), runs
after the deployment row is deleted, and strips a public name only when no remaining
team deployment still backs it, so a load-balanced replica is not revoked and
concurrent deletes cannot leave a ghost. The updated team row is refreshed in cache
so /models reflects the change immediately
* fix(jwt): attribute spend to resolved DB user_id on email/sso fuzzy match
When user_id_upsert is enabled with JWT auth and a pre-migration user row
exists whose user_email matches the JWT email but whose user_id is a UUID,
get_user_object resolves the legacy row via fuzzy lookup, but the JWT-claim
user_id (the email) still flowed into team-membership lookup,
JWTAuthBuilderResult.user_id, UserAPIKeyAuth and the spend tables. Spend was
orphaned under a phantom email id; /user/info and the Usage page showed $0
for the legacy user (GH #26789).
Treat the resolved user_object as the source of truth: add
_canonical_user_id_from_db, rebind inside get_objects, and return
effective_user_id so auth_builder unpacks it without adding statements.
Fixes#26789
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(jwt): log user_id rebind at DEBUG to avoid email PII in INFO streams
Greptile review on #29217: rebinding often logs JWT email claims at INFO.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(jwt): update passthrough allowlist mock for 5-tuple get_objects
Staging #29256 added a test that still mocked get_objects with a
4-tuple; our PR expanded the return to 5 values (effective_user_id).
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(google): add google-genai SDK proxy integration tests for Gemini and Vertex
Pin google-genai in the CI dependency group and exercise streaming/non-streaming
generate_content through the LiteLLM proxy in the existing unified_google_tests suite.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): address Greptile review for google-genai proxy SDK tests
Restore GOOGLE_APPLICATION_CREDENTIALS after the module proxy fixture tears down,
initialize temp-file tracking on the proxy SDK base class, and skip litellm reload
for proxy_genai_sdk tests so the module-scoped proxy server stays consistent.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): only load Vertex credentials when keys exist for proxy SDK tests
Avoid writing empty GOOGLE_APPLICATION_CREDENTIALS temp files so Vertex tests
skip cleanly without credentials, use a session-scoped proxy fixture, and clean up
per-test credential temp files.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(test): scope google-genai pin to unified_google_tests only
Remove google-genai from the ci dependency group and pin it in
tests/unified_google_tests/requirements.txt for local test installs.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(google): tie litellm reload skip to proxy fixture dependency
Replace the name-based reload guard with a check on whether the test
requests the google_genai_proxy_url fixture, so the skip stays correct
if the proxy SDK tests are renamed.
* fix(test): stop DatabaseURLSettings tests leaking DATABASE_URL into os.environ
The autouse env scrubber relied on monkeypatch.delenv, but apply_to_env
writes DATABASE_URL straight into os.environ, which monkeypatch never
tracks and therefore never undoes. The synthesized writer.example.com URL
leaked past the last test in this module and into proxy-infra tests that
read DATABASE_URL to decide whether to hit a real database, e.g.
test_deprecated_key_grace_period_cache_hit_path, turning an intended skip
into a ConnectError. Snapshot and restore the managed vars directly so the
original environment is reinstated regardless of how it was mutated.
* test(google): drop redundant per-test vertex credential setup
The session-scoped google_genai_proxy_url fixture already configures
GOOGLE_APPLICATION_CREDENTIALS before the proxy starts, and
_require_proxy_sdk skips when credentials are missing, so the per-test
_setup_vertex_credentials_if_needed helper and its temp-file tracking
never did any work. Remove it to keep the ABC self-contained.
* test(google): declare model_config contract on proxy SDK ABC
_skip_reason_if_credentials_missing reads self.model_config to pick the
provider, but that property was only declared on the sibling
BaseGoogleGenAITest. Make the dependency explicit by adding model_config
as an abstract property on BaseGoogleGenAIProxySDKTest so the ABC is
self-contained and a standalone subclass fails fast instead of hitting an
AttributeError.
* test(google): narrow streaming error catch to Exception
Catching BaseException in the streaming assertion swallowed
KeyboardInterrupt and SystemExit, turning a Ctrl-C into a test failure
message instead of letting pytest interrupt cleanly. Only genuine runtime
errors should be recorded as stream failures, so catch Exception.
* test(google): initialize proxy on the same loop that serves it
The proxy was initialized via asyncio.run() on the main thread, which
creates and tears down a throwaway event loop, while requests were served
on a separate loop in the worker thread. Any asyncio primitive bound to
the init loop would be unusable once serving started. Run initialize()
on the worker thread's loop right before server.serve() so setup and
request handling share a single event loop.
* test(google): drop redundant google-genai requirements pin
google-genai>=1.37.0,<2.0 is already declared in the proxy-runtime extra,
which the google_generate_content_endpoint_testing CI job installs via
uv sync --all-extras. The standalone tests/unified_google_tests/requirements.txt
duplicated that pin with a narrower ==1.37.0 specifier and was never
installed by CI, so it added a second source of truth without changing
what gets installed. Drop it and rely on the proxy-runtime extra.
* chore: revert incidental uv.lock exclude-newer bump
The google-genai ci pin was added and then dropped (it is already
provided by the proxy-runtime group), but each uv lock recomputed the
relative exclude-newer span, leaving only a timestamp bump in uv.lock.
Restore it to the base value so this test-only PR carries no lockfile
change.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
* Mark xAI models retiring on 2026-05-15 (#28788)
Per https://docs.x.ai/developers/migration/may-15-retirement, xAI is
retiring the following slugs on 2026-05-15 (auto-redirect to grok-4.3
with various reasoning efforts; callers continuing to use the old slugs
will be billed at grok-4.3 pricing):
grok-4-1-fast-reasoning{,-latest} -> grok-4.3 (low effort)
grok-4-1-fast-non-reasoning{,-latest} -> grok-4.3 (none)
grok-4-fast-reasoning -> grok-4.3 (low effort)
grok-4-fast-non-reasoning -> grok-4.3 (none)
grok-4-0709 -> grok-4.3 (low effort)
grok-code-fast-1{,-0825} -> grok-build-0.1
grok-3 -> grok-4.3 (none)
Only the direct xai/ slugs are tagged; third-party hosts (azure_ai,
oci, vercel_ai_gateway, perplexity/xai) run their own schedules. The
grok-3 retirement list explicitly names only the base grok-3 slug — the
-mini / -fast / -beta / -latest variants are not listed, so they remain
untouched.
* feat(moonshot): advertise json_schema response support on live models (#29683)
litellm.responses() already routes Moonshot through the responses->chat-completions
bridge, and Moonshot honors response_format json_schema on chat completions. The
cost-map entries left supports_response_schema unset, so discovery layers that gate
on that flag dropped Moonshot from structured-output / responses listings even though
the capability works end to end.
Set supports_response_schema on the nine models currently live on api.moonshot.ai:
kimi-k2.5, kimi-k2.6, the moonshot-v1 8k/32k/128k text and vision-preview variants,
and moonshot-v1-auto. Verified against the live API that each honors json_schema and
that litellm.responses() returns schema-valid structured output through the bridge.
* chore(moonshot): mark models retired from api.moonshot.ai as deprecated (#29685)
Thirteen Moonshot/Kimi models in the cost map no longer resolve on
api.moonshot.ai (all return 404). Stamp each with its deprecation_date from
platform.kimi.ai/docs/models rather than deleting the entries, so historical
cost calculation keeps resolving the names while tooling can surface the
retirement.
Dates: kimi-thinking-preview 2025-11-11; kimi-latest and its 8k/32k/128k context
variants 2026-01-28; the kimi-k2 preview/turbo/thinking series 2026-05-25; the
moonshot-v1 -0430 snapshots use their own 2024-04-30 snapshot date (Moonshot
publishes no discontinuation date for them).
* fix(moonshot): drop temperature for reasoning models (kimi-k2.5/k2.6) (#29687)
Kimi reasoning models reject every temperature except 1; a request with
temperature=0.2 returns "invalid temperature: only 1 is allowed for this model".
litellm only clamped temperature into [0.3, 1], so any value below 1 still 400'd.
Drop the temperature param entirely for reasoning models (gated on
supports_reasoning, the same signal transform_request already uses) so the model
default is used; the non-reasoning moonshot-v1 models keep the existing clamp.
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* feat(mcp): add per-server timeout configuration (#29672)
* feat(mcp): add per-server timeout configuration
* fix(mcp): address timeout field review comments
- use is not None guard instead of or for 0.0 edge case
- copy timeout in both LiteLLM_MCPServerTable constructions (health check path + _build_mcp_server_table)
- add timeout Float? column to all three schema.prisma files
- extend round-trip test to cover _build_mcp_server_table direction
- add test for zero timeout not treated as falsy
* fix(mcp): forward timeout in _build_temporary_mcp_server_record
* fix(mcp): return 504 instead of 500 when per-server timeout fires
* test(mcp): add 504 timeout regression test; fix black formatting
* Add jp. Bedrock cross-region inference profile for claude-opus-4-7 (#28567)
* fix(thinking): handle None thinking param in is_thinking_enabled (#28598)
Squash-merged by litellm-agent from Terrajlz's PR.
* feat(helm): support tpl rendering in podAnnotations (#28609)
Squash-merged by litellm-agent from devauxbr's PR.
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505) (#28575)
* Forward custom_llm_provider through the Responses API bridge (Fixes#28505)
When a Chat Completions request to a GPT-5.4+ model contains both
`tools` and `reasoning_effort`, `completion()` auto-routes through
`responses_api_bridge`. The bridge handler called
`litellm.responses()` / `litellm.aresponses()` without forwarding the
already-resolved `custom_llm_provider`, so the downstream call
re-invoked `get_llm_provider()` with `custom_llm_provider=None` and
stripped a second provider prefix from a `provider/provider/model`
deployment string.
For a deployment configured as `openai/openai/openai/gpt-5.5`,
the bridge flow sent `openai/gpt-5.5` to the upstream API instead of
the correct `openai/openai/gpt-5.5`. Upstream APIs that enforce
model-name allow-lists rejected this as `key_model_access_denied`.
Fix: pass the locally-resolved `custom_llm_provider` into both the
sync `responses()` and async `aresponses()` calls so the downstream
`_resolve_model_provider_for_responses` sees an explicit provider
and skips the second prefix-strip.
New regression test
`tests/test_litellm/completion_extras/test_responses_bridge_provider_propagation.py`
pins both call sites: each must forward `custom_llm_provider`.
* fix(28505): set custom_llm_provider on request_data instead of as duplicate kwarg
Greptile flagged that the previous patch passed custom_llm_provider as an
explicit kwarg to responses()/aresponses() while request_data already
carried it via the spread of sanitized_litellm_params, which would raise
TypeError: got multiple values for keyword argument on every real bridge
call.
Switches to assigning request_data['custom_llm_provider'] before the call
so the resolved provider wins over whatever sanitized_litellm_params spread
in, without duplicating the kwarg.
Updates the regression test to seed request_data with a sentinel
custom_llm_provider so it actually exercises the overwrite path (the
previous test mocked transform_request with a minimal dict and never hit
the conflict).
* chore: trigger shin-agent re-eval on retargeted staging base
* chore: trigger shin-agent re-eval against updated Greptile state
* Add jp. Bedrock cross-region inference profile for claude-opus-4-7
AWS Bedrock documents jp.anthropic.claude-opus-4-7 alongside the
existing us./eu./au./global. profiles for Claude Opus 4.7
(ap-northeast-1 Tokyo / ap-northeast-3 Osaka), but the entry is
missing from model_prices_and_context_window.json. Tokyo-region
users currently get an "unknown model" error when routing through
the JP geo profile.
Adds the entry to both the canonical file and the bundled backup,
mirroring the recent pattern for sonnet-4-6 (#27831). Pricing matches
the other regional profiles (10% premium over base/global).
Regression test pins all six documented profiles (base, global, us, eu,
au, jp) and asserts pricing parity between jp. and au. variants.
Source: https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-anthropic-claude-opus-4-7.html
---------
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* feat(soniox): add soniox audio transcription integration (#29508)
* feat(openmeter): add OPENMETER_TRUST_REQUEST_USER to prevent forged attribution (#29650)
The OpenMeter callback resolves the CloudEvent subject from kwargs["user"]
first, then falls back to the key-bound user_api_key_user_id. For
multi-tenant proxy deployments, a client can set `"user": "..."` in the
request body and cause their usage to be attributed to that arbitrary
string — a billing-attribution forgery risk.
Adds OPENMETER_TRUST_REQUEST_USER env var (default "true" for backward
compatibility). When set to "false", the request-supplied `user` field is
ignored and the subject is resolved solely from user_api_key_user_id.
Matches the existing env-var-driven config pattern in this file
(OPENMETER_API_KEY, OPENMETER_API_ENDPOINT, OPENMETER_EVENT_TYPE).
* feat(search): add you_com as a search provider (#28370)
* feat(search): add you_com as a search provider
Registers You.com Search API as a first-class `search_provider` in the
`search_tools` registry, alongside Tavily, Exa, Perplexity, etc.
- New adapter: litellm/llms/you_com/search/transformation.py
- POSTs to https://ydc-index.io/v1/search
- Auth: X-API-Key from YOUCOM_API_KEY (or explicit api_key)
- Maps Perplexity unified spec: max_results -> count,
search_domain_filter -> include_domains, country -> country
- Flattens results.web + results.news into a single SearchResult list;
snippet prefers snippets[0], falls back to description; page_age -> date
- Registry: SearchProviders.YOU_COM in litellm/types/utils.py and wired
into ProviderConfigManager.get_provider_search_config()
- Pricing entry: model_prices_and_context_window.json (placeholder $0.0;
happy to adjust to maintainers' preferred public number)
- Docs: example router config snippet and example proxy yaml updated
- Tests: tests/search_tests/test_you_com_search.py - 5 mocked tests
(payload shape, domain filter mapping, snippet fallback, news flattening,
missing-api-key error)
Refs upstream expansion signal: #15942
* review fixups: normalize api_base, lowercase country, scope env-var to test
Addresses Greptile inline review comments on #28370:
- get_complete_url: strip trailing slashes from api_base *before* the
endswith("/v1/search") check, so a custom base like ".../v1/search/"
doesn't become ".../v1/search/v1/search".
- transform_search_request: .lower() country before sending, matching
Tavily's convention so callers using the unified spec form ("US") get
consistent behavior across providers.
- Tests: replace direct os.environ writes with an autouse monkeypatch
fixture so YOUCOM_API_KEY is set per-test and removed afterwards.
The missing-key test now uses monkeypatch.delenv. New test asserts the
trailing-slash normalization above.
Reverts the ARCHITECTURE.md / example yaml edits per the reviewer note
that documentation changes belong in the litellm-docs repo.
* support keyless free tier (api.you.com/v1/agents/search) as default
You.com offers an IP-throttled keyless endpoint that returns the same
response shape as the keyed one (~100 queries/day, no signup). This is a
significant onboarding lever - mirrors the keyless DuckDuckGo/SearXNG
providers already in the search_tools registry.
Behavior:
- YOUCOM_API_KEY set -> keyed: POST https://ydc-index.io/v1/search
(X-API-Key header)
- no key -> free: POST https://api.you.com/v1/agents/search
(no auth)
- YOUCOM_API_BASE override -> honored as-is
Tests:
- New: test_you_com_search_keyless_free_tier - asserts URL + absence of
X-API-Key when no key is configured.
- New: test_you_com_search_validate_environment_keyless - asserts the
config no longer raises when the key is absent.
- Removed: test_you_com_search_raises_without_api_key (the precondition
no longer holds).
- Existing payload/domain-filter/etc tests still cover keyed mode via
the autouse YOUCOM_API_KEY fixture.
Verified both endpoints accept POST + return identical JSON shape:
results.web[] / results.news[] with title, url, snippets, description,
page_age.
* register you_com in provider_endpoints_support.json
Adding `litellm/llms/you_com/` requires a corresponding entry in
provider_endpoints_support.json or the
code-quality/check_provider_folders_documented CI check fails.
Follows the compact tavily/serper pattern - endpoints: { search: true }.
Local run of the check now reports "All 114 provider folders are documented".
* move tests under tests/test_litellm/llms/ so CI exercises them
The litellm CI workflows scope unit tests to `tests/test_litellm/...`
(see test-unit-llm-providers.yml: `tests/test_litellm/llms` path), so
tests living under `tests/search_tests/` are never run in CI - which is
why codecov reports 0% patch coverage for the new adapter even though
the unit tests exist and pass locally.
Move test_you_com_search.py into `tests/test_litellm/llms/you_com/` so
the test-unit-llm-providers job picks it up. 7/7 tests still pass at
the new location.
(Sibling search-only providers - tavily, exa_ai, brave, etc. - still
live only in `tests/search_tests/` and would benefit from the same
move, but that is out of scope for this PR.)
* fix(you_com): pin Accept-Encoding: identity to dodge keyless gzip bug
The keyless free-tier endpoint (api.you.com/v1/agents/search) advertises
Content-Encoding: gzip but returns a body that httpx's decoder rejects
with `zlib.error: Error -3 while decompressing data: incorrect header
check`, surfacing as litellm.APIConnectionError in user code. curl works
because it doesn't request compression by default.
Pin Accept-Encoding: identity in validate_environment so the upstream
server skips compression entirely. Harmless on the keyed endpoint
(ydc-index.io/v1/search) which negotiates content-encoding correctly.
The header uses setdefault so a caller-supplied Accept-Encoding still
takes precedence. (Server-side bug has been flagged to the You.com team
separately - once fixed there, this workaround can be removed.)
New unit test: test_you_com_search_pins_identity_accept_encoding.
---------
Co-authored-by: Sameer Kankute <sameer@berri.ai>
* docs: fix README typo (#29419)
Correct clear spelling mistakes in documentation without changing behavior.
Confidence: high
Scope-risk: narrow
Tested: git diff --check; uvx codespell on changed files
Not-tested: Full docs build not run; text-only changes
* Fix(langfuse): pass httpx_client to Langfuse in langfuse_prompt_management to respect SSL_VERIFY (#29480)
* fix(langfuse): pass ssl_verify to Langfuse httpx client
* fix_langfuse_
* add unit tests
* addressed comments
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(models): add minimax/MiniMax-M3 to model cost map (#29412)
Add MiniMax's new flagship MiniMax-M3 to the native minimax provider:
512K context, 128K max output, native multimodal (supports_vision),
reasoning, prompt caching. Pricing (USD/M tokens): input 0.6 / output
2.4 / cache read 0.12. M3 has no active prompt-cache-write tier, so
cache_creation_input_token_cost is omitted.
Updated both the root model_prices_and_context_window.json (remote
source) and the bundled litellm/model_prices_and_context_window_backup.json
(local fallback), keeping them in sync.
* fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log (#29394)
* fix(logging): handle ResponseCompletedEvent in anthropic_messages streaming spend log
* fix(logging): extend terminal event handling to ResponseIncompleteEvent and ResponseFailedEvent; fix return type annotation
* feat(provider): Add Neosantara provider as OpenAI Compatible (#29646)
* Add Neosantara provider
* Register Neosantara provider enum
* Address Neosantara provider review feedback
* Add Neosantara packaged endpoint support
---------
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* fix: address greptile and veria review feedback
- langfuse: guard httpx_client injection behind version check (>= 2.7.3)
- soniox: propagate audio_transcription_duration in _hidden_params for spend tracking
- soniox: give SONIOX_API_BASE env var priority over caller-supplied api_base
- mcp: replace CancelledError catch with asyncio.wait_for + TimeoutError
* chore(mcp): add migration for per-server timeout column
* fix(test): add tool_use_system_prompt_tokens to model prices schema validator
* fix: mcp timeout test uses real asyncio.wait_for timeout; you_com get_complete_url respects resolved api_key
* fix: forward resolved api_key into you_com endpoint selection and apply timeout to soniox polling GETs
The search flow resolves api_key in validate_environment but never passed it
into get_complete_url, so a programmatic api_key (with no YOUCOM_API_KEY in the
env) set the X-API-Key header yet still selected the keyless free-tier endpoint.
Forward api_key through both the search entrypoint and the http handler so the
keyed endpoint is chosen.
HTTPHandler.get/AsyncHTTPHandler.get had no timeout parameter, so the Soniox
poll and transcript-fetch GETs silently used the client global default instead
of the caller timeout. Add a per-request timeout to get() and forward the
configured timeout from the Soniox handler.
* fix(soniox): price stt-async-v4 per second so transcriptions are billed
The handler stores audio_transcription_duration in _hidden_params, but the
model carried only token cost fields and the response has no token usage, so
the transcription cost path fell through to cost_per_second and returned $0.
An authenticated caller could transcribe Soniox audio without decrementing
their budget. Switch the entry to output_cost_per_second at Soniox's published
$0.10/hour async rate so the stored duration produces a real charge.
* fix(langfuse): use a dedicated httpx client for the SDK injection
The httpx_client handed to the Langfuse SDK came from _get_httpx_client(),
which returns LiteLLM's globally cached HTTPHandler. If Langfuse closed that
client on teardown it would invalidate the shared client used by every other
LiteLLM HTTP call. Build a dedicated httpx.Client instead, still resolving SSL
verification and client certificate from LiteLLM's configuration.
* fix(soniox): prefer caller-supplied api_base over SONIOX_API_BASE env var
* fix(cohere): support max_completion_tokens on cohere v2 chat (default route) (#29779)
* fix(cohere): support max_completion_tokens on cohere v2 chat
The default cohere_chat route resolves to CohereV2ChatConfig, which did not
list or map max_completion_tokens, so get_optional_params raised
UnsupportedParamsError for the standard OpenAI parameter (the modern
replacement for the deprecated max_tokens). The v1 config already maps it to
cohere's max_tokens; mirror that in v2 and add v2 regression tests.
* fix(cohere): make max_completion_tokens take precedence over max_tokens on v2
When both max_tokens and max_completion_tokens are supplied, prefer
max_completion_tokens explicitly rather than relying on dict iteration order,
and cover both orderings with a regression test.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: hectorc98 <hector.chamorroalvarez@adyen.com>
Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com>
Co-authored-by: Terrajlz <info@jouleselectrictech.com>
Co-authored-by: Bruno Devaux <devaux.br@gmail.com>
Co-authored-by: Dan Lemon <dan@danlemon.com>
Co-authored-by: Saswat <saswatds@users.noreply.github.com>
Co-authored-by: Brian Sparker <brainsparker@users.noreply.github.com>
Co-authored-by: Zhao73 <156770117+Zhao73@users.noreply.github.com>
Co-authored-by: Urain Ahmad Shah <60431964+urainshah@users.noreply.github.com>
Co-authored-by: shin-berri <shin-laptop@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: kape <168134658+kapelame@users.noreply.github.com>
Co-authored-by: danisalvaa <159898202+danisalvaa@users.noreply.github.com>
Co-authored-by: Just R <remixingmagelang@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: abhay23-AI <abhaytrivedi22@gmail.com>
* fix(auth): expand all-team-models sentinel in can_key_call_model
Keys with models=["all-team-models"] were denied during batch JSONL
model validation because can_key_call_model matched the literal string
against the model name. Add _resolve_key_models_for_auth_check to
expand the sentinel to team_models before the check, consistent with
get_key_models in model_checks.py and the completion-route bypass.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(auth): document empty team_models unrestricted access behavior; add regression test
Adds a docstring note to _resolve_key_models_for_auth_check explaining that
when team_models is empty, all-team-models resolves to [] which is treated as
unrestricted access (consistent with get_key_models behavior on other auth
paths). Adds a test to lock in this behavior.
* fix(auth): deny all-team-models access when key has no team_id
A key configured with models=["all-team-models"] but no team_id could
previously resolve to an empty allowlist, which _check_model_access_helper
treats as unrestricted access. Now the sentinel is only expanded when
team_id is set; otherwise the unresolved sentinel stays in the model list
and causes a deny (no real model name matches it). Same fix applied to
get_key_models in model_checks.py for consistency across batch and
non-batch auth paths.
* style: black format model_checks.py
* Fix batch all-team-models auth
* style: black format batch_rate_limiter.py
* fix(test): add tool_use_system_prompt_tokens to model prices schema validator
* fix(batch): catch get_team_object errors to avoid 404 escaping batch auth
* fix(batch): apply per-member model scope check after team auth in batch validation
* Fail closed on batch team auth fetch errors
* test(batch): cover team_object grant and member-scope denial in batch auth
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(galileo): use ingest traces API and standard logging payload
Switch hosted Galileo logging to /ingest/traces with nested trace/span payloads, read metrics from standard_logging_object, and include cost and total tokens on trace metrics.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(galileo): route username/password auth to v2 traces ingest
Hosted Galileo no longer serves /observe/ingest; JWT login should post the same trace payload to /v2/projects/{project_id}/traces.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(galileo): address Greptile review on logging and timestamps
Use debug-level logs for per-request Galileo callback messages and fall back to start_time/end_time when standard_logging_object omits startTime/endTime.
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(galileo): add Galileo to proxy UI callback configuration
Expose Galileo in the admin callback selector and config APIs so credentials can be configured through the dashboard instead of YAML only.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(galileo): align response type logging with Langfuse
Mirror Langfuse input/output handling for rerank, speech, transcription,
realtime, pass-through, and other response types so Galileo ingest no longer
skips supported call types.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(galileo): redact trace payload in debug logs and format with black
Avoid logging prompts and model responses in flush debug output while
keeping structural metadata for troubleshooting.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(galileo): stop logging full trace payload in debug output
Log only flush URL and trace count so prompts and model responses are not
written to application logs when debug logging is enabled.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix Galileo token totals and prompt messages
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* Add OAuth M2M support for A2A agents targeting Databricks Apps
Databricks App endpoints reject static bearer tokens and require a
short-lived OAuth token minted via the workspace OIDC token endpoint.
A2A agents could previously only authenticate outbound with static_headers
or client header passthrough, so Databricks App agents could not be
registered.
Agents configured with a databricks_oauth block in litellm_params now mint
and cache a client_credentials token and attach it as the outbound
Authorization header on both message/send and message/stream calls,
overriding any statically configured Authorization.
* Add tests covering Databricks App OAuth token error paths
Cover the HTTP status error, transport error, non-object JSON body, and
invalid expires_in fallback branches in the token cache so the failure
handling is locked in by regression tests.
* Harden Databricks App OAuth token cache
Cap the cache TTL at the token's own lifetime so a token whose validity is
shorter than the refresh buffer is never cached and served stale; include a
digest of client_secret in the cache key so a rotated secret mints a fresh
token instead of reusing the old one; and prune the per-key lock when its
cached token is evicted so the lock map stays bounded by the live key set.
* Clear per-key locks on Databricks OAuth cache flush
* fix(a2a/databricks): mint OAuth token via Basic auth header, not unsupported auth= kwarg
litellm's AsyncHTTPHandler.post (what get_async_httpx_client returns) has no
auth parameter, so minting a Databricks App OAuth token raised
"AsyncHTTPHandler.post() got an unexpected keyword argument 'auth'" before any
network call ever left the proxy, breaking the feature end to end. The handler
also calls raise_for_status() internally and re-raises a MaskedHTTPStatusError
(a subclass of httpx.HTTPStatusError), so the explicit raise_for_status() after
post() was dead code.
Build the HTTP Basic Authorization header by hand and pass it via headers, which
is what the Databricks workspace OIDC token endpoint documents for client
authentication. The token-cache tests now model the real handler contract with
create_autospec so the rejected auth= signature is enforced; the previous mocks
accepted any kwargs and silently hid the bug.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* Prune Databricks OAuth lock on the short-lived-token path
When expires_in is below the refresh buffer the token is intentionally
not cached, so _remove_key never runs for that key and the per-key lock
created by _get_lock leaked permanently. Drop the lock in that branch so
_locks stays bounded by the live key set, and assert the cleanup in the
short-lived-token test
* Gate A2A Databricks OAuth on the databricks_oauth block at the call site
Make the gating explicit where the header is applied so it is clear that only
agents configured with a databricks_oauth block enter the OAuth path; every
other agent is left untouched. Add a regression test asserting a non-Databricks
agent never invokes the token resolver.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
