mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-12 23:01:41 +00:00
9886 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
oss-agent-shin
|
2f33a86e2f
|
fix(proxy): populate Exception.args so str(ProxyException) returns message (LIT-3094) (#29015)
* fix(proxy): populate Exception.args so str(ProxyException) returns message
Adds super().__init__(self.message) to ProxyException.__init__ so that
str(exc) returns the stored message instead of empty string. Fixes LIT-3094.
* test(proxy): regression tests for ProxyException.args (LIT-3094)
* fix(proxy): populate Exception.args so str(ProxyException) returns message (LIT-3094)
* fix(proxy): clean up unintended drift; keep only ProxyException.args fix (LIT-3094)
(cherry picked from commit
|
||
|
ryan-crabbe-berri
|
d0f29513b7
|
fix(guardrails): return 400 not 500 when AIM blocks a request (#30573)
* fix(guardrails): return 400 not 500 when AIM blocks a request
AIM guardrail blocks raised a bare HTTPException whose type and param
serialized as the literal string "None", which broke OpenAI-SDK error
parsing for downstream consumers. Switching AIM to raise a ProxyException
surfaced a second bug: the shared error funnel re-derived the HTTP status
from a nonexistent status_code attribute and downgraded the 400 to a 500.
The funnel now honors an already-normalized ProxyException rather than
rebuilding it, and ProxyException is excluded from llm_exceptions alerting
so a content-policy block no longer pages on-call as an LLM API failure
Resolves LIT-3751
* fix(guardrails): route all AIM rejection paths through ProxyException
The block-action fix left two AIM rejection paths raising a bare
HTTPException: the multimodal anonymize rejection and the output-side
block. Both serialized type and param as the literal string "None", the
same malformed shape the block fix removed. Funnel all three through a
shared _rejection helper so they return a conformant OpenAI error body.
The output block carries content_policy_violation; the multimodal
rejection stays a plain invalid_request_error because it is a usage
error, not a policy violation
Resolves LIT-3751
* fix(guardrails): record AIM ProxyException blocks in failure logs
Switching AIM blocks from HTTPException to ProxyException made
_is_proxy_only_llm_api_error return False for them, so
_handle_logging_proxy_only_error was skipped and the blocked prompt was
dropped from the configured failure loggers. Classify ProxyException as a
proxy-only error alongside HTTPException so guardrail blocks are recorded
again, matching the prior behavior. The llm_exceptions alert suppression
is a separate check and stays in place
Resolves LIT-3751
* style(guardrails): use str | None over Optional[str] in AIM _rejection
* style(guardrails): collapse AIM _rejection signature per black
(cherry picked from commit
|
||
|
Shivam Rawat
|
71cc179eca
|
fix(integrations): cap Anthropic cache_control injection at 4 blocks (#30480)
* fix(integrations): cap Anthropic cache_control injection at 4 blocks
Respect Anthropic's 4 cache_control breakpoint limit by counting client-supplied blocks, skipping messages that already carry cache_control, and stopping further auto-injection once the limit is reached.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(integrations): reserve cache slot for tool_config and short-circuit cap
Address review feedback on the cache_control cap: break out of the injection loop before resolving target indices once the limit is reached, and reserve one of the four breakpoint slots when a tool_config injection point is present so the cachePoint appended by the Bedrock transform does not push the total past Anthropic's limit.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit
|
||
|
Mateo Wang
|
f05280f71b
|
fix: stop use_chat_completions_api flag from leaking into provider request body (#29447)
* fix: stop use_chat_completions_api flag from leaking into provider request body
use_chat_completions_api is a LiteLLM control flag that forces the
/responses -> /chat/completions bridge. It was missing from
all_litellm_params, so get_non_default_completion_params treated it as a
model-specific param and forwarded it to the upstream provider. A
model-level "use_chat_completions_api: true" in the proxy config therefore
reached the chat-completions path and was rejected by strict providers
(OpenAI/Anthropic) with HTTP 400 for an unknown body field.
Register it as a known internal param so it is stripped on every path
(completion, the responses bridge that calls litellm.completion, and
filter_out_litellm_params).
Adds a regression test driving litellm.completion() with a mocked OpenAI
client that asserts the flag never reaches the request body.
* test: clarify extra_body assertion in use_chat_completions_api leak test
Replace the misleading 'not in ... or {}' precedence idiom with an explicit
parenthesized guard that also handles extra_body being None.
(cherry picked from commit
|
||
|
Yassin Kortam
|
4f1168068d
|
fix(datadog): split oversized batches on 413 instead of re-queueing forever (#29444)
(cherry picked from commit
|
||
|
Yassin Kortam
|
a68db8b5bb
|
fix(guardrails): stop re-initializing DB guardrails on every poll (#30542)
* fix(guardrails): stop re-initializing DB guardrails on every poll
InMemoryGuardrailHandler._has_guardrail_params_changed compared the
in-memory LitellmParams against the raw dict loaded from the DB. The
in-memory side carries every field default and coerces enums via
model_dump(), while the DB side only holds the keys originally stored,
so the two shapes never compared equal and the guardrail was rebuilt on
every poll cycle.
Each rebuild created a fresh instance, but delete_in_memory_guardrail
only removed the old callback from litellm.callbacks. Request handling
promotes guardrail callbacks into the success/failure/async lists, so
the previous instance stayed referenced there and instances accumulated.
Normalize both sides through LitellmParams(...).model_dump() before
diffing, and purge the callback from every callback list on delete.
* refactor(guardrails): narrow params-normalization fallback to ValidationError
The comparison normalizer caught a bare Exception and silently fell back
to the raw dict, which hid the cause and quietly degraded the affected
guardrail back to re-initializing on every poll. Catch only the
ValidationError that LitellmParams construction can raise, log a warning
so the offending row is diagnosable, and let any other error surface
instead of being swallowed.
* refactor(callbacks): add remove_callback_from_all_lists helper to manager
Move the knowledge of which callback lists a callback can be promoted
into out of the guardrail registry and into LoggingCallbackManager, where
the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail
now delegates to the new helper instead of iterating the lists itself.
(cherry picked from commit
|
||
|
Yassin Kortam
|
05efbc6454
|
fix(guardrails): run pre_call hook once for model-level guardrails (#30543)
* fix(guardrails): run pre_call hook once for model-level guardrails
A CustomGuardrail attached to a deployment via litellm_params.guardrails
gets its async_pre_call_hook invoked twice per request: once by the proxy
pre-call loop and again by async_pre_call_deployment_hook after the router
spreads the model-level guardrails into the top-level request kwargs.
Record in request metadata that the proxy pre-call loop already ran a given
guardrail, and have the deployment hook skip it when the marker is present.
Direct-SDK usage never runs the proxy loop, so the deployment hook stays the
sole invocation there and still fires exactly once.
The marker key is stripped from untrusted caller metadata so a request body
cannot suppress a model-only guardrail by pre-seeding it.
* fix(guardrails): mark pre_call dedup on the post-hook request data
Record the exactly-once marker after async_pre_call_hook runs, on the data
object that flows downstream, rather than before it. A guardrail whose hook
returns a brand-new request dict (instead of mutating or spreading the one it
received) would otherwise discard the marker, letting the deployment hook
re-run the guardrail a second time.
(cherry picked from commit
|
||
|
Sameer Kankute
|
80f0c38e60
|
fix(proxy): populate access_via_team_ids on /v1/model/info (#30274)
* fix(proxy): populate access_via_team_ids on /v1/model/info
Team metadata enrichment previously only ran on /v2/model/info with
include_team_models=true, leaving /v1/model/info without
access_via_team_ids for project model-picker flows.
Co-authored-by: Cursor <cursoragent@cursor.com>
* docs(dashboard): sync OpenAPI schema for /v1/model/info query params
Add include_team_models and teamId to the generated schema for /model/info
and /v1/model/info after the proxy endpoint gained team-access filtering.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): always return direct_access on /v1/model/info
Set direct_access to true or false on every enriched model so clients
can filter without treating a missing field as ambiguous.
Co-authored-by: Cursor <cursoragent@cursor.com>
* perf(proxy): fail fast when teamId is set without a connected DB on /v1/model/info
Raise the db_not_connected error before building, enriching, and translating the model list instead of after, so a teamId query against a proxy with no database no longer wastes the full enrichment pipeline.
* fix(proxy): fail fast when include_team_models is set without a database
include_team_models=True relies on _populate_team_access_on_models to set
direct_access/access_via_team_ids, which only runs when a database is connected.
Without one, _filter_models_to_user_accessible discarded every model and the
endpoint returned an empty list with HTTP 200. Mirror the teamId guard so the
request fails fast with a clear db_not_connected error before any model-list work.
* fix(proxy): populate direct_access on single-model /model/info lookup
The /v1/model/info list path populates model_info.direct_access (and
access_via_team_ids) when a database is connected, but the
litellm_model_id single-model lookup returned early without it. This
made the two endpoints disagree, breaking the parity assertion in
test_get_specific_model. Run the same population on the single-model
path so both responses match.
* fix(proxy): apply no-DB fast-fail before litellm_model_id branch
The teamId/include_team_models no-DB guard sat after the litellm_model_id
early return, so ?litellm_model_id=X&teamId=Y with no DB returned 200 with
unpopulated access fields instead of the 500 raised on every other path.
Move the guard ahead of the branch so the fast-fail is uniform.
* fix(proxy): apply teamId/include_team_models filters on single-model lookup
The litellm_model_id early-return branch in model_info_v1 populated the
team access fields but returned before the teamId and include_team_models
filters ran, so a single-model lookup surfaced the deployment regardless
of team access when the DB was connected. Run both filters on the
single-model list before returning so the documented query params behave
the same with and without litellm_model_id.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adaptation: the generated ui schema.d.ts hunk is dropped (absent on this line,
regenerated at UI build, no Python dependency). proxy_server.py and the test apply
verbatim once the model_info_v1 BYOK pipeline is present from #29731 and #30025.
(cherry picked from commit
|
||
|
Sameer Kankute
|
b083ebef7d
|
fix(proxy): align /v1/model/info with router deployments (#30025)
* fix(proxy): align /v1/model/info with router deployments Return router model_list entries (including team-scoped models) with team access metadata instead of wildcard-expanded names from get_complete_model_list. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): gate v1 team filter and honor key allowlists Only apply get_all_team_and_direct_access_models for admin or user-bound keys, then intersect with key/team model restrictions to avoid empty lists for service tokens and metadata leaks for restricted keys. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): skip v1 team filter when user row is missing Require a DB-backed user before applying team-access filtering on /v1/model/info, and skip the trailing filter in get_all_team_and_direct_access_models when user context cannot be resolved. Co-authored-by: Cursor <cursoragent@cursor.com> * Revert "fix(proxy): skip v1 team filter when user row is missing" This reverts commit |
||
|
yuneng-jiang
|
91b2013644
|
fix(proxy): stop team BYOK model name corruption on model edit (#29731)
* fix(proxy): stop team model name corruption on edit (#28382) (#29001)
Team-scoped ("Team-BYOK") models store an internal routing key
model_name_{team_id}_{uuid} in the model_name column and the user-facing
name in model_info.team_public_model_name. The internal name leaked into
/v1, /v2, and /model/info responses; the dashboard bound its edit form to
it, so any non-rename save (e.g. a TPM tweak) PATCHed the internal name
back. The update path then treated it as a rename, overwriting
team_public_model_name and rewriting the team's models[] ACL with the
mangled string -- breaking team key calls with team_model_access_denied.
Two-layer fix:
- Read path (root cause): add _translate_model_name_for_response and apply
it in model_info_v2 and _get_proxy_model_info so /v1, /v2, and
/model/info surface the public name for team-scoped rows. The DB column
and router index keep the internal name as the routing key; this is a
presentation-layer swap on a shallow copy (never mutates input).
- Write path (defense in depth): harden _get_public_model_name so a value
matching the internal shape, or a no-op against the current DB column,
is never treated as a rename -- for both the top-level model_name and an
explicit model_info.team_public_model_name.
Tests: regression for the reported scenario, full branch coverage of
_get_public_model_name, two internal-shape guard cases, an end-to-end
PATCH through _update_team_model_in_db (asserts the team ACL is untouched),
and four response-translation cases. 60 passed (model management),
181 passed (proxy server).
* fix(ui): key Agent Builder agent selection on model_info.id (#29729)
* fix(ui): key Agent Builder agent selection on model_info.id
Once team-scoped BYOK models can share a public name (the backend now
returns the public name on /model/info instead of the internal routing
key), selecting agents by model_name collides. Key selection, create,
update and delete on the stable model_info.id instead, falling back to
model_name only for config-defined agents that have no id.
* fix(ui): add name-match fallback to post-create agent selection
If the just-created agent's id is not yet present in the re-fetched
list, try matching by name before falling back to the first agent.
Addresses greptile review on #29729.
---------
Co-authored-by: tushar8408 <32977767+tushar8408@users.noreply.github.com>
(cherry picked from commit
|
||
|
yuneng-jiang
|
44ff751e54
|
fix(proxy): return deprecated-key lookup result directly in get_data combined view (#30327)
The grace-period branch assigned the recursive get_data result (a
finished LiteLLM_VerificationTokenView) back into the variable that the
combined-view dict normalization then subscripts, raising TypeError on
every request made with a rotated key inside its grace window; auth
surfaced that as a 401. Return the recursive result directly instead.
Regression test drives the full get_data flow: old hash misses the view,
deprecated table resolves to the active token, and the call must return
the view object
(cherry picked from commit
|
||
|
Sameer Kankute
|
f59192b87a
|
fix(passthrough): skip [DONE] sentinels and non-JSON SSE frames in Anthropic streaming logging
Targeted subset of staging commit |
||
|
Yassin Kortam
|
82a971a730
|
fix(passthrough): resolve costing model when body model is unknown (#30160)
Adaptation: in the test file, #30160's six own tests (cost-calculation model
resolution + extract_model) are taken; the trailing
test_passthrough_logging_sets_response_cost_with_server_tool_use_dict, which is
diff context present at the PR's base but not on this line, is dropped. Source
handler applies verbatim.
(cherry picked from commit
|
||
|
Yassin Kortam
|
c58c59e6b3
|
fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures (#29986)
Adaptation: this line's auth_exception_handler.py does not import
seed_request_identity. The OTEL request-identity feature has its call site in
user_api_key_auth.py here; the second call site staging added in the exception
handler is not on this line. The picked tests mock
auth_exception_handler.seed_request_identity to keep the failure path off OTEL,
so those four patch entries (three in test_auth_exception_handler.py, one in
the test_user_api_key_auth.py builder helper) are removed because the symbol is
absent and the handler never calls it here. The 503/401 behavioral assertions
are unchanged and pass. Separately, the one-line context hunk that reformats an
assert inside test_auto_register_passes_validated_org_context_to_generated_key
is dropped; that test is not present on this line. Source files apply verbatim.
(cherry picked from commit
|
||
|
Yassin Kortam
|
f8c22cdd8a
|
feat(proxy): add option to disable server-side prepared statements for DB lookups (#29984)
Adaptation: the generated ui/litellm-dashboard/src/lib/http/schema.d.ts hunk
is dropped; that file is absent on this line and is regenerated at UI build,
with no Python dependency. Source changes apply verbatim.
(cherry picked from commit
|
||
|
Yassin Kortam
|
c4a7b27e91
|
fix(proxy): recover from cached-plan errors by reconnecting the Prisma client (#29983)
(cherry picked from commit
|
||
|
yuneng-jiang
|
dac0f13ad9
|
test(proxy/utils): pin PrismaClient get_data behavior (subset of #29488)
Scaffolding dependency for the #29983 and #30327 regression tests, which
modify test_prisma_client_get_data.py. Only conftest.py and
test_prisma_client_get_data.py are taken from #29488; the other twelve test
files in that PR pin behavior unrelated to this backport and are dropped.
test_prisma_client_get_data.py and conftest.py are self-contained (no imports
from the dropped siblings) and pass on this line.
(cherry picked from commit
|
||
|
Armaan Sandhu
|
6a8e568f8a
|
feat(proxy): add disable_budget_reservation general setting (#27639) (#29493)
* feat(proxy): add disable_budget_reservation general setting (#27639) * feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639) * docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639) * ci: re-trigger flaky docker build (prisma generate ECONNRESET) * fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639) Provenance: #29493 merged into litellm_oss_staging_080626, which was promoted to litellm_internal_staging by the aggregator squash |
||
|
ishaan-berri
|
24b9655cd4
|
fix: completion_cost AttributeError on streaming Anthropic web_search responses (#26153) (#27346)
Cherry-picked from staging squash |
||
|
Kent
|
ce5604413b
|
feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (#29788)
Applied as the squash diff of PR #29788 (head |
||
|
Kent
|
655e531846
|
feat(bedrock_mantle): route Responses API to native OpenAI endpoint (#29490)
Backport prerequisite for #29788. Applied as the squash diff of PR #29490,
which landed upstream inside the litellm_oss_staging_040626 sync
(
|
||
|
yuneng-jiang
|
c3edd95666
|
fix(guardrails): read CrowdStrike AIDR identity from both metadata bags (#29991)
Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first
(cherry picked from commit
|
||
|
Kenan Yildirim
|
2723601f1e
|
feat(guardrails): capture user and model metadata in CrowdStrike AIDR
(cherry picked from commit
|
||
|
Sameer Kankute
|
beb460b79d
|
fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009)
* fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit |
||
|
Mateo Wang
|
a2ab173a5f
|
Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI (#30064)
* Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI
Adds cost map entries for claude-fable-5 ($10/$50 per MTok, 1M context,
128K output, adaptive thinking only) on the Anthropic API, Bedrock
converse (base, global, and us/eu geo inference profiles at the 10%
regional premium), Vertex AI, and Azure AI (Microsoft Foundry, which
serves Fable 5 with the full 1M context window unlike Opus 4.8).
Registers anthropic.claude-fable-5 in BEDROCK_CONVERSE_MODELS, lists the
model in the setup wizard, and extends the reasoning effort e2e grid.
The Bedrock, Vertex, and Azure grid cells carry fail_reason markers
until the CI accounts are provisioned: Bedrock needs the provider data
sharing opt-in Fable 5 requires, and the Foundry resource needs a
claude-fable-5 deployment.
The first-party entry carries provider_specific_entry {us: 1.1} for the
inference_geo premium and deliberately no fast multiplier since Fable 5
has no fast mode.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drop removed sampling params for Claude 4.7+ when drop_params is set
Fable 5, Opus 4.7, and Opus 4.8 removed sampling params: the API rejects
top_p, top_k, and any temperature other than 1 with a 400. LiteLLM was
forwarding them even with drop_params enabled because the Anthropic and
Bedrock converse transformations passed temperature/top_p through
unconditionally.
Mirror the GPT-5/o-series handling: temperature=1 still passes through,
other values and any top_p are dropped when drop_params is set, and
without drop_params a clean client-side UnsupportedParamsError tells the
caller how to opt in, instead of surfacing the raw provider error.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Drive sampling param gating from the cost map and cover top_k
Greptile review follow-ups on the sampling param fix: the restriction for
Fable 5 / Opus 4.7 / 4.8 is now declared as supports_sampling_params: false
on every affected cost map entry (perplexity excluded; that route is
OpenAI-compatible and maps sampling params upstream) and read back through
a tri-state map lookup, keeping the name check only as a fallback for
provider-routed ids whose hosted map entries predate the flag, the same
layering supports_adaptive_thinking uses. top_k bypasses map_openai_params
as a provider-specific kwarg, so it is gated at the shared
AnthropicConfig.transform_request boundary (direct, Bedrock invoke, Vertex,
Azure) and in the Bedrock converse _handle_top_k_value path, with
drop_params threaded through the converse transform helpers.
Also updates the reasoning effort grid cell count assertion for the four
Fable 5 rows added on this branch (29 x 11 cells).
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* Declare supports_sampling_params in the cost map schema
The model map validation schema uses additionalProperties: false, so the
new flag must be declared for the 28 entries that carry it; this was the
one failing job (misc / Run tests) on the previous commit.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix(bedrock): gate top_k=0 on converse to match Anthropic boundary
Truthiness check let top_k=0 silently disappear on models that removed
sampling params, while AnthropicConfig.transform_request treats 0 as
present and raises UnsupportedParamsError (or drops when drop_params is
set). Switch to 'is not None' so converse, direct Anthropic, invoke,
Vertex, and Azure all behave the same for top_k=0.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
|
||
|
Sameer Kankute
|
bde16ff5b7
|
test(proxy): patch utils.get_server_root_path in passthrough auth tests
After removing get_server_root_path from pass_through_endpoints, route and JWT tests must mock litellm.proxy.utils where normalization reads it. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
3d00874438
|
fix(proxy): match passthrough registry routes bare-to-bare with SERVER_ROOT_PATH
After #28547, get_request_route strips the deployment prefix while registry lookup still re-inflated stored paths via SERVER_ROOT_PATH, causing 404s under paths like /llmproxy/ml. Compare normalized bare routes in both is_registered_pass_through_route and get_registered_pass_through_route. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
mateo-berri
|
c5d214cf81
|
fix(key_generate): harden GHSA-q775 session-token exemption against default_key_generate_params
Capture _requested_team_id before the default_key_generate_params loop runs and
key the UI/CLI session-token budget-ceiling exemption off it, instead of the
post-defaults data.team_id. On an install that sets
default_key_generate_params.team_id, a session token requesting a personal key
(no explicit team_id) would otherwise have data.team_id auto-filled, flipping
is_ui_session_team_key on and bypassing the delegated-authority ceiling -- the
exact escalation GHSA-q775 closed. Mirrors the existing pre-defaults capture of
_requested_max_budget. Adds a regression test.
https://claude.ai/code/session_01RT583b1khYC3wjLrQ5hT5h
(cherry picked from commit
|
||
|
yuneng-jiang
|
26e9689c73
|
fix(key_generate): exempt UI/CLI session tokens from the budget ceiling for team keys (#29612)
Non-admin users creating a team key through the UI were rejected with
"max_budget cannot exceed the caller's own max_budget (0.25)". The request is
authenticated by a UI/CLI session token whose max_budget is the per-session chat
spend cap (max_ui_session_budget, default $0.25), and the delegated-authority
budget ceiling (GHSA-q775-qw9r-2r4g) treated that cap as a delegation limit.
Skip the ceiling only when a session token creates a team key (data.team_id set);
that key's spend is bounded by the team budget at request time. Personal keys and
every other non-admin caller keep the ceiling, so a session token cannot mint an
arbitrary-budget personal key.
(cherry picked from commit
|
||
|
Mateo Wang
|
ed04563db5
|
fix: passthrough endpoints duplicate logs (#29598)
* fix duplicate cost callbacks for anthropic streaming pass-through
Two bugs caused _PROXY_track_cost_callback to see stream=True +
complete_streaming_response=None on every streaming pass-through request,
making the dedup guard in dispatch_success_handlers permanently inactive:
1. pass_through_endpoints.py created the Logging object with stream=False
for all requests. _is_assembled_stream_success short-circuits on
self.stream is not True, so has_dispatched_final_stream_success was
never set and any second dispatch went through unchecked.
Fix: set logging_obj.stream = True after stream detection.
2. _create_anthropic_response_logging_payload set complete_streaming_response
inside the try block after litellm.completion_cost(), so a pricing error
caused an early return without setting it on model_call_details.
Fix: set complete_streaming_response before the try block.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix stream
* add stream to logging obj
* test(pass_through): give mock logging object a real model_call_details dict
The anthropic passthrough logging payload now records the assembled
response on model_call_details before cost calculation, which requires
model_call_details to support item assignment. In production it is always
a dict; the existing unit test stubbed the logging object with a bare Mock
whose attribute is not subscriptable, so the new assignment raised
TypeError. Use a real dict to match the production logging object.
* test(pass_through): cover streaming logging-obj stream flag
The streaming branch of pass_through_request that marks the logging object
as streaming (logging_obj.stream and model_call_details["stream"]) had no
unit coverage, so the patch coverage gate flagged it. Add a regression test
that drives a streaming pass-through request through pass_through_request and
asserts the logging object is flagged as a stream before dispatch.
* test(pass_through): cover SSE-response stream flag fallback branch
The auto-detected streaming branch of pass_through_request (when a request
that was not flagged as streaming returns a text/event-stream response) sets
logging_obj.stream and model_call_details["stream"] but had no unit coverage,
so the codecov patch gate failed at 60%. Drive a non-streaming pass-through
request whose upstream response is SSE through pass_through_request and assert
the logging object is flagged as a stream before dispatch.
* fix(pass_through): gate complete_streaming_response on stream flag
perform_redaction only scrubs complete_streaming_response when
model_call_details["stream"] is True. Setting it unconditionally for
non-streaming Anthropic pass-through responses left the assembled
response unredacted in model_call_details, which is handed to logging
callbacks as kwargs when message logging is disabled. Only record it for
actual streaming responses so redaction always applies.
---------
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
(cherry picked from commit
|
||
|
Mateo Wang
|
2110b41a16
|
fix(vertex): strip output_config.effort for Vertex Claude models that reject it (Haiku 4.5) (#29585)
* fix(vertex): strip output_config.effort for models that reject it
Haiku 4.5 on Vertex AI does not support output_config.effort and 400s with
"output_config.effort: Extra inputs are not permitted". PR #27074 emptied
VERTEX_UNSUPPORTED_OUTPUT_CONFIG_KEYS so effort would forward for Opus/Sonnet
4.6+, but that made the strip unconditional across every Vertex Anthropic
model, including ones that don't support it. Claude Code injects effort into
its default Messages payload, so `claude --model claude-haiku-4.5` started
failing.
Make the sanitizer model-aware: drop output_config.effort for models that
don't advertise output_config support (or any reasoning effort level) while
forwarding it for those that do. The fix covers both the chat-completion and
Messages pass-through transformation paths since they share the helper.
* chore(vertex): log at debug when dropping unsupported output_config.effort
Operators pointing an unregistered Vertex Claude alias that does support
effort would otherwise see it stripped with no signal. Debug level keeps it
out of normal logs since Claude Code sends effort on every request.
(cherry picked from commit
|
||
|
milan-berri
|
8373712086
|
fix(key_generate): allow team members to create keys on org-scoped teams (#29310)
* fix(key_generate): allow team members to create keys on org-scoped teams
When a virtual key is created for a team, enterprise logic inherits the
team's organization_id onto the key (add_team_organization_id). Since the
VERIA-55 org-IDOR fix, /key/generate then required the caller to be an
explicit LiteLLM_OrganizationMembership member of that org, returning
403 "Caller is not a member of organization_id=<uuid>". Admins normally
only add users to teams (not orgs), so self-serve key creation regressed
for any user on an org-scoped team (regression since v1.84.0-rc.1).
Skip the org-membership check when organization_id was inherited from the
key's team (organization_id == team_table.organization_id). Team-level
authorization already gates this path, so team membership is sufficient.
The membership check still runs when a caller assigns an organization_id
that did not come from the key's team, preserving the IDOR protection.
Adds regression tests covering both the team-inherited (allowed) and
foreign-org (still blocked) cases.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(key_generate): cover mismatched team org IDOR path on generate
Add test_generate_key_foreign_org_with_mismatched_team_still_enforces_membership
for the case where a team is present but request organization_id differs from
team_table.organization_id. Enterprise inheritance is no-op'd in the test so
the guard is exercised directly; membership validation must still run.
Addresses Greptile review on #29310.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit
|
||
|
Shivam Rawat
|
521ddc3b18
|
fix(proxy): resolve managed video model ids for auth (#29545)
* fix(proxy): resolve managed video model ids for auth
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(proxy): cover character_id router model resolution
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
(cherry picked from commit
|
||
|
michelligabriele
|
80cf50dedb
|
fix(v3 limiter): cap no-max_tokens TPM floor at smallest configured limit (#28805) | ||
|
michelligabriele
|
117136cccc
|
fix(openai-moderation): wire streaming flags through to unified dispatcher (#27324) | ||
|
yuneng-jiang
|
7d1bd9d9f4
|
fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter (#29358)
* fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter
ResetBudgetJob's batched update_data path shipped the full key/user/team
model on each reset. Prisma rejects object_permission_id and budget_limits
on the update input type, so any row carrying those fields detonated the
entire batch -- spend never reset, budget_reset_at never advanced. After
v1.84.0 started populating object_permission_id on UI-created keys, this
fires routinely.
_reset_budget_common also zeroed the cross-pod spend counter before the
DB write, so failed resets left enforcement reading 0 from the counter
while the DB still held the over-budget spend, admitting requests past
the cap until the counter naturally re-saturated from new reservations.
Switch the write to per-row narrow updates ({spend, budget_reset_at})
via db.batch_, and move the counter invalidation out of
_reset_budget_common so it only fires after the DB write commits. On
DB-write failure the counter is left untouched, enforcement continues
to block, and the next scheduler tick can retry without leaving a
bypass window.
Fixes #27730.
* fix(reset_budget): address Greptile review on #29358
- Strengthen the bypass-half regression test: replace the for-loop over
call_args_list (vacuously true when empty) with assert_not_called(),
so the test would actually flag a re-introduction of counter-zeroing
via any code path.
- Add the same explanatory docstring on _write_user_reset_updates and
_write_team_reset_updates that _write_key_reset_updates already has,
so all three helpers point future maintainers at #27730.
* test(reset_budget): update test_proxy_budget_reset for new batch-write path
Same shape as the previous test_reset_budget_job.py update: keys/users/teams
now write through prisma.db.batch_().<table>.update, not update_data, so the
tests need a batcher mock and updated assertions. Adds:
- _wire_batcher_for_test helper that returns a list which accumulates per-row
batch updates captured from prisma_client.db.batch_().
- _attrify helper that wraps dict fixtures so getattr(item, "token") works
alongside the dict item-access the fake_reset_* mocks rely on. The new
narrow-write helpers use getattr to pull out the row's id, and would
silently skip plain dicts otherwise.
- Updates 3 partial_failure tests to assert against the batch-call list
(rows by id, payload contains only {spend, budget_reset_at}) instead of
update_data.assert_awaited_once + data_list inspection.
- Updates test_reset_budget_continues_other_categories_on_failure: only
budget + enduser still flow through update_data; key/user/team go through
the batch path now.
- Wires the batcher mock into 3 service_logger_*_success tests so commit()
is actually awaitable and the success hook fires.
These tests were silently passing locally only because the editable install
in .venv pointed at the main repo, not the worktree — running pytest with
PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
|
||
|
ryan-crabbe-berri
|
90b5104475
|
feat(mcp/auth): additive key access-group grants + opt-in member assignment (#29313)
* fix(mcp): make key.access_group_ids grants additive over team ceiling A key whose unified access_group_ids grant a private MCP server was having that grant intersected against its team's MCP ceiling, so a key in a team scoped to other servers (or with no own scope) lost the granted server entirely. Resolve access_group_ids once as ungated additive grants and union them on top of the key/team ceiling instead of folding them into the key scope that gets intersected. * test(mcp): align key access-group tests with additive-grant model The previous commit moved key.access_group_ids resolution out of the intersected key ceiling (_get_allowed_mcp_servers_for_key) and into the ungated additive grant path (_get_key_access_group_mcp_server_extras), unioned on top of the team ceiling. Five tests from #28890/#29195 still asserted the old gated / in-key-scope contract and failed: - _get_allowed_mcp_servers_for_key now returns the object_permission ceiling only and never resolves access_group_ids; two tests now assert the group resolver is not called from that path (with and without an object_permission present). - The extras path is ungated, so a group whose assigned_team_ids / assigned_key_ids exclude the caller still contributes its servers. - The end-to-end test asserts the grant surfaces via the extras path rather than the base key path. - Dropped test_key_access_group_ids_empty_returns_no_extras; the empty case is already covered by the extras family's no-groups test. * feat(auth): gate member access-group assignment on keys behind opt-in Non-admin team members could attach access_group_ids to keys they create or update, letting them self-grant resources (MCP servers/models) the team admin never intended. Add an opt-in KEY_ACCESS_GROUP_ASSIGNMENT team-member permission (default-deny) enforced at /key/generate and /key/update; proxy and team admins bypass. Surfaces automatically as a checkbox in the team Member Permissions UI. * fix(auth): gate access-group assignment on /key/regenerate too RegenerateKeyRequest inherits access_group_ids and prepare_key_update_data persists it, so a non-admin key owner could self-grant access groups by regenerating. Apply the same opt-in member gate using the existing key's team. * test(auth): cover member access-group gate and additive MCP grants Add unit tests for enforce_member_can_assign_access_groups (deny without opt-in, allow with opt-in, and proxy-admin / team-admin / non-team-key bypasses) and for _get_key_access_group_mcp_server_extras (no-auth and no-resolved-servers return empty, resolved ids are expanded, errors degrade to no grants). |
||
|
Shivam Rawat
|
dc4f5b12ef
|
fix(proxy): enforce allowed_passthrough_routes for auth=true pass-thr… (#29256)
* fix(proxy): enforce allowed_passthrough_routes for auth=true pass-through
Pass-through endpoints with auth=true were injected into openai_routes,
so teams with openai_routes access bypassed per-team allowed_passthrough_routes.
Gate auth-enforced pass-through at JWT, virtual-key, and non-admin route checks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): clarify JWT passthrough denial
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): make pass-through auth checks method-aware
Prevent allowlist bypass when the same path is registered with different auth settings per HTTP method.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix passthrough route auth checks
* fix(proxy): reject unregistered pass-through HTTP methods
Enforce method-aware JWT checks and return 405 when stale FastAPI routes accept requests outside the current pass-through registry.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(proxy): remove duplicate request_method in JWT team lookup
Fixes SyntaxError on proxy startup caused by passing request_method twice to find_team_with_model_access.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix passthrough route auth enforcement
* fix(proxy): raise passthrough-specific 403 directly in virtual-key path
* fix(proxy): load team for RBAC role-claim JWT passthrough gating
* Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326)
This reverts the Bedrock CI account migration (#28728). The original account
(888602223428) was put under an AWS security restriction after a leaked key
and has since been reactivated, while the replacement account (941277531214)
lacks access to several models the suites exercise (legacy Bedrock Claude 3
models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship
Opus). Pointing CI back at the reactivated account restores that coverage.
This is the exact inverse of #28728: all hardcoded 941277531214 references go
back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs
and their suffixes, batch execution role ARN, and the example proxy config),
the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs
revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge
Base revert to their original ids, and the live-call tests go back to the
legacy model strings. The grid_spec fail_reason workaround for the unentitled
Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field
added after the migration.
The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at
941277531214 and must be set to the reactivated account's fresh credentials
separately via the CircleCI API; AWS_REGION_NAME stays us-west-2.
(cherry picked from commit
|
||
|
Sameer Kankute
|
ba2699740c
|
feat(pass_through): extend passthrough_managed_object_ids to Azure (#29160)
* feat(pass_through): extend passthrough_managed_object_ids to Azure
Adds managed ID minting/resolution for Azure passthrough endpoints
(/azure/...) alongside the existing OpenAI passthrough support.
Key changes:
- pass_through_endpoints.py: detect azure/azure_ai custom_llm_provider
(string or enum) to set _is_managed_id_provider and _managed_id_provider;
both INPUT and OUTPUT rewrite blocks now fire for Azure.
- llm_passthrough_endpoints.py: forward custom_llm_provider into
create_pass_through_route so it reaches pass_through_request (was None).
- managed_id_rewriter.py: extend _PASSTHROUGH_PREFIX_RE and _canonical_path
to strip /azure/openai prefix and add /v1/ for Azure paths that omit it;
add ("azure", method, path) entries to BUILTIN_OUTPUT_ID_FIELD_MAP for
files and batches endpoints.
- managed_id_codec.py / types/utils.py: supporting codec and enum constant.
- proxy_server.py: register llm_passthrough_router before batches_router to
prevent route collision for /openai_passthrough/* paths.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): remove unused imports for ruff F401
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): satisfy mypy for optional parsed_body
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(pass_through): compute query params string after managed-ID rewrite
Move requested_query_params_str computation to after the managed-ID
input rewrite block so logging_url reflects the rewritten raw-provider
query params actually sent upstream, instead of the original
managed IDs.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Add support for managed ids for passthrough responses api
* Add support for list batches and list files
* style: run Black on passthrough managed ID files
Fix CI formatting for managed_id_rewriter.py and pass_through_endpoints.py.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(passthrough): parse json file_object and implement before-cursor pagination
- Parse row.file_object via json.loads when Prisma returns it as a string;
mirrors openai_files_endpoints/common_utils.py so list responses keep all
stored detail fields (status, timestamps, etc.).
- Implement the previously-parsed-but-unused 'before' cursor for list
pagination by flipping fetch order to ascending with a 'gt' bound on
created_at, then reversing rows so the response stays newest-first.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* Remove logger
* refactor: split list_passthrough_ids_from_db to fix PLR0915
Extract pagination, fetch, and serialization helpers so the main list
function stays under the statement limit without changing behavior.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: scope passthrough managed ID dedup and list by provider
Validate embedded provider before reusing deduped file/object rows so
OpenAI and Azure cannot share the same managed ID for an identical raw ID.
Filter list responses to rows whose managed IDs decode to the current
provider, with over-fetch scanning when needed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(managed_id_rewriter): cap pagination trim at effective limit
When raw_limit > 100, fetch_limit is capped at 101 (one extra row to
detect has_more), but trimming with rows[:raw_limit] failed to drop
the sentinel row. Use the capped effective limit instead.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: cross-provider object collision and fail-closed list error handling
Greptile P1: move provider check before access check in _mint_or_reuse_object
so a cross-provider raw ID collision (OpenAI and Azure share the same batch_
ID) falls through to mint a new provider-scoped row instead of raising 404.
Veria-ai medium: _fetch_provider_scoped_list_rows now always returns
(page, has_more) — DB errors break out of the scan loop and return matched
rows so far. list_passthrough_ids_from_db never returns None for a recognised
list route, so the caller can never fall through to the upstream provider.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: namespace passthrough model_object_id by provider to prevent unique violation
Store model_object_id as 'passthrough:{provider}:{raw_id}' instead of the
bare raw ID so OpenAI and Azure can each own a row for the same raw batch ID
without hitting the @unique constraint. Dedup lookup uses the same namespaced
key so it is implicitly provider-scoped and the _managed_id_matches_provider
check is no longer needed on the object path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: gate list interception on managed_files hook like input/output rewrites
Without the hook no managed IDs are minted so the DB is empty. Intercepting
GET /v1/files without the hook returned an empty list and hid the caller's
real upstream files/batches. Matches the guard used by the input and output
rewrite blocks.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: set has_more=True when scan cap is hit with a full final DB batch
When max_scans (20) is exhausted and the last DB page was full-sized,
there are almost certainly more rows beyond the scan window. Track
last_batch_full across iterations so the scan_cap_hit condition sets
has_more=True in that case, preventing silent pagination truncation in
high-mixed-provider pools.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(managed_id_rewriter): scope pagination cursor lookup to caller-owned rows
Prevent a cross-tenant timing oracle by constraining the after/before
cursor row lookup to the caller's owner_filter, and cover the real Azure
responses path form (no /v1/) in tests.
* fix(managed_id_rewriter): align passthrough list metadata with direct GET
Persist upstream file metadata when minting a managed file ID and rewrite
nested batch file IDs before snapshotting the object, so DB-served file/batch
list responses return the same fields and managed IDs as a direct endpoint
GET.
* fix(managed_id_rewriter): degrade to raw id on cross-owner object collision
The OUTPUT (mint) path of _mint_or_reuse_object raised HTTPException(404)
when a dedup hit on the namespaced model_object_id belonged to a different
owner, converting a successful upstream batch/response creation into a 404
for the caller. Two upstream accounts under one provider name can issue the
same raw id, so this is reachable in multi-tenant deployments.
Return the caller's raw id unmanaged instead: the upstream create already
succeeded, a new managed row can't be minted (model_object_id is @unique),
and reusing the other owner's managed id would later fail the access check.
* fix(managed_id_rewriter): scope list cursor by provider and cap body-rewrite recursion depth
* perf(managed_id_rewriter): push batch list provider scope to DB and anchor canonical-path prefix
Object (batch) list rows store model_object_id as passthrough:{provider}:{raw},
so the provider filter is now applied at the indexed DB column, collapsing the
application-layer multi-scan to a single query for that table. File rows keep
the decode-based scan since they have no provider column.
Anchor the canonical-path prefix regex at a path boundary so routes such as
/openai_realtime/... are no longer mis-stripped.
* fix(managed_id_rewriter): refresh stored batch snapshot on reuse
The dedup-reuse path in _mint_or_reuse_object returned the existing managed id
without updating the stored file_object, so DB-served list responses kept the
creation-time snapshot and showed null output_file_id/error_file_id even after
the batch completed. Refresh the snapshot when an owned row is reused so the
list reflects the batch's latest state.
* fix(managed_id_rewriter): deny cross-owner object access on retrieve/cancel/delete
Returning the raw id when can_access_resource fails only made sense for create
responses, where the caller's own upstream create succeeded under a raw id that a
different owner already holds. On retrieve/cancel/delete the caller reaches that
branch only by supplying another tenant's raw id (which bypasses the managed-id
input gate), so echoing the upstream object back leaked it cross-tenant. Restrict
the raw fallback to create routes and return 404 otherwise.
* fix(managed_id_rewriter): deny cross-owner file access on retrieve/delete
_mint_or_reuse_file scoped the raw file dedup lookup to the current caller, so a
raw file-... id belonging to another tenant was never found and the OUTPUT path
minted a fresh managed id for that same upstream file under the caller. A raw id
only reaches this path by skipping the managed-id input gate (raw provider ids
are opt-out), so a different-owner row means the caller is touching someone
else's file. Look up flat_model_file_ids globally and run can_access_resource;
deny with 404 on retrieve/delete and leave the raw id unmanaged on create, which
mirrors the cross-owner handling already in _mint_or_reuse_object.
* fix(managed_id_rewriter): deterministic provider-scoped file dedup
Replace the unscoped find_first in _mint_or_reuse_file with a find_many
ordered by created_at and an application-layer provider filter. The file
table has no provider column, so a raw file id shared across OpenAI and
Azure could map to one row per provider; find_first then picked a row
non-deterministically and, on a provider mismatch, minted a fresh managed
row on every call, accumulating duplicates. Selecting the oldest matching
same-provider row the caller can access keeps reuse stable and prevents
duplicate rows while preserving the cross-tenant deny/leave-raw behaviour.
* refactor(pass_through): scope passthrough managed IDs on the explicit provider
Move the openai/azure detection out of pass_through_request into
resolve_passthrough_managed_id_provider in llms/base_llm/managed_resources,
and key managed-ID rewriting on the forwarded custom_llm_provider rather than
the upstream URL's EndpointType. The helper documents why azure and azure_ai
collapse to one "azure" scope (they share the same Azure OpenAI files/batches
surface, so an ID minted on one must resolve on the other) and returns None for
any other provider so a third-party OpenAI-compatible endpoint never triggers
managed-ID minting.
Add TestManagedIdProviderScope covering the azure_ai -> azure collapse and the
non-openai/azure exclusion.
* test(log_db_metrics): assert sanitized event_metadata contract
test_log_db_metrics_success still asserted the legacy event_metadata
shape (function_name/function_kwargs/function_args), which #28909
intentionally removed so that live Prisma clients, OTel spans, and
secrets never land on a service-log span. The decorator now emits only
a sanitized payload: None when no table_name is present, and
{"table_name": ...} when it is. Update the test to verify both branches
of that contract.
* fix(managed_id_rewriter): page provider-scoped file list by offset
The file list scan advanced its cursor with a strict created_at boundary.
When several rows shared a created_at timestamp and a non-matching provider
row sat on the page boundary, the next query skipped the remaining rows at
that timestamp, dropping matching files from the response. Page by a stable
offset over a total order (created_at plus the unique id column) so tied
rows are never skipped or repeated.
* fix(managed_id_rewriter): push file-list provider scope to the DB
The file-list helper had no provider column to query, so it scanned the
table and filtered by decoding each managed ID in the application layer,
capped at 20 pages. For an admin with a large mixed-provider file pool
that cap could truncate a page.
Mint now writes a _passthrough_provider:{provider} marker into
flat_model_file_ids, giving the file table the same DB-queryable provider
scope object rows already get from the namespaced model_object_id. The
list helper pushes the scope into the query so a single round-trip serves
the page. The scan loop, the offset paging, and the cap are gone, so pages
can no longer truncate, leak the other provider, or skip rows that share a
created_at timestamp.
* fix(managed_id_rewriter): deny raw provider IDs that map to another tenant's managed resource
Clients only ever receive managed IDs on passthrough, so a raw file/batch/response ID for another tenant's managed object can only be recovered by decoding that tenant's managed ID. Raw IDs were forwarded upstream untouched (deliberate opt-out), which on a retrieve/cancel/delete executed upstream before the response-side ownership check ran, leaking a cross-tenant action.
Guard raw provider IDs on the input path: when a raw file-/batch_/resp_ ID resolves to a managed row the caller cannot access, return 404 before forwarding. Genuinely unmanaged raw IDs (no DB row) and IDs the caller owns are left untouched, preserving the opt-out.
* test(managed_id_rewriter): cover azure_ai and pre-versioned azure passthrough paths
* fix(managed_id_rewriter): fall back to raw id when persistence fails
A DB persistence failure after a successful upstream create left the
client holding a minted managed ID with no backing row, so every later
resolve returned 404 and the resource was permanently unreachable. Mint
the managed ID only when the row is stored; on persistence failure return
the raw provider id, matching the no-persistence-available fallback, so
the freshly-created resource stays reachable.
* fix(managed_id_rewriter): log only rewritten query param keys
* fix(managed_id_rewriter): use compound (created_at, id) list cursor boundary
A timestamp-only lt/gt cursor boundary skips list rows that share the
cursor row's created_at across a page boundary, silently dropping them.
Compare the unique id (the secondary sort key) alongside created_at so
the page walk stays complete when timestamps tie.
* fix(managed_id_rewriter): converge concurrent object creates on one managed id
* fix(managed_id_rewriter): bound raw-id guard DB lookups per request
The INPUT guard fired one DB lookup for every file-/batch_/resp_ prefixed
string in the path, query, and body. The file-id guard is an unindexed
array-containment scan over LiteLLM_ManagedFileTable, so an authenticated
caller could amplify a single passthrough request into thousands of
full-table scans by packing a body with id-shaped strings.
De-dupe raw ids within a request and cap the distinct guard lookups,
failing closed with 400 instead of skipping the guard. Legitimate callers
hold managed ids (resolved via an indexed unified_*_id lookup, not the
guard), so the cap only trips under abuse.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Shivam Rawat
|
7ca796beb1
|
fix(proxy): restrict vector store index create/delete to proxy admins (#29202)
* fix(proxy): restrict vector store index create/delete to proxy admins Prevent non-admin API keys from registering indexes via POST /v1/indexes or deleting Azure AI Search indexes through managed pass-through routes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): tighten vector store index lifecycle checks Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Shivam Rawat
|
4c3efe9c7c
|
fix(guardrails): return HTTP 400 for litellm content filter blocks (#28418)
* fix(guardrails): return HTTP 400 for litellm content filter blocks Align litellm_content_filter hard rejects with the standard guardrail block status code so clients receive 400 instead of 403. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): return HTTP 400 for custom code guardrail blocks Pre-call custom code guardrail blocks now raise HTTPException(400) instead of using the passthrough ModifyResponseException path that returned a synthetic 200 response. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(guardrails): preserve custom code passthrough blocks Keep standalone custom code guardrail blocks on the passthrough contract while covering policy pipeline block handling for passthrough-style guardrail interventions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Mateo Wang
|
152b1177e5
|
test(reasoning-effort-grid): cover Claude Opus 4.8 across provider routes (#29327)
* test(logging): align DB metrics event_metadata assertions with safe redaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. * test(reasoning-effort-grid): add claude-opus-4-8 across provider routes Adds claude-opus-4-8 to the anthropic, azure, vertex and bedrock-converse routes (275 cells total) so the reasoning-effort wire translation is covered for the new model. The bedrock opus-4-8 and opus-4-7 cells reuse the self-heal path: they run the call and skip only on Bedrock's "is not available for this account" reply, then assert in full once the model is entitled. The azure and vertex opus-4-8 cells stay xfail until a Foundry deployment exists and Vertex availability is confirmed. The shared xhigh+max capability set is renamed to _CAPS_XHIGH_MAX now that more than one model uses it. |
||
|
yuneng-jiang
|
94a043efb2
|
refactor(proxy/auth): normalize Bearer prefix in safe-hash helper (#29343)
* refactor(proxy/auth): normalize Bearer prefix in safe-hash helper
UserAPIKeyAuth._safe_hash_litellm_api_key now strips a leading
"Bearer "/"bearer " prefix before its existing sk-/JWT classification, so
the helper produces the same hashed output regardless of whether the
caller stripped the Authorization header prefix or passed the header
value through unchanged.
* refactor(proxy/auth): make Bearer-prefix strip case-insensitive
Per RFC 7235 the HTTP authorization scheme token is case-insensitive.
Replace the two-prefix loop with a single case-insensitive check so the
helper normalizes "Bearer ", "bearer ", "BEARER ", and any mixed-case
variant before classifying the remainder as sk- or JWT. The contract
test gains coverage of "BEARER " and "BeArEr ".
* test(mcp): align auth-handler test expectations with safe-hash helper
The two MCP auth tests asserted that UserAPIKeyAuth(api_key="Bearer ...")
retained the raw header bytes on the api_key field. _safe_hash_litellm_api_key
now normalizes that input — stripping the Bearer prefix and hashing the
resulting sk- key — so the expectations move to the normalized form:
the bare token in the parametrize case, and hash_token("sk-...") in the
backward-compat assertion. This matches what the real auth flow produces
(the builder strips Bearer and the DB stores the hashed token), so the
mocks now line up with production rather than with the un-normalized
validator output.
|
||
|
Mateo Wang
|
bfbb5d2375
|
fix(ci): make litellm_internal_staging green (logging test + Bedrock Opus 4.7 self-heal) (#29344)
* test(logging): align DB metrics event_metadata assertions with safe redaction PR #28909 hardened log_db_metrics to emit a minimal, non-sensitive event_metadata (only table_name when present, otherwise None) instead of dumping function_name, function_kwargs, and function_args onto the span. The test in test_log_db_redis_services was not updated and still asserted "function_name" in event_metadata, which raised TypeError (argument of type 'NoneType' is not iterable) and turned the logging_testing CI job red on litellm_internal_staging. Update test_log_db_metrics_success to assert event_metadata is None when no table_name is passed, and add test_log_db_metrics_event_metadata_is_safe as a regression guard verifying that only the table name surfaces and that sensitive kwargs (tokens, prisma client) are never dumped. * test(bedrock): self-heal opus-4-7 grid cells when unentitled on CI The bedrock-claude-opus-4-7 converse cells are unentitled on the Bedrock CI account, so they were marked xfail. xfail keeps reporting them as expected failures even after access is granted, so the wire translation never gets verified again. Now the cell makes the call and skips only when Bedrock replies "is not available for this account"; the moment the model is entitled the same cells run their full assertions with no edit. A focused unit test pins the tolerance predicate so any other failure still surfaces loudly and the available path still runs the assertions. |
||
|
Shivam Rawat
|
2d4c13c00f
|
fix(mcp): preserve omitted fields on PUT /v1/mcp/server partial updates (#29253)
* fix(mcp): preserve omitted fields on PUT /v1/mcp/server partial updates Use model_dump(exclude_unset=True) for updates so schema defaults (transport=sse, allow_all_keys=false, etc.) are not written when callers omit them. Serialize JSON fields from the filtered dict and only force is_byok on create. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(mcp): allow explicit alias=None on MCP server partial updates Snapshot caller-provided fields before normalization so omitted alias is not written while an intentional alias=None still clears the stored value. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Shivam Rawat
|
ace3c65ab3
|
fix(mcp): preserve source_url in GET /v1/mcp/server list responses (#29249)
* fix(mcp): preserve source_url in GET /v1/mcp/server list responses
The list endpoint builds responses from the in-memory registry, but
source_url was dropped during the DB-to-registry roundtrip even though
GET /v1/mcp/server/{id} returned it correctly from the database.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests/mcp): set source_url on MagicMock table records
MagicMock auto-creates source_url as a mock object, which fails MCPServer
Pydantic validation after source_url was wired through build_mcp_server_from_table.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
Mateo Wang
|
f11c12d157
|
Revert "chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728)" (#29326)
This reverts the Bedrock CI account migration (#28728). The original account (888602223428) was put under an AWS security restriction after a leaked key and has since been reactivated, while the replacement account (941277531214) lacks access to several models the suites exercise (legacy Bedrock Claude 3 models, Cohere, Nova Canvas image gen, Bedrock batch inference, and flagship Opus). Pointing CI back at the reactivated account restores that coverage. This is the exact inverse of #28728: all hardcoded 941277531214 references go back to 888602223428 (provisioned/imported-model ARNs, AgentCore runtime ARNs and their suffixes, batch execution role ARN, and the example proxy config), the S3 buckets revert to litellm-proxy and load-testing-oct, the guardrail IDs revert to wf0hkdb5x07f and ff6ujrregl1q, the SageMaker endpoint and Knowledge Base revert to their original ids, and the live-call tests go back to the legacy model strings. The grid_spec fail_reason workaround for the unentitled Opus cells is dropped while keeping the unrelated bedrock_effort_ceiling field added after the migration. The CircleCI AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY env vars still point at 941277531214 and must be set to the reactivated account's fresh credentials separately via the CircleCI API; AWS_REGION_NAME stays us-west-2. |
||
|
Shivam Rawat
|
967fed1fa1
|
feat(enterprise): add RESEND_FROM_EMAIL for self-hosted Resend sends (#28830)
Allow self-hosted installs to override the default LiteLLM sender address via RESEND_FROM_EMAIL, matching SendGrid's SENDGRID_SENDER_EMAIL pattern. Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
4cc3dd7aad
|
feat(context_management): compact_20260112 polyfill for non-Anthropic providers (#28868)
* feat(anthropic/messages): in-gateway context_management polyfill for non-Anthropic providers
- Add `context_management/` module with `clear_tool_uses_20250919` editor
dispatched before chat-completions translation on `/v1/messages`
- Hard-protect most-recently completed tool_result from being cleared
- Attach `context_management.applied_edits` to both non-streaming and
streaming (final `message_delta`) responses
- Bedrock Converse: forward `context_management`; filter to
`compact_20260112`-only edits with `compact-2026-01-12` beta header
- token_counter: guard Anthropic-format tools (no `function` key) to
prevent AttributeError during polyfill token counting
- Streaming: handle empty-choices usage-only trailing chunks
- Skip polyfill when `litellm.drop_params = True`
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(bedrock): pop None context_management before sending to Bedrock Converse
If context_management is forwarded as None (e.g. when mapping returns
None for an invalid format), _filter_context_management_for_bedrock_converse
previously returned early without removing the key, leaving
"context_management": null in the request and causing a validation
error. Pop the key when the value is not a dict.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bedrock/converse): pop None context_management; extract helpers to fix PLR0915
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic/messages): check per-request drop_params alongside global
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(anthropic/messages): preserve drop_params for downstream and respect explicit False
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: lazy debug logging in clear_tool_uses; remove unused context_management constants
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic/messages): guard context_management polyfill with try/except
Wrap apply_context_management() in a try/except so any failure (e.g.
litellm.token_counter raising on an unknown tokenizer or unexpected
message format) is logged but does not crash the underlying LLM
request. The polyfill is a best-effort additive feature; on failure we
forward the original messages without applied edits.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(token_counter): guard None input_schema in Anthropic tool fallback
Use `or {}` instead of `.get(..., {})` so explicit null parameters do not
raise AttributeError when formatting function definitions for token counting.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: minimize context_management polyfill threading
- Use None (not empty list) for polyfill_applied_edits when context
management isn't requested, so semantics of 'feature not requested'
vs 'feature requested but no edits applied' are distinct.
- In the streaming iterator, only pass applied_edits to the per-chunk
translator on the final (finish_reason) chunk; intermediate chunks
ignore it anyway, and this makes intent explicit on both sync and
async paths.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(context_management): align tool_use counts and normalize list spec
- _count_tool_uses now requires a string id, matching _collect_tool_use_ids_in_order so the tool_uses trigger can't fire on blocks that aren't clearable.
- apply_context_management dispatcher now accepts the OpenAI list form and normalizes it via AnthropicConfig.map_openai_context_management_to_anthropic, so the polyfill path no longer silently no-ops on list input.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(context_management): add compact_20260112 polyfill for non-Anthropic providers
Implements an in-gateway compaction polyfill that summarizes long conversations
using a configurable model when `compact_20260112` is requested for non-Anthropic
targets (e.g. OpenAI, Gemini), matching Anthropic's context management beta
behaviour for those providers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(compact): skip tool_result-only user turns; bedrock: elif for context_management
- compact_20260112 Phase D: when keeping the last user turn after a full
summary, skip role=user turns whose content is exclusively tool_result
blocks. Such turns translate to OpenAI tool-role messages with no
preceding assistant tool_calls (those got summarized away), which
non-Anthropic providers reject. Fall back to a synthetic continuation
prompt if no eligible user question exists, so the downstream call
always has a non-empty user message.
- bedrock converse: chain the context_management param as elif so it
follows the same if/elif pattern as the surrounding thinking/
reasoning_effort checks.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): post-compaction question selection, system type, sync stream merge
- compact.py: select last user question from effective_messages (post-compaction slice) instead of raw messages, so prior summarized turns aren't reintroduced
- handler.py: widen _prepare_completion_kwargs system parameter type to Union[str, List[Dict]] matching PolyfillResult.system
- streaming_iterator.py: mirror async hold-and-merge logic in sync __next__ so context_management is attached to the final merged message_delta when stop_reason and usage arrive in separate chunks
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic/messages): apply context_management on sync path; clear held stop_reason chunk in async iterator
- Sync `anthropic_messages_handler` was silently dropping the
`context_management` kwarg via `ANTHROPIC_ONLY_REQUEST_KEYS` after the
polyfill was moved into the async handler. Bridge to the async
dispatcher with `run_async_function` so `litellm.messages.create()`
callers keep working (regressed e.g. `clear_tool_uses_20250919`).
- In the streaming iterator's `__anext__` `StopIteration` handler, clear
`self.holding_stop_reason_chunk` after capturing it (matches `__next__`)
so a subsequent call doesn't re-emit the same chunk.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bugfixes): bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(anthropic): silently drop trailing chunks after usage; remove dead _polyfill_result key
- streaming_iterator: in sync __next__, after the usage chunk has been
merged and emitted, silently consume any trailing provider events
via 'continue' instead of forwarding them through the queue. Trailing
chunks would translate to content_block_delta or message_delta and
violate Anthropic SSE ordering after the final message_delta. The
async __anext__ already drops these via 'if not self.queued_usage_chunk:'
gating, so this aligns sync and async behavior.
- handler: drop unused '_polyfill_result' from ANTHROPIC_ONLY_REQUEST_KEYS.
PolyfillResult is passed as an explicit arg to the adapter methods, never
through extra_kwargs, so the entry was dead code.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* refactor(anthropic): extract usage-merge helper; guard empty slice-only compaction result
- Extract the duplicated hold-and-merge usage logic from the sync __next__ and
async __anext__ paths into a shared _merge_usage_into_held_stop_reason_chunk
helper so the subtle cache-token / context_management attachment lives in
exactly one place.
- In the compact_20260112 slice-only path, fall back to _select_last_user_question
when _strip_compaction_blocks produces an empty list (e.g. messages ending on
an assistant turn whose only content was the compaction block) so the
downstream API never receives an empty messages array.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* refactor(anthropic/context_management): streaming iterator compaction fixes and compact polyfill improvements
- Extract usage-merge helper; guard empty slice-only compaction result
- Silently drop trailing chunks after usage; remove dead _polyfill_result key
- Fix bedrock None context_mgmt; stream per-instance queue; sync polyfill; trailing-chunk passthrough
- Apply context_management on sync path; clear held stop_reason chunk in async iterator
- Fix post-compaction question selection, system type, sync stream merge
- Skip tool_result-only user turns; bedrock: elif for context_management
- Add streaming iterator compaction test suite
Co-authored-by: Cursor <cursoragent@cursor.com>
* revert(html): restore flat *.html naming in _experimental/out
Reverses the accidental rename from *.html → */index.html introduced in
|
||
|
Yassin Kortam
|
3be3c1dea1
|
feat(otel): add team_metadata, http.route, and model names to inference spans (#29319)
Some checks are pending
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
|