Commit graph

5537 commits

Author SHA1 Message Date
mateo-berri
04410967ff Merge remote-tracking branch 'origin/main' into litellm_anthropic_guardrail_system_and_tool_use
main's test_one_text_per_row_over_a_system_prompt_is_rejected_by_name assumed the
top-level system prompt stays out of the scanned texts. This branch scans it, so one
text per structured row now lines up and the rewrite is applied; the test asserts that,
and a multi-block system prompt keeps the length-guard rejection covered.
2026-09-14 22:19:18 -07:00
Mateo Wang
c16c172a65
Merge pull request #40939 from BerriAI/litellm_responses_per_message_guardrail_rewrite
fix(guardrails): write per-message guardrail rewrites back onto Responses input items
2026-09-14 21:57:29 -07:00
mateo-berri
e01d97ea08 fix(guardrails): read Prompt Security modified rows with the slot count's own predicate
A chat row whose content carried an empty text part counted two slots in the
chat completions handler while Prompt Security read one text out of the
modified row, so the structured rewrite was dropped and the request got the
named rejection. One shared helper now lists a row's slot texts and both the
slot count and the modified-row reader use it.
2026-09-14 21:31:22 -07:00
Mateo Wang
2bdafa1206
Merge branch 'main' into fix/headroom-protect-cache-control-rows 2026-09-14 20:53:07 -07:00
mateo-berri
8b80971586 Merge remote-tracking branch 'origin/main' into litellm_anthropic_guardrail_system_and_tool_use 2026-09-14 19:27:58 -07:00
mateo-berri
16c326537f fix(guardrails): define UnappliableRequestRewrite in the shared guardrail translation utils
The three guardrail translation handlers imported the exception from the proxy policy engine through a function-local import, which CodeQL flagged as a cyclic import. The exception and its helper now live next to the handlers in the shared guardrail translation utils, and the tests import it from there.

The Prompt Security modify-mode helper is also restructured into early-return TypedDict displays so the LIT002 budget stays at its limit
2026-09-14 18:39:54 -07:00
Mateo Wang
1ba97665b2
Merge pull request #41046 from BerriAI/devin/1789368982-cli-strenum-py310
fix(cli): drop enum.StrEnum so the CLI imports on Python 3.10
2026-09-14 18:32:30 -07:00
mateo-berri
6574d83eae Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts:
#	litellm/proxy/guardrails/guardrail_hooks/prompt_security/prompt_security.py
2026-09-14 18:19:38 -07:00
Mateo Wang
dad6e9be4a
Merge branch 'main' into fix/headroom-protect-cache-control-rows 2026-09-14 17:58:17 -07:00
yucheng-berri
8481bc27f9
Merge pull request #41132 from BerriAI/litellm_bedrock_grounding_from_plain_messages
fix(bedrock guardrails): derive contextual grounding source and query from plain messages
2026-09-14 17:56:58 -07:00
yucheng-berri
b67137b67f
Merge pull request #40702 from BerriAI/litellm_model_armor_logging_only_streaming
feat(model_armor): logging_only mode scans completed streams after delivery
2026-09-14 17:23:32 -07:00
Yassin Kortam
d1fc231545
Merge pull request #41145 from BerriAI/litellm_health_check_skip_save_on_failed_read_main 2026-09-14 16:48:47 -07:00
yucheng-berri
99245f9323
Merge pull request #41131 from BerriAI/litellm_prompt_security_created_status
fix(prompt_security): keep polling file sanitization through non-terminal statuses
2026-09-14 16:43:37 -07:00
tin-berri
c626ff098b
Merge pull request #40877 from BerriAI/litellm_lit7658_cache_cost_v0_fresh
feat(proxy): predict prompt-cache costs across deployments
2026-09-14 16:25:41 -07:00
yassin
c6a36fb8b6 fix(health): skip background health check DB writes when the latest-row read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 6d2c4899b0)
2026-09-14 23:03:30 +00:00
yassin
ece2bbfed7 fix(proxy): resolve x-litellm-call-id from response metadata when routes omit call_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 23:03:27 +00:00
yucheng
d8d7210b57 test(bedrock guardrails): cover tagged messages on the apply_guardrail response path with the flag off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:28:58 +00:00
yucheng
2e11f7bc7b test(bedrock guardrails): shorten grounding test docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:09:59 +00:00
ryan-crabbe-berri
7b6d223105
Merge pull request #40196 from BerriAI/litellm_team_admin_all_proxy_models
fix(ui): let team admins grant a team all proxy models
2026-09-14 15:05:01 -07:00
yucheng
442af3aab6 fix(bedrock guardrails): gate plain-message grounding behind contextual_grounding_from_messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:04:42 +00:00
yucheng
3a3075b8a2 fix(prompt_security): poll only on queued statuses, keep 500 for terminal or missing status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:01:39 +00:00
Yassin Kortam
3e417fa6e0
Merge pull request #41113 from BerriAI/litellm_realtime_release_max_parallel_slot
fix(proxy): release max_parallel_requests slot when a realtime session ends without LLM callbacks
2026-09-14 14:56:11 -07:00
Yassin Kortam
24bfd5fba1
Merge pull request #41102 from BerriAI/litellm_team_membership_once_main
fix(auth): load team membership once per request and skip prisma on an L1 hit
2026-09-14 14:46:06 -07:00
yassin
b4d0f4ad26 refactor(realtime): move session ownership marker keys into constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:33:48 +00:00
yucheng
26e2e208ad fix(bedrock guardrails): derive contextual grounding source and query from plain messages
Bedrock only runs a contextualGroundingPolicy when the ApplyGuardrail payload
carries grounding_source and query qualifiers. Callers sending ordinary system
and user messages never got those, so a configured grounding threshold was
silently skipped on /v1/chat/completions and /guardrails/apply_guardrail.

When no explicit grounding_source or query tags are present, system and
developer text is sent as grounding_source and the latest user message as
query. The apply_guardrail response branch now forwards the request messages,
which it previously dropped.

Resolves LIT-4224

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:18:16 +00:00
yassin
db8dfe93a5 fix(auth): wait for the in-flight membership load before evicting its cache key on invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:10:43 +00:00
yassin
2441e8a2a9 Merge remote-tracking branch 'origin/main' into litellm_realtime_release_max_parallel_slot 2026-09-14 21:03:28 +00:00
yucheng
fb60f80c80 fix(prompt_security): keep polling file sanitization through non-terminal statuses
Prompt Security reports a queued sanitization job as status "created" before
it moves to "in progress" and "done". The poller treated anything other than
those two known strings as an error and returned HTTP 500 on the first poll, so
every image or file request through the guardrail failed while the vendor job was
still queued.

Only "done" is terminal now. Every other status is logged and polled again until
max_poll_attempts or the outer file_sanitization_timeout, after which the existing
fail-open or fail-closed (408) policy applies.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:03:02 +00:00
Yassin Kortam
4123b4bc2b
Merge pull request #41086 from BerriAI/litellm_team_list_org_admin_membership_lit3723
fix(proxy): keep org admins' own team memberships in other orgs visible on team list
2026-09-14 13:57:08 -07:00
yassin
ffeea30f23 test(auth): cover a stale membership write landing after a fresh reload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:55:22 +00:00
tin-berri
2cad7a49af
Merge pull request #41116 from BerriAI/litellm_auto_router_session_api_access
fix(proxy): allow LLM API keys to read auto-router sessions
2026-09-14 13:49:03 -07:00
yassin
abc85ba607 fix(proxy): leave the realtime max_parallel_requests slot to the limiter failure callback when a refusal was logged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:47:37 +00:00
yassin
91c964a338 fix(auth): evict the membership cache entry when invalidation lands during the cache write
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:46:09 +00:00
Yassin Kortam
161ae2f3b4
Merge pull request #41107 from BerriAI/litellm_hide_credentials_hint_when_ui_password_set
fix(proxy): hide default credentials login hint when UI_PASSWORD is set
2026-09-14 13:44:09 -07:00
Tin Chi Lo
3177c37e22 fix(cli): show routed models and session stats for LLM API keys 2026-09-14 13:36:00 -07:00
Yassin Kortam
856aedd14a
Merge pull request #40817 from BerriAI/litellm_redis_timeout_log_throttle
fix(redis): log a timeout streak once per interval instead of one line per cache call
2026-09-14 13:34:45 -07:00
yassin
1b31be1a9c fix(proxy): leave the realtime max_parallel slot to the success callback when one is enqueued
Releasing the slot unconditionally from the route raced the limiter's own
success handler on the logging worker: both could read the same stashed
acquisition before either cleared it, and under the integer in-memory
fallback that double-decrements the counter. The route now releases only on
exits without a success callback (pre-call rejection, pre-call cancellation,
and Phase 2 exits without the success stamp), matching the HTTP disconnect
path's ownership rule.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:23:23 +00:00
yassin
4ac168b3fc fix(auth): drop in-flight membership load on invalidation so it cannot repopulate the cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:13:39 +00:00
ryan
2d3b63fb5a fix(proxy): report parent org models on /team/info only to team managers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:01:47 +00:00
yassin
81ffc3125f fix(proxy): release realtime max_parallel_requests slot when the task is cancelled during pre-call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:50:01 +00:00
yassin
39f6ac4788 perf(proxy): serialize /model/info listing once with orjson
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:47:51 +00:00
Yassin Kortam
f67cc62032
Merge pull request #40995 from BerriAI/litellm_all_deployments_cooldown_error
fix(router): name the all-deployments-in-cooldown error on 429 responses
2026-09-14 12:39:42 -07:00
ryan
9cdfe311e9 test: assert only the org ceiling /team/info returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:39:18 +00:00
ryan
5ddb0fe691 fix(ui): keep the team model select loading until /team/info reports the org ceiling
useTeam seeds its cache from the team list, which has no organization_models, so the select briefly rendered unfiltered. Also drop the Prisma include assertion from the backend test.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:38:05 +00:00
Yassin Kortam
7c55c4ba7d
Merge pull request #40925 from BerriAI/litellm_proxy_error_headers_from_litellm_response_headers
fix(proxy): forward provider request id headers on mapped error responses
2026-09-14 12:36:59 -07:00
ryan-crabbe-berri
3f42a4fc98 test: suppress TQ008 on the two module-state patches
team_info resolves prisma_client and get_all_team_memberships from module
state, so there is no collaborator to inject; the two sibling team_info tests
patch the same way.

Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
2026-09-14 19:30:44 +00:00
ryan-crabbe-berri
b6a5563d05 fix(ui): let team admins grant a team all proxy models
The team edit form only offered "All Proxy Models" when the dashboard could
read the parent organization, and /organization/info 403s for anyone who is
not a proxy admin or an admin of that org. A team admin with the internal_user
proxy role therefore saw only "No Default Models", which is the opposite of
what they wanted, and had no way to grant their team everything on the proxy.

/team/info now reports the parent org's model ceiling as organization_models,
which the same authorization already admits, and ModelSelect reads the ceiling
from the team it is editing before falling back to the organization. That also
makes the individual model list respect the org's allow-list instead of
listing every proxy model to a caller whose save would be rejected.

useTeam was typed as Team while returning the /team/info envelope, so its one
other caller unwrapped it behind a cast. It now returns the team itself.

Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
2026-09-14 19:30:44 +00:00
yassin
f9d0cd6409 chore(proxy): drop explanatory docstrings from credentials hint helper and tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:29:30 +00:00
yassin
ebcd9bcb18 fix(proxy): release max_parallel_requests slot when a realtime session ends without LLM callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:29:11 +00:00
yassin
ff02390685 test(auth): drop docstrings from team membership tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:26:14 +00:00