Commit graph

50471 commits

Author SHA1 Message Date
yucheng
b37ce94075 refactor(guardrails): rename scoped-out evaluation status from not_run to skipped
The per-guardrail status a scoped-out evaluation records is now skipped, matching the
skip_*_in_guardrail settings that cause it. Request-level rollup still maps it to not_run
so the StandardLoggingPayload status contract is unchanged

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:15:53 +00:00
yucheng
6b9990e068 fix(agent_365): treat Entra AADSTS50027xx invalid_client as a caller assertion failure
Entra reports a forged or garbled On-Behalf-Of assertion as invalid_client, the
same OAuth error it uses for a bad gateway client secret. Read the AADSTS
error_codes so a malformed assertion sends the caller back to sign in at connect
and at the tool call instead of reporting the gateway's credentials as broken.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:12:12 +00:00
yucheng
2e11f7bc7b test(bedrock guardrails): shorten grounding test docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:09:59 +00:00
yassin
df5e193645 Merge remote-tracking branch 'origin/main' into pr-40997
Some checks failed
ai-gateway image / ai-gateway release image (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

# Conflicts:
#	ui/litellm-dashboard/src/components/templates/key_edit_view.tsx
2026-09-14 22:07:48 +00:00
Devin AI
6f9915b650 fix(utils): keep litellm params out of provider request bodies
get_litellm_params() keys such as model_alias_map, completion_call_id and
custom_prompt_dict were missing from all_litellm_params, so when a bridge
forwarded litellm_params as completion kwargs they were swept into extra_body
and OpenAI rejected the call with "Unknown parameter: 'model_alias_map'".

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:07:27 +00:00
ryan-crabbe-berri
7fd541efb9
Merge pull request #41048 from HUAHAODIA/litellm_ratchet_strict_rules
chore(lint): graduate 12 rules from the strict-gate ratchet
2026-09-14 15:06:16 -07:00
ryan-crabbe-berri
7b6d223105
Merge pull request #40196 from BerriAI/litellm_team_admin_all_proxy_models
fix(ui): let team admins grant a team all proxy models
2026-09-14 15:05:01 -07:00
yucheng
442af3aab6 fix(bedrock guardrails): gate plain-message grounding behind contextual_grounding_from_messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:04:42 +00:00
yucheng
0e27a523e0 fix(mcp): probe Entra at connect, challenge only default-on Agent 365 guardrails, key listed tools by signer
Connect-time sign-in challenge now asks Entra to exchange the presented assertion instead of only checking
its compact-JWS shape, so expired, wrong-audience or forged bearers get the RFC 9728 challenge while gateway
credential and provider failures still surface on the tool call. Only default_on guardrails the caller has not
opted out of advertise or challenge, since anonymous metadata cannot see key-selected guardrails. Servers whose
Authorization is minted per caller by MCPJWTSigner list tools per caller instead of sharing one cache slot

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:04:02 +00:00
ryan-crabbe-berri
51926c2e79
Merge pull request #41023 from BerriAI/litellm_ui_move_metadata_tags_to_tags_field
fix(ui): move tags typed into key metadata JSON into the Tags field
2026-09-14 15:03:54 -07:00
yucheng
3a3075b8a2 fix(prompt_security): poll only on queued statuses, keep 500 for terminal or missing status
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:01:39 +00:00
berriai-litellm-provider-info-sync[bot]
c3f8c07c43
chore(prices): sync Vertex AI prices: 14 models
vertex_ai/deep-research-pro-preview-12-2025: cache_read_input_token_cost
vertex_ai/gemini-2.5-flash-image: input_cost_per_token_flex, output_cost_per_token_flex, input_cost_per_token_batches, input_cost_per_token_priority, output_cost_per_token_batches
vertex_ai/gemini-3-flash-preview: input_cost_per_token_flex, output_cost_per_token_flex, input_cost_per_token_batches, output_cost_per_token_batches, cache_read_input_token_cost_flex, cache_read_input_audio_token_cost
vertex_ai/gemini-3-pro-image: input_cost_per_token_flex, output_cost_per_token_flex, cache_read_input_token_cost, input_cost_per_token_priority, output_cost_per_token_priority, cache_read_input_token_cost_flex, cache_read_input_token_cost_priority, input_cost_per_token_above_200k_tokens, output_cost_per_token_above_200k_tokens, cache_read_input_token_cost_above_200k_tokens, input_cost_per_token_above_200k_tokens_priority, output_cost_per_token_above_200k_tokens_priority, cache_read_input_token_cost_above_200k_tokens_priority
vertex_ai/gemini-3.1-flash-image: input_cost_per_token_flex, output_cost_per_token_flex, cache_read_input_token_cost, input_cost_per_token_batches, output_cost_per_token_batches, cache_read_input_token_cost_flex
vertex_ai/gemini-3.1-flash-lite: cache_read_input_audio_token_cost
vertex_ai/gemini-3.1-flash-lite-image: input_cost_per_token_flex, output_cost_per_token_flex, cache_read_input_token_cost_flex
vertex_ai/gemini-3.1-pro-preview: input_cost_per_token_flex, output_cost_per_token_flex, cache_read_input_token_cost_flex
vertex_ai/gemini-3.5-flash: 
vertex_ai/gemini-3.5-flash-lite: cache_read_input_token_cost_priority
vertex_ai/gemini-3.6-flash: 
vertex_ai/gemini-3.7-flash: 
vertex_ai/gemini-3.8-flash: 
vertex_ai/gemini-embedding-2:
2026-09-14 22:00:57 +00:00
ryan
ef71222349 Merge remote-tracking branch 'origin/main' into litellm_bulk_user_delete 2026-09-14 22:00:10 +00:00
Yassin Kortam
e766277846
Merge pull request #40991 from BerriAI/litellm_team_model_cooldown_siblings
fix(router): cool down team deployments on 429 when a sibling serves the same public model
2026-09-14 14:57:07 -07:00
Yassin Kortam
3e417fa6e0
Merge pull request #41113 from BerriAI/litellm_realtime_release_max_parallel_slot
fix(proxy): release max_parallel_requests slot when a realtime session ends without LLM callbacks
2026-09-14 14:56:11 -07:00
ryan-crabbe-berri
f9ba42860c fix(azure_ai): keep api_version as a property alongside the new getter
The PLR0206 fix renamed the property to get_api_version and dropped it.
AzureFoundryModelInfo().api_version does resolve today, so removing it
breaks external callers. Restore it as a zero-arg property delegating to
the staticmethod, which satisfies PLR0206 either way.
2026-09-14 14:56:07 -07:00
ryan
52a3b0ede7 fix(ui): trim and drop blank tags moved out of key metadata JSON
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:50:25 +00:00
yujonglee
9bbb869b13
Merge pull request #41120 from BerriAI/litellm_trace_profiler_threads
refactor(harness): expand independent trace coverage
2026-09-14 14:50:18 -07:00
Devin AI
e4ebeae800 fix(prices): add text output rate to Vertex TTS entries, sync gemini-embedding-2 alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:49:15 +00:00
Yassin Kortam
24bfd5fba1
Merge pull request #41102 from BerriAI/litellm_team_membership_once_main
fix(auth): load team membership once per request and skip prisma on an L1 hit
2026-09-14 14:46:06 -07:00
yassin
fbc1011d27 fix(bedrock): end the realtime session when the client disconnects instead of waiting for Nova Sonic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:42:42 +00:00
Devin AI
f45e20e6c2 fix(prices): mark Vertex Gemini TTS entries as audio_speech
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:41:43 +00:00
kerry
abbf5aae20 chore(prices): flag gpt-5.5-cyber as a reasoning model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:41:10 +00:00
yassin
f8e26deb54 fix(proxy): bind JWT agent lookup at startup regardless of agent source
Some checks failed
ai-gateway image / ai-gateway release image (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Move jwt_handler.bind_agent_lookup out of the YAML and DB agent loading paths and into
ProxyStartupEvent._initialize_jwt_auth so agents created via the API or UI after startup,
with no agents in config and no DB agent reload, still resolve for agent_id_jwt_field

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:38:00 +00:00
yucheng
87bf5730da fix(guardrails): carry MCP caller identity in synthetic metadata instead of trusting top-level fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:37:44 +00:00
ryan
df87a49f8b fix(ui): offer no models when an org team's ceiling never arrives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:35:55 +00:00
yassin
b4d0f4ad26 refactor(realtime): move session ownership marker keys into constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:33:48 +00:00
yucheng
5355cbe235 fix(mcp): keep Agent 365 evaluation faithful to each caller's tool call
Key the listed-tool cache by every request input that changes the upstream catalog (caller identity, per-server auth header, forwarded headers, header-driven stdio env, relayed bearer) so one caller's tool metadata never reaches Agent 365 for another. Advertise Entra in anonymous protected-resource discovery only for default_on guardrails. Check the key's server grant before issuing the Agent 365 sign-in challenge so a key without access gets the grant's 403. Group conversationId by the MCP session id, falling back to the call id. Run Agent 365 in the parallel block-only phase so it judges the arguments the sequential guardrails hand upstream. Apply tool_name_to_description to local-registry tool metadata sent to Agent 365

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:28:18 +00:00
Devin AI
6738246586 chore(models): credit absorbed Inception Mercury 2.5 cache pricing from #41016
Co-Authored-By: Nanduu24 <reachnanduu24@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:26:40 +00:00
berriai-litellm-provider-info-sync[bot]
93e65995d6
chore(prices): sync Vertex AI prices: 4 models
gemini-2.5-flash: cache_read_input_audio_token_cost
gemini-2.5-flash-lite: cache_read_input_audio_token_cost
gemini-3-flash-preview: cache_read_input_audio_token_cost
gemini-3.1-flash-lite: cache_read_input_audio_token_cost
2026-09-14 21:26:02 +00:00
kerry
ddf13e9505 Merge remote-tracking branch 'origin/main' into litellm-providers/price-sync 2026-09-14 21:21:55 +00:00
yucheng
90ba447974 fix(guardrails): keep top-level caller identity for MCP pre-call custom code guardrails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:20:46 +00:00
kerry-berri
c6e4c5582d
Merge pull request #41093 from BerriAI/litellm_fallback_backfill_opt_in_main
feat(model_info): provider-scoped fill_missing_for_providers backfill from fallback generalization rules
2026-09-14 14:20:36 -07:00
yucheng
26e2e208ad fix(bedrock guardrails): derive contextual grounding source and query from plain messages
Bedrock only runs a contextualGroundingPolicy when the ApplyGuardrail payload
carries grounding_source and query qualifiers. Callers sending ordinary system
and user messages never got those, so a configured grounding threshold was
silently skipped on /v1/chat/completions and /guardrails/apply_guardrail.

When no explicit grounding_source or query tags are present, system and
developer text is sent as grounding_source and the latest user message as
query. The apply_guardrail response branch now forwards the request messages,
which it previously dropped.

Resolves LIT-4224

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:18:16 +00:00
yucheng
33fb6625ad test(guardrails): cover nameless evaluated entries in the malformed usage batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:18:00 +00:00
yassin
d2342f06ce fix(bedrock): stamp the realtime success ownership marker when Nova Sonic spend is logged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:13:29 +00:00
yassin
db8dfe93a5 fix(auth): wait for the in-flight membership load before evicting its cache key on invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:10:43 +00:00
yassin
4d4d3fb18a fix(proxy): bind agent registry into JWTHandler and keep persisted agent id on AUTO_REGISTER race
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:09:14 +00:00
yassin
2441e8a2a9 Merge remote-tracking branch 'origin/main' into litellm_realtime_release_max_parallel_slot 2026-09-14 21:03:28 +00:00
yucheng
b2a946562f fix(guardrails): resolve caller identity from metadata buckets in custom code guardrail
The sandbox read user_api_key_* off the top level of request_data, where the proxy never writes them, and only forwarded request_data["metadata"], which is empty on /v1/messages, /v1/responses, batches and files because those routes keep proxy state in litellm_metadata. Merge both buckets (litellm_metadata wins) and resolve ids from the merged dict

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:03:17 +00:00
yucheng
fb60f80c80 fix(prompt_security): keep polling file sanitization through non-terminal statuses
Prompt Security reports a queued sanitization job as status "created" before
it moves to "in progress" and "done". The poller treated anything other than
those two known strings as an error and returned HTTP 500 on the first poll, so
every image or file request through the guardrail failed while the vendor job was
still queued.

Only "done" is terminal now. Every other status is logged and polled again until
max_poll_attempts or the outer file_sanitization_timeout, after which the existing
fail-open or fail-closed (408) policy applies.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:03:02 +00:00
Yujong Lee
da839d4a11 fix(harness): skip unavailable Rust traces 2026-09-14 14:01:28 -07:00
Yassin Kortam
4123b4bc2b
Merge pull request #41086 from BerriAI/litellm_team_list_org_admin_membership_lit3723
fix(proxy): keep org admins' own team memberships in other orgs visible on team list
2026-09-14 13:57:08 -07:00
Yassin Kortam
ee03bad8c6
Merge pull request #41064 from BerriAI/litellm_bedrock_realtime_propagate_provider_failures
fix(bedrock/realtime): propagate deferred Nova Sonic stream failures to the router
2026-09-14 13:56:26 -07:00
jesus
f9298d8979 style(tests): drop unrelated formatting churn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:56:10 +00:00
yassin
ffeea30f23 test(auth): cover a stale membership write landing after a fresh reload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:55:22 +00:00
jesus
efad8deb71 fix(alerting): send llm_exceptions Slack alert for 5xx HTTPException/ProxyException
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:55:02 +00:00
yucheng
937179bde1 fix(guardrails): never map empty guardrail names onto a sibling id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:53:11 +00:00
yucheng
586d51f15e fix(guardrails): skip malformed guardrail entries instead of failing the usage batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 20:50:41 +00:00
tin-berri
2cad7a49af
Merge pull request #41116 from BerriAI/litellm_auto_router_session_api_access
fix(proxy): allow LLM API keys to read auto-router sessions
2026-09-14 13:49:03 -07:00