Commit graph

9968 commits

Author SHA1 Message Date
Ishaan Jaff
1f412bc6d8
[Feat] Add Tool Policies for AI Gateway (#22732)
* fix: fix ui render

* fix: fix minor bugs

* refactor: use prisma functions instead of raw sql (safer)

* fix(add-new-tiles-to-tool-policies): allow developer to see what's available

* feat: ensure tool allowlist runs correctly for tool names + mcp's

* refactor: more ui improvements

* feat: working key tool blocking

* feat(tools): show tool logs

* refactor: backend code improvements

* refactor: improve log viewer for tools

* fix: address PR review feedback for tool access control

- Add missing blocked_tools column to root schema.prisma (schema drift)
- Invalidate ToolPolicyRegistry after policy mutations so changes take effect immediately
- Remove dead code: unused get_effective_policies, get_tool_policies_cached, and helpers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: race condition in permission resolution and remove duplicate allowlist check

- Use atomic update_many with object_permission_id=None to prevent concurrent
  requests from creating orphaned permission rows and losing tool blocks
- Remove duplicate allowed_tools enforcement from guardrail (already enforced
  in auth layer via check_tools_allowlist)
- Move inline uuid import to module level

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* update to account for  userAgent

* UI - Add ToolDetails

* input/output policy

* LiteLLM_PolicyAttachmentTable

* LiteLLM_PolicyAttachmentTable

* fix: add _enqueue_tool_registry_upsert

* fix: tool mgmt endpoints

* tool mgmt endpoints

* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix: sync root schema.prisma and fix test_tool_registry_writer for input/output policy

- Migrate root schema.prisma LiteLLM_ToolTable from call_policy to
  input_policy/output_policy, add missing user_agent and last_used_at columns
  (now consistent with litellm/proxy/schema.prisma and litellm-proxy-extras)
- Fix SpendLogToolIndex comment across all three schema files
- Fix all call_policy references in test_tool_registry_writer.py:
  swapped update_tool_policy arguments, wrong get_tools_by_names return type
  assertions, _mock_tool_row setting call_policy instead of input_policy

Addresses Greptile review feedback on PR #22732.

Made-with: Cursor

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-03-03 20:22:20 -08:00
Sameer Kankute
120201049e
Merge pull request #22666 from Point72/ephrimstanley/batch-fixes-mar3
Managed batches - Address PR bot comments from #22464
2026-03-04 09:06:41 +05:30
Sameer Kankute
7a83acf086
Merge pull request #22620 from OiPunk/codex/litellm-22619-deepinfra-function-calling
fix: add missing supports_function_calling for deepinfra models
2026-03-04 08:51:21 +05:30
Krish Dholakia
90eb6729d5
Agent Tracing - support context_id based trace id propogation + nested llm calls (#22626)
* style(ui/): distinguish agent calls from llm calls on ui

* feat: initial grouping working

* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls

* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call

allows stable trace id usage

* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict

_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: propagate trace/session id from headers in MCP server calls

Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 18:19:12 -08:00
Cesar Garcia
4ab79451d9
Merge pull request #22552 from shanemort1982/fix/register-model-custom-pricing-fields
fix: pass all custom pricing fields to register_model in completion() and embedding()
2026-03-03 22:40:45 -03:00
Ryan Crabbe
52ec73c07d fix(proxy): improve team expiry enforcement validation
- Change status codes from 400 to 500 for team metadata misconfig errors
  (callers can't fix admin-set config, 400 is misleading)
- Add anchor value validation to batch endpoint (matching files endpoint)
- Coerce seconds to int to handle string values from metadata
- Add error-path tests: missing keys, invalid anchor, status code assertions
- Add happy-path test: team injects expiry when caller sends nothing
2026-03-03 17:29:39 -08:00
yuneng-jiang
ab101671c7
Merge pull request #22725 from BerriAI/litellm_team_admin_reset_spend
[Fix] Team Admin Reset Key Spend
2026-03-03 16:50:46 -08:00
Cesar Garcia
a2f3beb26f
Update tests/test_litellm/llms/base_llm/test_base_model_iterator.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-03-03 21:47:55 -03:00
yuneng-jiang
0a1b2635d7 fix: allow team admins to access /key/{key}/reset_spend route
The route-level auth check was blocking internal_user role (team admins)
from reaching /key/{key}/reset_spend because KEY_RESET_SPEND was missing
from key_management_routes. Added it so team admins pass the route check
and the endpoint's existing _check_proxy_or_team_admin_for_key enforces
actual authorization.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 15:54:34 -08:00
milan-berri
98b9bc8b72
fix: resolve base_model in /cost/estimate for Azure custom deployments (#22724)
The _resolve_model_for_cost_lookup function was only checking
litellm_params.model when resolving model names from the router.
For Azure custom deployment names (e.g. azure/openai/gpt-5.3-codex),
this deployment name doesn't exist in the model cost map, so cost
returned /bin/zsh.

Now checks model_info.base_model and litellm_params.base_model first,
falling back to litellm_params.model only if no base_model is set.
This matches how the router resolves base_model everywhere else.
2026-03-03 15:43:02 -08:00
Julio Quinteros Pro
2415035fcd
Merge pull request #22719 from BerriAI/fix/vertex-response-format-test
fix: account for vertex_ai schema diff in response_format test
2026-03-03 19:59:24 -03:00
Julio Quinteros Pro
5dfa96f62f
Merge pull request #22718 from BerriAI/fix/jwt-test-mock-robustness
fix: use direct AsyncMock in JWT OIDC discovery tests
2026-03-03 19:59:15 -03:00
Julio Quinteros Pro
a07d041881 fix: apply same AsyncMock pattern to remaining OIDC discovery test
Address Greptile review: test_resolve_jwks_url_resolves_oidc_discovery_document
also used the inconsistent patch.object pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 19:56:51 -03:00
Julio Quinteros Pro
9b92ea16ab fix: update response_format test for vertex_ai's intentional schema diff
Vertex AI / Gemini uses Pydantic's model_json_schema() which omits
additionalProperties: False (Gemini rejects it). The test expected
the same schema for all providers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 19:55:18 -03:00
Julio Quinteros Pro
eb658693a3 fix: use direct AsyncMock assignment instead of patch.object in JWT tests
The patch.object with new_callable=AsyncMock can behave inconsistently
across Python versions, causing mock_response.status_code to return a
MagicMock instead of the assigned value. Direct assignment is simpler
and more reliable.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 19:51:09 -03:00
Julio Quinteros Pro
e6a1194c28
Merge pull request #22717 from BerriAI/fix/types-utils-video-tokens-test
fix: add video_tokens to completion_tokens_details test
2026-03-03 19:48:44 -03:00
Julio Quinteros Pro
2943df96ba fix: add video_tokens to expected completion_tokens_details in test
The CompletionTokensDetailsWrapper type now includes video_tokens field,
but this test's expected dict was not updated to include it.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 19:46:20 -03:00
Julio Quinteros Pro
fc9d06ceca
Merge pull request #22716 from BerriAI/fix/vertex-function-response-tests
fix: update vertex AI tests for function_response role=user
2026-03-03 19:44:54 -03:00
Julio Quinteros Pro
5a0aba9fb7 fix: update vertex AI tests to expect role=user on function_response messages
The Gemini API requires role="user" on function_response content blocks
(added in commit 273cf12afa), but these tests were never updated to match.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 19:40:28 -03:00
Julio Quinteros Pro
2c5c38333d
Merge pull request #22701 from BerriAI/fix/streaming-and-azure-gpt5-test-failures
Fix Anthropic streaming sync and Azure GPT-5.1 logprobs tests
2026-03-03 19:36:44 -03:00
ryan-crabbe
8aa7d53864
Merge pull request #22705 from BerriAI/litellm_feat_batch_expiry_setting_teams
Litellm feat batch expiry setting teams
2026-03-03 14:36:12 -08:00
Julio Quinteros Pro
6b4bc99202 Fix Anthropic streaming sync __next__ and Azure GPT-5.1 logprobs
Two independent fixes for pre-existing test failures on main:

1. Anthropic streaming: The sync __next__ method used a simple
   holding_chunk pattern that lost chunks when multiple events needed
   to be returned. Refactored to use the same chunk_queue approach as
   the async __anext__ method. Also fixed tests that used ModelResponse
   (which defaults finish_reason to 'stop') instead of ModelResponseStream.

2. Azure GPT-5.1 logprobs: The base OpenAI class includes logprobs for
   gpt-5.1+ models, but Azure hasn't verified support for gpt-5.1.
   Added explicit removal of logprobs/top_logprobs for gpt-5.1 (non-5.2)
   models in the Azure config.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 18:16:11 -03:00
Ryan Crabbe
08613b24cb feat(proxy): add team-level file expiry enforcement 2026-03-03 11:03:14 -08:00
Ryan Crabbe
3d15bcdb11 feat(proxy): add team-level batch output expiry enforcement 2026-03-03 10:58:31 -08:00
yuneng-jiang
4c1b15d685
Merge pull request #22687 from BerriAI/litellm_fix_spend_log_cleanup
[Fix] Spend Log Cleanup: lock tracking, integer retention, skip log level
2026-03-03 10:48:20 -08:00
Ryan Crabbe
43cec8c980 feat(batches): support output_expires_after passthrough 2026-03-03 10:46:45 -08:00
Cesar Garcia
8a1e915bb6
Merge pull request #22403 from shivaaang/fix/openrouter-image-edit
feat(openrouter): add image edit support for OpenRouter models
2026-03-03 15:31:04 -03:00
Cesar Garcia
a8b5a876bf
Merge pull request #21491 from Chesars/fix/20998-remove-hardcoded-reasoning-summary
fix(anthropic): remove hardcoded reasoning summary in adapter
2026-03-03 15:29:11 -03:00
Chesars
833c1bc45d merge: resolve conflict with staging, remove hardcoded summary from reasoning test 2026-03-03 15:28:45 -03:00
Cesar Garcia
b2c7d2e049
Merge pull request #21577 from Chesars/fix/gemini-streaming-tool-calls-finish-reason
fix(gemini): correct streaming finish_reason for tool calls
2026-03-03 15:25:24 -03:00
Cesar Garcia
da941e4261
Merge pull request #22589 from Chesars/fix/vertex-preserve-any-type-schema
fix(vertex): preserve type schema semantics for JsonValuefields
2026-03-03 15:19:16 -03:00
yuneng-jiang
224c617119 Fix spend log cleanup: lock tracking, integer retention, skip log level
- Only release distributed lock in finally if it was actually acquired;
  prevents spurious Redis release_lock calls on early returns
- Treat bare integer maximum_spend_logs_retention_period as days (e.g. 3 → "3d")
  instead of silently failing with a ValueError
- Elevate "Skipping cleanup" log from info to error so misconfigured
  retention settings are visible without verbose logging
- Add tests for all three fixes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 10:12:08 -08:00
yuneng-jiang
76e3dba0f8 fix mcp server created_at and updated_at timestamps being overwritten with current time
- Add created_at field to MCPServer type (was missing)
- Map created_at from LiteLLM_MCPServerTable in build_mcp_server_from_table()
- Use server.created_at and server.updated_at instead of datetime.now() in _build_mcp_server_table() and health check table builder
- Add regression tests to verify timestamps are preserved through round-trip conversions

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-03 09:41:45 -08:00
Harshit28j
b44755db96 fix(proxy): make common_checks opt-in for custom auth via custom_auth_run_common_checks
Replaces the skip_route_check approach from PR #22662 with a configurable
opt-in flag. By default, common_checks() is not run for custom auth flows,
preserving backwards compatibility with pre-#22164 behavior.

Users who want budget/team/route enforcement on custom auth can enable it:
  general_settings:
    custom_auth_run_common_checks: true

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 22:50:34 +05:30
Sameer Kankute
daa0397f30
Merge pull request #22674 from BerriAI/litellm_gemini-3.1-flash-lite-preview
Day 0 gemini 3.1 flash lite preview support
2026-03-03 22:23:17 +05:30
Ephrim Stanley
b83373d29c Managed batches - Address PR bot comments from #22464 2026-03-03 11:01:49 -05:00
Harshit28j
6d535e5639 fix(proxy): allow custom auth routes to bypass route authorization checks
Custom user-added routes (e.g. /ldap/ngs/ready) used with Depends(user_api_key_auth) were being rejected as admin-only after _run_post_custom_auth_checks was introduced in commit 14badde13c.

The route authorization check in common_checks is designed for LiteLLM's own management routes. Custom auth flows that add their own routes should be trusted since the custom auth function already validated the request. Budget and expiry checks still run.

Add skip_route_check parameter to common_checks() and pass skip_route_check=True from _run_post_custom_auth_checks() to skip route authorization while preserving budget/team/model checks.

Regression test added: test_common_checks_skip_route_check_for_custom_auth

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
2026-03-03 20:20:47 +05:30
Sameer Kankute
c1b39a6425
Merge pull request #22651 from BerriAI/litellm_encrypted_content_affinity_2
Add support for encrypted content affinity
2026-03-03 19:43:47 +05:30
Sameer Kankute
a9ee8ff5d2
Merge pull request #22641 from BerriAI/litellm_remove_default_litellm_thinking
[Chore]Remove defualt hardcoded thinking levels for gemini 3 family
2026-03-03 19:42:40 +05:30
Sameer Kankute
9ffbd9e30e
Merge pull request #22464 from Point72/ephrimstanley/batch-fixes-feb27
Managed batches fixes for vertex
2026-03-03 18:53:53 +05:30
Sameer Kankute
521f804350 Fix encrypted content streaming affinity issue 2026-03-03 18:37:22 +05:30
Sameer Kankute
2bc4da76ce Update the tests 2026-03-03 18:36:07 +05:30
Sameer Kankute
18bf3f2df6 Fix mock github test 2026-03-03 18:36:07 +05:30
Sameer Kankute
a88a17796b Fix logging and encrypted content extraction 2026-03-03 18:36:07 +05:30
Sameer Kankute
9f627c67d8 Add tests for encrypted_content_affinity 2026-03-03 18:36:07 +05:30
Sameer Kankute
394c49d303 Add tests for encrypted_content_affinity 2026-03-03 18:36:07 +05:30
David Steele
565a52780b
Merge remote-tracking branch 'upstream/main' into pr-22553 2026-03-03 09:48:39 +00:00
Sameer Kankute
851be58775 Add day 0 support of gemini-3.1-flash-lite-preview 2026-03-03 15:07:47 +05:30
Sameer Kankute
213423cb45 Fix test case 2026-03-03 15:05:20 +05:30
Krish Dholakia
67f90254ed
feat(guardrails): team-based guardrail registration and approval workflow (#22459)
* feat(guardrails): team-based guardrail registration and approval workflow

Add team-based guardrail submission system where teams can register
Generic Guardrail API guardrails for admin review. Includes:

- POST /guardrails/register endpoint for team-scoped submissions
- Admin review endpoints (list/get/approve/reject submissions)
- Team Guardrails tab in the UI dashboard
- extra_headers support for forwarding client headers to guardrail APIs
- Prisma schema migration for status, submitted_at, reviewed_at fields
- Documentation for team-based guardrails and static/dynamic headers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(guardrails): address review feedback - SSRF, silent failure, redundant query

- Validate api_base URL scheme (http/https only) and hostname in
  register_guardrail to prevent SSRF via team submissions
- Return warning field in approve response when in-memory initialization
  fails so admins know the guardrail won't work until next sync cycle
- Eliminate redundant DB query in list_guardrail_submissions by fetching
  all team guardrails once and deriving both filtered list and summary
  counts from the single result set

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(guardrails): add pending_review status guard to reject endpoint

Prevent rejecting already-active or already-rejected guardrails, which
would create a DB/memory inconsistency (active in memory but rejected
in DB). Now mirrors the approve endpoint's status check.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 22:06:49 -08:00