Add new /public/endpoints endpoint that returns which providers support each LiteLLM
endpoint (e.g., chat_completions, embeddings). The endpoint reads from a local backup
JSON file bundled with the package, caches the result in-process, and transforms the
raw provider-centric data into an endpoint-centric response format.
Changes:
- Add litellm/provider_endpoints_support_backup.json (copy of root source file)
- Add Pydantic response models (EndpointProvider, SupportedEndpoint, SupportedEndpointsResponse)
- Add /public/endpoints route with transformation and caching logic
- Add 16 comprehensive tests covering HTTP layer and transformation functions
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Return all supported endpoints and which providers support them. Includes endpoint display names, URL paths, and per-provider support lists. Results are cached for the process lifetime.
Also adds comprehensive test coverage for the new endpoint.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Phase 2 (per-worker mark_process_dead on shutdown) only ever fired
when all workers shut down together, making it redundant — Phase 1
wipes everything on next startup anyway. This aligns with the
prometheus_client docs: just wipe the directory between runs.
Adds cleanup utilities for PROMETHEUS_MULTIPROC_DIR to prevent
unbounded RAM/disk growth from stale .db files in multi-worker setups.
Three-part lifecycle aligned with upstream prometheus_client docs:
1. Startup: wipe entire directory before workers fork (clean slate)
2. Shutdown: mark_process_dead() for own PID (removes gauge_live* only)
3. Periodic (hourly): scan for dead PIDs and call mark_process_dead()
Counter/histogram files are never individually deleted at runtime to
avoid partial counter resets that cause false spikes in rate()/increase().
Also auto-creates PROMETHEUS_MULTIPROC_DIR when prometheus callback is
configured with multiple workers and the env var is not already set.
Replace Prisma ORM count/find_many calls with two query_raw calls that
only project the key_alias column. The Prisma client wrapper does not
support SELECT projection via find_many, so raw SQL is used to keep
memory usage proportional to the page size rather than total key count.
Update tests to mock query_raw instead of count/find_many.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
LiteLLM_VerificationTokenActions.find_many() does not support the
select keyword argument. Remove it and drop the corresponding test.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Add select={"key_alias": True} to the find_many call so only the alias
column is fetched from the database instead of full token rows. Add
five unit tests in test_key_management_endpoints.py covering response
shape, pagination skip/take computation, search filter injection,
absence of contains filter when no search term is given, and the
select-only-alias optimization.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The tests were mocking `filter_server_ids_by_ip` but the production
code in server.py now calls `filter_server_ids_by_ip_with_info` which
returns a (server_ids, blocked_count) tuple. Update all 8 mock sites
to use the correct method name and return signature.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(proxy): improve auth exception logging levels and add structured context
Downgrade expected auth failures (ProxyException, HTTPException < 500,
BudgetExceededError) from ERROR to WARNING log level to reduce noise from
routine rejected requests (e.g. missing/invalid API keys on polled endpoints
like /schedule/model_cost_map_reload/status).
Unexpected exceptions and HTTPException with status >= 500 still log at
ERROR with full traceback.
Enrich log messages with structured context: route, HTTP method, masked
API key (using existing abbreviate_api_key), error type, and error code.
All fields also passed via log extra dict for log aggregation tools.
Fixes#21293
* Update tests/test_litellm/proxy/auth/test_auth_exception_handler.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Adds a new block_code_execution guardrail that detects markdown fenced code blocks
in request/response content and blocks or masks them by language. Includes full
UI integration, type definitions, compliance test dataset, and 26 unit tests.
Key guardrail capabilities:
- Regex-based fenced code block detection with configurable blocked languages
- Confidence scoring with tunable threshold
- Execution-intent heuristics (request-side only) with conflict resolution
- Block or mask actions for detected code
- Support for pre_call, post_call, and during_call event hooks
Security hardening:
- Response-side blocking skips intent heuristics (LLM output doesn't contain
user intent phrases, so checking would silently disable post_call blocking)
- No-execution short-circuit includes conflict resolution: if both no-execution
and execution phrases match, execution intent wins
- Tightened overly broad phrases to prevent trivial bypass
- _normalize_escaped_newlines only applies to pure-escaped payloads to avoid
corrupting content that discusses escape sequences
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add post-call hook for Lakera guardrail and mask PII in responses
* Add post-call hook for Lakera and mask PII in responses
* Fix post-call hook: pass event_type to call_v2_guard
* Address Greptile review: return ModelResponse, fix mutation, add header, test location, mask order
- PII masking path: return ModelResponse instead of dict so deployment hook accepts it
- Avoid mutating request data: deep copy original_messages and messages in _mask_pii_in_messages
- Add guardrail header in PII-only return path
- Add test in tests/test_litellm/ (test_lakera_ai_v2.py) per PR checklist
- Sort PII payload spans by (start,end) descending so multiple spans in one message mask correctly
Co-authored-by: Cursor <cursoragent@cursor.com>
* Updated ponteital for index mismatch when choices have null content and inconsistent on_flagged access pattern
* Update litellm/proxy/guardrails/guardrail_hooks/lakera_ai_v2.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update to explicitly state supported endpoints - chat completions
* Fix minor lint error on masked_entity_count
---------
Co-authored-by: Steve <steve.giguere@lakera.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Backend - Spend Log Storage for Realtime Calls:
- Collect user voice transcripts and text input during WebSocket sessions
- Store collected messages in spend logs when store_prompts_in_spend_logs enabled
- Capture tool definitions from session.update and tool calls from response.done
- Enrich proxy_server_request with tools and response with tool_calls for UI
Backend - WebSocket Auth:
- Support browser-based auth via Sec-WebSocket-Protocol subprotocol
- Echo back subprotocol on WebSocket accept
UI - Realtime Playground:
- New RealtimePlayground component with WebSocket voice+text chat
- Mic recording (PCM16 24kHz), server VAD, audio playback, text input
- Handle binary WebSocket frames (Blob/ArrayBuffer decoding)
- Add /v1/realtime endpoint option to playground endpoint selector
UI - Tools Section for Realtime Logs:
- Extract tool calls from realtime response format (response.tool_calls
and response.results[].response.output[].type=function_call)
Tests:
- 15 new backend tests for realtime streaming and spend log storage
- 4 new UI tests for realtime tool call extraction
Fixes pre-existing build errors:
- ToolPolicies.tsx: duplicate import, antd styles type
- create_key_button.tsx: missing message import
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(proxy): add max_iterations limiter for agent session loops (#22058)
Adds a new proxy hook that enforces a per-session cap on the number of
LLM calls an agentic loop can make. Callers send a session_id with each
request, and the hook counts calls per session, returning 429 when the
configured max_iterations limit is exceeded.
- Uses Redis Lua script for atomic increment (multi-instance safe)
- Falls back to in-memory cache when Redis unavailable
- Follows parallel_request_limiter_v3 pattern
- Configurable via key metadata: {"max_iterations": 25}
- Session counters auto-expire via TTL (default 1hr)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add new code execution dataset
* feat(agent_endpoints/): allow giving agents keys
* fix: ui fixes
* feat: allow assigning mcp servers to agents
* fix: eliminate duplicate DB queries in MCP agent auth and N+1 in agent listing (#22110)
- Extract _get_agent_object_permission helper so _get_allowed_mcp_servers_for_agent
and _get_agent_tool_permissions_for_server share a single DB fetch instead of
each independently querying the same agent row (was 1+N queries per MCP request)
- Use include={"object_permission": True} on find_many in get_all_agents_from_db
to eagerly load permissions in one query instead of N+1
- Use include={"object_permission": True} on create/update/find_unique in all
agent CRUD operations, removing attach_object_permission_to_dict follow-up calls
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Failure spend logs were missing key metadata (key alias, user ID, team ID,
team alias) in two scenarios:
1. Auth errors (401 ProxyException): auth_exception_handler creates a
minimal UserAPIKeyAuth with only api_key and request_route set — all
other fields are null. The failure hook now looks up the full key object
from cache/DB using the key hash to populate the missing fields.
2. Post-auth failures (provider errors, rate limits): key fields are
present but team_alias is always null because LiteLLM_VerificationTokenView
SQL view does not include team_alias. The failure hook now looks up the
team object from cache to populate team_alias.
Both lookups are non-fatal and wrapped in try/except.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add a `request_duration_ms` column to `LiteLLM_SpendLogs` to track request
duration. New rows are computed at write time. Legacy rows use a COALESCE
fallback in the `/spend/logs/ui` query to compute duration on the fly from
`endTime - startTime`. The field is also sortable in the UI endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(agents): assign virtual keys to agents
- Add agent_id field to LiteLLM_VerificationToken (schema.prisma + _types.py)
- Pass agent_id through key generation endpoint so keys can be scoped to an agent
- Refactor Add Agent wizard to 3-step flow (Configure → Assign Key → Ready)
- Configure: all agent fields, custom/other type with just name+description
- Assign Key: create new key or reassign existing key to agent
- URL is now optional for easy discovery
- Add "Agent" ownership option to Create Key modal on Virtual Keys page
with agent selector dropdown
- Extract CreatedKeyDisplay into shared component, reused in both flows
- Add keyCreateForAgentCall networking helper
- Add test for agent_id key generation
* fix(agents): code quality fixes from self-review
- Fix test_generate_key_helper_fn_agent_id: remove bare except clause,
use explicit assert mock_insert.called, use .kwargs for clean arg access
- Remove no-op conditional in handleNext (both branches were identical)
- Validate selectedExistingKey before calling keyUpdateCall
- Validate selectedAgentId before setting on formValues in create_key_button
* fix(ui): replace deprecated Tremor Button with Ant Design Button in CreatedKeyDisplay
* Add Ask AI chat component to Usage page
- Create UsageAIChatModal component with streaming chat interface
- Integrate with existing model hub for model selection
- Pass usage data context (spend, models, providers, keys) to AI
- Add Ask AI button next to Export Data button in global view
- Add tests for the new component and integration
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Convert Ask AI from modal to right-side sliding panel
- Replace UsageAIChatModal with UsageAIChatPanel
- Panel slides in from right side, usage page stays visible
- Full-height panel with header, model selector, chat area, and input
- Smooth CSS transition for open/close animation
- Update tests for new panel component (34 tests passing)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Remove build output directory from tracking
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add backend AI usage chat endpoint with tool calling
Backend:
- New /usage/ai/chat SSE streaming endpoint
- AI agent has get_usage_data tool that queries /user/daily/activity/aggregated
- Follows same architecture as policy AI suggest (litellm.acompletion + tools)
- Non-admin users are restricted to their own data
- 12 backend unit tests
Frontend:
- Panel now calls /usage/ai/chat backend endpoint via SSE
- Removed direct OpenAI client calls from frontend
- Added usageAiChatStream networking function following enrichPolicyTemplateStream pattern
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Make model selection optional, default to gpt-4o-mini on backend
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add team/tag tools, status indicators, and improved AI agent
- AI agent now has 3 tools: get_usage_data, get_team_usage_data, get_tag_usage_data
- Stream status events (Thinking... Fetching... Analyzing...) to UI
- Frontend shows spinner + status text during tool execution
- Better system prompt guiding tool selection
- Entity summariser for team/tag data with ranked breakdowns
- 13 backend tests, 34 frontend tests passing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix: inject today's date into system prompt so AI resolves relative dates correctly
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Show tool calls as distinct steps + render markdown in responses
- Backend emits tool_call events with tool_name, label, args, and status
- Frontend shows each tool call as a step with ✓/spinner/✗ indicator
- Tool call steps show icon, label, date range, and filters
- AI responses rendered with ReactMarkdown (bold, lists, tables, code)
- Cursor-like UX: Thinking → tool calls → Analyzing → streamed answer
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Refactor backend for code quality: proper types, constants, all functions ≤50 LOC
- TypedDict for SSE events (SSEStatusEvent, SSEToolCallEvent, etc.) and ToolHandler
- Constants for table names, entity fields, temperature, page sizes, top-N limits
- Shared _query_activity() eliminates duplicated fetch logic
- _accumulate_breakdown() + _ranked_lines() replace inline aggregation loops
- Extracted _process_tool_call() and _stream_final_response() from main stream fn
- Black + Ruff clean, all 15 functions verified ≤50 LOC
- Replaced Tremor Button with Antd Button in panel (Tremor deprecated per AGENTS.md)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Address greptile review: security fixes and input validation
- Restrict team/tag tools to admin-only users (non-admins only get get_usage_data)
- Constrain ChatMessage.role to Literal['user', 'assistant'] to prevent system prompt injection
- Add test for base tools restriction (non-admin gets 1 tool, admin gets 3)
- Issues 3 (unused imports) and 4 (inline datetime) were already fixed in prior commit
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Address greptile round 2: sanitize errors, defense-in-depth allowlist, revert tsconfig
- Sanitize error messages: generic 'An internal error occurred' sent to client,
full exception logged server-side via verbose_proxy_logger
- Defense-in-depth: _process_tool_call validates fn_name against role-based
allowlist before dispatch (even though LLM only receives allowed tools)
- Revert tsconfig.json jsx back to 'preserve' (Next.js recommended default)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Role-scoped system prompt + additional test coverage
- System prompt is now role-aware: admin sees all 3 tool descriptions,
non-admin only sees get_usage_data (consistent with tool filtering)
- Added tests: non-admin prompt excludes team/tag tools, date injection
- 15 backend tests, 34 frontend tests passing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix LLM arg validation + cap conversation size at 20 messages
- _resolve_fetch_kwargs uses .get() with ValueError for missing dates
(handles malformed LLM tool arguments gracefully)
- MAX_CHAT_MESSAGES = 20 constant; backend truncates to last 20
- Frontend also sends only last 20 messages per request
- Prevents excessive token usage and context-length errors
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(proxy): tool policies - auto-discover tools, manage policies, guardrail enforcement
- New LiteLLM_ToolTable in schema.prisma to store discovered tools
- Auto-discovery: tools seen in LLM responses get upserted via ToolDiscoveryQueue
(hooks into DBSpendUpdateWriter, same pipeline as spend tracking)
- Management endpoints: GET /v1/tool/list, GET /v1/tool/{name}, POST /v1/tool/policy
- ToolPolicyGuardrail: blocks tool_calls in responses based on policy setting
- UI: Tool Policies page under Guardrails section with policy selector,
filters by policy/team/key, live tail, sortable table
- Unit tests for queue, writer, endpoints, guardrail
* feat(tool-policies): track call_count + discover tools from request body and /messages API
- Add call_count column to LiteLLM_ToolTable; incremented on every flush
- Extract tools from request body too (not just response tool_calls):
- OpenAI /chat/completions: tools[].function.name
- Anthropic /messages pass-through: request_body.tools[].name
- Show call_count column in UI table (sortable)
- UI: drop dual_llm option, keep only trusted/blocked
* fix: address greptile review feedback
- Remove redundant @@index([tool_name]) from schema.prisma (tool_name has @unique which already creates an index)
- Replace gen_random_uuid()::text with str(uuid.uuid4()) for portability
- Rewrite test_tool_registry_writer.py to mock execute_raw/query_raw (actual implementation) instead of Prisma model methods
- Fix test patches in test_tool_management_endpoints.py to target source modules since imports are inside function bodies
- Add "Tool Policies" page title to ToolPolicies.tsx
* fix: address greptile review round 2
- Replace NOW() with Python datetime parameter in tool_registry_writer (SQLite portability)
- Fix cache key collision in tool_policy_guardrail: use null-byte separator instead of colon
- Remove type==function filter from request-side tool extraction to match response-side behavior
- Clear seen_tool_names on flush so call_count increments per batch cycle not per pod lifetime
* fix: address greptile review round 3
- Fix test_seen_names_persist_across_flushes to match actual per-flush-cycle behavior
- Update module docstring in tool_discovery_queue.py to accurately describe flush behavior
- Add created_at/updated_at to raw SQL INSERT in batch_upsert_tools and update_tool_policy
* fix: cache tool policies per tool name not per combination
Previously the cache key was built from the full set of tool names in a
request, so each unique combination of tools got its own cold cache entry
and triggered a separate DB query. With N distinct tools across requests
this was effectively a DB hit on every request.
Now each tool name is cached individually. Cache hits are checked per
tool, only missing tools are fetched from DB in a single batch query,
and each result is cached separately. Once a tool's policy is warm,
any subsequent request using that tool benefits from the cache regardless
of what other tools are in the request.
* Update ui/litellm-dashboard/src/components/ToolPolicies.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(router): preserve _hidden_params in FallbackStreamWrapper so x-litellm-overhead-duration-ms is emitted for streaming requests
* test(router): add regression test for FallbackStreamWrapper _hidden_params preservation
Replace 11 separate asyncio.create_task() calls per request with a
single batched task that runs all spend-update helpers sequentially.
This reduces task scheduling overhead at high RPS (11,000 -> 1,000
tasks/sec at 1K RPS) and cuts 5 copy.deepcopy(payload) calls to 1
shared copy.
Also fixes a mutation bug where the daily agent spend handler received
the raw payload without deepcopy, unlike all other daily helpers.
* auth_with_role_name add region_name arg for cross-account sts
* update tests to include case with aws_region_name for _auth_with_aws_role
* Only pass region_name to STS client when aws_region_name is set
* Add optional aws_sts_endpoint to _auth_with_aws_role
* Parametrize ambient-credentials test for no opts, region_name, and aws_sts_endpoint
* consistently passing region and endpoint args into explicit credentials irsa
* fix env var leakage
* fix: bedrock openai-compatible imported-model should also have model arn encoded
* fix: team assignment fails for keys with special model names (#21880)
---------
Co-authored-by: An Tang <ta@stripe.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
