Accurate logging is more important than avoiding the rare DB lookup
on the async failure path. Remove check_cache_only=True so lookups
fall back to DB when the key/team is not in cache.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Failure spend logs were missing key metadata (key alias, user ID, team ID,
team alias) in two scenarios:
1. Auth errors (401 ProxyException): auth_exception_handler creates a
minimal UserAPIKeyAuth with only api_key and request_route set — all
other fields are null. The failure hook now looks up the full key object
from cache/DB using the key hash to populate the missing fields.
2. Post-auth failures (provider errors, rate limits): key fields are
present but team_alias is always null because LiteLLM_VerificationTokenView
SQL view does not include team_alias. The failure hook now looks up the
team object from cache to populate team_alias.
Both lookups are non-fatal and wrapped in try/except.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Google's Interactions API spec changed the status enum:
- Values are now lowercase (was uppercase)
- 'UNSPECIFIED' value was removed
Updated test to match the current spec from:
https://ai.google.dev/static/api/interactions.openapi.json
Add a `request_duration_ms` column to `LiteLLM_SpendLogs` to track request
duration. New rows are computed at write time. Legacy rows use a COALESCE
fallback in the `/spend/logs/ui` query to compute duration on the fly from
`endTime - startTime`. The field is also sortable in the UI endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(agents): assign virtual keys to agents
- Add agent_id field to LiteLLM_VerificationToken (schema.prisma + _types.py)
- Pass agent_id through key generation endpoint so keys can be scoped to an agent
- Refactor Add Agent wizard to 3-step flow (Configure → Assign Key → Ready)
- Configure: all agent fields, custom/other type with just name+description
- Assign Key: create new key or reassign existing key to agent
- URL is now optional for easy discovery
- Add "Agent" ownership option to Create Key modal on Virtual Keys page
with agent selector dropdown
- Extract CreatedKeyDisplay into shared component, reused in both flows
- Add keyCreateForAgentCall networking helper
- Add test for agent_id key generation
* fix(agents): code quality fixes from self-review
- Fix test_generate_key_helper_fn_agent_id: remove bare except clause,
use explicit assert mock_insert.called, use .kwargs for clean arg access
- Remove no-op conditional in handleNext (both branches were identical)
- Validate selectedExistingKey before calling keyUpdateCall
- Validate selectedAgentId before setting on formValues in create_key_button
* fix(ui): replace deprecated Tremor Button with Ant Design Button in CreatedKeyDisplay
Add a reusable KeyInfoHeader component to replace the inline header in KeyInfoView.
Extract LabeledField as a common component for labeled metadata display with copyable
support, default_user_id handling, and empty value placeholders.
- Migrate all icons from lucide-react/heroicons to @ant-design/icons
- Use antd native copyable with descriptive tooltips (Copy Key Alias, Copy Key ID, etc.)
- Show DefaultProxyAdminTag for default_user_id values
- Add canModifyKey, regenerateDisabled, regenerateTooltip props for permission gating
- Add tests for KeyInfoHeader (19 tests) and LabeledField (8 tests)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These params were silently dropped for Chat Completions because they
were missing from the supported params whitelist. Also adds
prompt_cache_retention to the Responses API TypedDict and fixes
misleading cache_control comments in OpenAI prompt caching docs.
* Add Ask AI chat component to Usage page
- Create UsageAIChatModal component with streaming chat interface
- Integrate with existing model hub for model selection
- Pass usage data context (spend, models, providers, keys) to AI
- Add Ask AI button next to Export Data button in global view
- Add tests for the new component and integration
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Convert Ask AI from modal to right-side sliding panel
- Replace UsageAIChatModal with UsageAIChatPanel
- Panel slides in from right side, usage page stays visible
- Full-height panel with header, model selector, chat area, and input
- Smooth CSS transition for open/close animation
- Update tests for new panel component (34 tests passing)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Remove build output directory from tracking
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add backend AI usage chat endpoint with tool calling
Backend:
- New /usage/ai/chat SSE streaming endpoint
- AI agent has get_usage_data tool that queries /user/daily/activity/aggregated
- Follows same architecture as policy AI suggest (litellm.acompletion + tools)
- Non-admin users are restricted to their own data
- 12 backend unit tests
Frontend:
- Panel now calls /usage/ai/chat backend endpoint via SSE
- Removed direct OpenAI client calls from frontend
- Added usageAiChatStream networking function following enrichPolicyTemplateStream pattern
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Make model selection optional, default to gpt-4o-mini on backend
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add team/tag tools, status indicators, and improved AI agent
- AI agent now has 3 tools: get_usage_data, get_team_usage_data, get_tag_usage_data
- Stream status events (Thinking... Fetching... Analyzing...) to UI
- Frontend shows spinner + status text during tool execution
- Better system prompt guiding tool selection
- Entity summariser for team/tag data with ranked breakdowns
- 13 backend tests, 34 frontend tests passing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix: inject today's date into system prompt so AI resolves relative dates correctly
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Show tool calls as distinct steps + render markdown in responses
- Backend emits tool_call events with tool_name, label, args, and status
- Frontend shows each tool call as a step with ✓/spinner/✗ indicator
- Tool call steps show icon, label, date range, and filters
- AI responses rendered with ReactMarkdown (bold, lists, tables, code)
- Cursor-like UX: Thinking → tool calls → Analyzing → streamed answer
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Refactor backend for code quality: proper types, constants, all functions ≤50 LOC
- TypedDict for SSE events (SSEStatusEvent, SSEToolCallEvent, etc.) and ToolHandler
- Constants for table names, entity fields, temperature, page sizes, top-N limits
- Shared _query_activity() eliminates duplicated fetch logic
- _accumulate_breakdown() + _ranked_lines() replace inline aggregation loops
- Extracted _process_tool_call() and _stream_final_response() from main stream fn
- Black + Ruff clean, all 15 functions verified ≤50 LOC
- Replaced Tremor Button with Antd Button in panel (Tremor deprecated per AGENTS.md)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Address greptile review: security fixes and input validation
- Restrict team/tag tools to admin-only users (non-admins only get get_usage_data)
- Constrain ChatMessage.role to Literal['user', 'assistant'] to prevent system prompt injection
- Add test for base tools restriction (non-admin gets 1 tool, admin gets 3)
- Issues 3 (unused imports) and 4 (inline datetime) were already fixed in prior commit
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Address greptile round 2: sanitize errors, defense-in-depth allowlist, revert tsconfig
- Sanitize error messages: generic 'An internal error occurred' sent to client,
full exception logged server-side via verbose_proxy_logger
- Defense-in-depth: _process_tool_call validates fn_name against role-based
allowlist before dispatch (even though LLM only receives allowed tools)
- Revert tsconfig.json jsx back to 'preserve' (Next.js recommended default)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Role-scoped system prompt + additional test coverage
- System prompt is now role-aware: admin sees all 3 tool descriptions,
non-admin only sees get_usage_data (consistent with tool filtering)
- Added tests: non-admin prompt excludes team/tag tools, date injection
- 15 backend tests, 34 frontend tests passing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix LLM arg validation + cap conversation size at 20 messages
- _resolve_fetch_kwargs uses .get() with ValueError for missing dates
(handles malformed LLM tool arguments gracefully)
- MAX_CHAT_MESSAGES = 20 constant; backend truncates to last 20
- Frontend also sends only last 20 messages per request
- Prevents excessive token usage and context-length errors
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
The /mcp endpoint requires a trailing slash because the MCP server
is mounted as a sub-application using app.mount(). Starlette's mount
behavior causes a 307 redirect from /mcp to /mcp/, which many MCP
clients fail to handle.
Updates documentation examples to use /mcp/ consistently.
_cached_get_model_group_info uses @lru_cache but had no invalidation,
causing stale model group info (TPM/RPM limits) after dynamic deployment
changes. Add cache_clear() at all 5 model_list mutation sites.
* feat(proxy): tool policies - auto-discover tools, manage policies, guardrail enforcement
- New LiteLLM_ToolTable in schema.prisma to store discovered tools
- Auto-discovery: tools seen in LLM responses get upserted via ToolDiscoveryQueue
(hooks into DBSpendUpdateWriter, same pipeline as spend tracking)
- Management endpoints: GET /v1/tool/list, GET /v1/tool/{name}, POST /v1/tool/policy
- ToolPolicyGuardrail: blocks tool_calls in responses based on policy setting
- UI: Tool Policies page under Guardrails section with policy selector,
filters by policy/team/key, live tail, sortable table
- Unit tests for queue, writer, endpoints, guardrail
* feat(tool-policies): track call_count + discover tools from request body and /messages API
- Add call_count column to LiteLLM_ToolTable; incremented on every flush
- Extract tools from request body too (not just response tool_calls):
- OpenAI /chat/completions: tools[].function.name
- Anthropic /messages pass-through: request_body.tools[].name
- Show call_count column in UI table (sortable)
- UI: drop dual_llm option, keep only trusted/blocked
* fix: address greptile review feedback
- Remove redundant @@index([tool_name]) from schema.prisma (tool_name has @unique which already creates an index)
- Replace gen_random_uuid()::text with str(uuid.uuid4()) for portability
- Rewrite test_tool_registry_writer.py to mock execute_raw/query_raw (actual implementation) instead of Prisma model methods
- Fix test patches in test_tool_management_endpoints.py to target source modules since imports are inside function bodies
- Add "Tool Policies" page title to ToolPolicies.tsx
* fix: address greptile review round 2
- Replace NOW() with Python datetime parameter in tool_registry_writer (SQLite portability)
- Fix cache key collision in tool_policy_guardrail: use null-byte separator instead of colon
- Remove type==function filter from request-side tool extraction to match response-side behavior
- Clear seen_tool_names on flush so call_count increments per batch cycle not per pod lifetime
* fix: address greptile review round 3
- Fix test_seen_names_persist_across_flushes to match actual per-flush-cycle behavior
- Update module docstring in tool_discovery_queue.py to accurately describe flush behavior
- Add created_at/updated_at to raw SQL INSERT in batch_upsert_tools and update_tool_policy
* fix: cache tool policies per tool name not per combination
Previously the cache key was built from the full set of tool names in a
request, so each unique combination of tools got its own cold cache entry
and triggered a separate DB query. With N distinct tools across requests
this was effectively a DB hit on every request.
Now each tool name is cached individually. Cache hits are checked per
tool, only missing tools are fetched from DB in a single batch query,
and each result is cached separately. Once a tool's policy is warm,
any subsequent request using that tool benefits from the cache regardless
of what other tools are in the request.
* Update ui/litellm-dashboard/src/components/ToolPolicies.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- Wait for strategy select (API data loaded) before clicking Save
- Assert specific payload content in setCallbacksCall
- Move NotificationsManager import to top of file
Cache the dict(request.headers) result on request.state._cached_headers
so subsequent calls within the same request return the cached dict
instead of re-creating it each time.
Replace 15 call sites across 9 files that called dict(request.headers)
with _safe_get_request_headers(request) which caches the result on
request.state. Mutation sites use .copy() to protect the shared cache.
Reorder the check to use str.startswith(tuple) first, which immediately
returns False for non-o-series models (the common case), avoiding the
genexpr + 198-element set lookup. Line profiling shows 4.75x speedup
(13.6µs → 2.9µs per call, 2.45s → 0.52s across 180k calls).
Avoid constructing a default Usage() object that gets immediately
overwritten by convert_to_model_response_object. Set usage=None
instead; the real Usage is assigned via setattr later.
Also fix Bedrock Qwen2/Qwen3 transform_response to assign a new
Usage object instead of mutating a potentially missing one.
- Remove re-raise in _store_transactions_in_redis so one Redis
push failure doesn't drop remaining transaction types
- Downgrade per-push success log from info to debug to reduce noise
- Fix misleading error message in update_database — entity spend
updates run as independent tasks and are not affected by this catch
Elevate silent debug-level and bare except:pass error paths to
warning/error so spend tracking failures are visible in production logs.
All new log messages are prefixed with "Spend tracking -" for easy
filtering. Changes cover the full request-to-DB lifecycle:
enqueue, in-memory flush, Redis buffer push/pop, DB commit,
cache updates, spend log writes, and pod lock management.
Also fixes a copy-paste bug in _update_team_cache that logged
"end user" instead of "team".
* fix(router): preserve _hidden_params in FallbackStreamWrapper so x-litellm-overhead-duration-ms is emitted for streaming requests
* test(router): add regression test for FallbackStreamWrapper _hidden_params preservation
Set prometheus_emit_stream_label: true in litellm_settings to emit a
stream label (True/False/None) on litellm_proxy_total_requests_metric.
Opt-in to avoid breaking cardinality on existing deployments.
Non-owner Internal Users could see and interact with the "Edit Settings"
button in the key Settings tab for keys they don't own. The button was
gated by `rolesWithWriteAccess.includes(userRole)` (role-only check)
instead of `canModifyKey` (ownership-aware), unlike the Regenerate and
Delete buttons which already used the correct check.
Replace the condition with `canModifyKey` so the Edit Settings button
follows the same proxy-admin / team-admin / key-owner logic as the
other action buttons. Add tests covering all permission paths.
Replace 11 separate asyncio.create_task() calls per request with a
single batched task that runs all spend-update helpers sequentially.
This reduces task scheduling overhead at high RPS (11,000 -> 1,000
tasks/sec at 1K RPS) and cuts 5 copy.deepcopy(payload) calls to 1
shared copy.
Also fixes a mutation bug where the daily agent spend handler received
the raw payload without deepcopy, unlike all other daily helpers.
* staged first pass
* black
* Update litellm/proxy/health_check.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* simpler
* restore cached logo
* fix tests for perform_health_check max_concurrency arg
* implement pr suggestion
* and the helm chart
* add configureable resources and probes to the deployment in the helm chart
* more helm chart unittests
* move some background healthcheck loggin to debug
---------
Co-authored-by: Sean Glover <sglover@athenahealth.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
