* feat(proxy): add max_iterations limiter for agent session loops (#22058)
Adds a new proxy hook that enforces a per-session cap on the number of
LLM calls an agentic loop can make. Callers send a session_id with each
request, and the hook counts calls per session, returning 429 when the
configured max_iterations limit is exceeded.
- Uses Redis Lua script for atomic increment (multi-instance safe)
- Falls back to in-memory cache when Redis unavailable
- Follows parallel_request_limiter_v3 pattern
- Configurable via key metadata: {"max_iterations": 25}
- Session counters auto-expire via TTL (default 1hr)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add new code execution dataset
* feat(agent_endpoints/): allow giving agents keys
* fix: ui fixes
* feat: allow assigning mcp servers to agents
* fix: eliminate duplicate DB queries in MCP agent auth and N+1 in agent listing (#22110)
- Extract _get_agent_object_permission helper so _get_allowed_mcp_servers_for_agent
and _get_agent_tool_permissions_for_server share a single DB fetch instead of
each independently querying the same agent row (was 1+N queries per MCP request)
- Use include={"object_permission": True} on find_many in get_all_agents_from_db
to eagerly load permissions in one query instead of N+1
- Use include={"object_permission": True} on create/update/find_unique in all
agent CRUD operations, removing attach_object_permission_to_dict follow-up calls
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Added unit tests for all pricing calculator components with 64 passing tests across 5 test files:
- multi_export_utils.test.ts (16 tests for PDF/CSV export functions)
- use_multi_cost_estimate.test.ts (15 tests for cost estimation hook)
- multi_export_dropdown.test.tsx (8 tests for export dropdown component)
- multi_cost_results.test.tsx (15 tests for results display and UI states)
- index.test.tsx (10 tests for main calculator component)
Also fixed missing page description for tool-policies page in page_metadata.ts.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* Add Ask AI chat component to Usage page
- Create UsageAIChatModal component with streaming chat interface
- Integrate with existing model hub for model selection
- Pass usage data context (spend, models, providers, keys) to AI
- Add Ask AI button next to Export Data button in global view
- Add tests for the new component and integration
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Convert Ask AI from modal to right-side sliding panel
- Replace UsageAIChatModal with UsageAIChatPanel
- Panel slides in from right side, usage page stays visible
- Full-height panel with header, model selector, chat area, and input
- Smooth CSS transition for open/close animation
- Update tests for new panel component (34 tests passing)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Remove build output directory from tracking
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add backend AI usage chat endpoint with tool calling
Backend:
- New /usage/ai/chat SSE streaming endpoint
- AI agent has get_usage_data tool that queries /user/daily/activity/aggregated
- Follows same architecture as policy AI suggest (litellm.acompletion + tools)
- Non-admin users are restricted to their own data
- 12 backend unit tests
Frontend:
- Panel now calls /usage/ai/chat backend endpoint via SSE
- Removed direct OpenAI client calls from frontend
- Added usageAiChatStream networking function following enrichPolicyTemplateStream pattern
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Make model selection optional, default to gpt-4o-mini on backend
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Add team/tag tools, status indicators, and improved AI agent
- AI agent now has 3 tools: get_usage_data, get_team_usage_data, get_tag_usage_data
- Stream status events (Thinking... Fetching... Analyzing...) to UI
- Frontend shows spinner + status text during tool execution
- Better system prompt guiding tool selection
- Entity summariser for team/tag data with ranked breakdowns
- 13 backend tests, 34 frontend tests passing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix: inject today's date into system prompt so AI resolves relative dates correctly
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Show tool calls as distinct steps + render markdown in responses
- Backend emits tool_call events with tool_name, label, args, and status
- Frontend shows each tool call as a step with ✓/spinner/✗ indicator
- Tool call steps show icon, label, date range, and filters
- AI responses rendered with ReactMarkdown (bold, lists, tables, code)
- Cursor-like UX: Thinking → tool calls → Analyzing → streamed answer
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Refactor backend for code quality: proper types, constants, all functions ≤50 LOC
- TypedDict for SSE events (SSEStatusEvent, SSEToolCallEvent, etc.) and ToolHandler
- Constants for table names, entity fields, temperature, page sizes, top-N limits
- Shared _query_activity() eliminates duplicated fetch logic
- _accumulate_breakdown() + _ranked_lines() replace inline aggregation loops
- Extracted _process_tool_call() and _stream_final_response() from main stream fn
- Black + Ruff clean, all 15 functions verified ≤50 LOC
- Replaced Tremor Button with Antd Button in panel (Tremor deprecated per AGENTS.md)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Address greptile review: security fixes and input validation
- Restrict team/tag tools to admin-only users (non-admins only get get_usage_data)
- Constrain ChatMessage.role to Literal['user', 'assistant'] to prevent system prompt injection
- Add test for base tools restriction (non-admin gets 1 tool, admin gets 3)
- Issues 3 (unused imports) and 4 (inline datetime) were already fixed in prior commit
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Address greptile round 2: sanitize errors, defense-in-depth allowlist, revert tsconfig
- Sanitize error messages: generic 'An internal error occurred' sent to client,
full exception logged server-side via verbose_proxy_logger
- Defense-in-depth: _process_tool_call validates fn_name against role-based
allowlist before dispatch (even though LLM only receives allowed tools)
- Revert tsconfig.json jsx back to 'preserve' (Next.js recommended default)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Role-scoped system prompt + additional test coverage
- System prompt is now role-aware: admin sees all 3 tool descriptions,
non-admin only sees get_usage_data (consistent with tool filtering)
- Added tests: non-admin prompt excludes team/tag tools, date injection
- 15 backend tests, 34 frontend tests passing
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Fix LLM arg validation + cap conversation size at 20 messages
- _resolve_fetch_kwargs uses .get() with ValueError for missing dates
(handles malformed LLM tool arguments gracefully)
- MAX_CHAT_MESSAGES = 20 constant; backend truncates to last 20
- Frontend also sends only last 20 messages per request
- Prevents excessive token usage and context-length errors
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui/): initial commit adding a compliance testing playground
allow proxy admins to test policies and guardrails against datasets
* feat(ui/): make score more friendly
* feat(policy_endpoints.py): new helper function for testing policies
* feat(policy_endpoints.py): expose new endpoint for testing policies and guardrails
enables compliance playground to work as expected
* feat(complianceui.tsx): show returned text
* Add 6 new EU PII patterns for GDPR compliance
- fr_nir: French Social Security Number (NIR/INSEE) with validation
- eu_iban_enhanced: Enhanced IBAN detection with specific format
- fr_phone: French phone numbers (+33, 0033, 0 formats)
- eu_vat: EU VAT identification numbers (all 27 member states)
- eu_passport_generic: Generic EU passport format
- fr_postal_code: French postal codes with contextual keywords
* Add GDPR Art. 32 EU PII Protection policy template
- Comprehensive GDPR Article 32 compliance policy
- 4 guardrail groups: National IDs, Financial, Contact Info, Business IDs
- Masks French NIR/INSEE, EU IBANs, French phones, EU VAT numbers
- Includes EU passport numbers and email addresses
- Medium complexity template with indigo icon
* Add comprehensive tests for EU PII patterns
- Test French NIR validation (sex digit, month range)
- Test enhanced IBAN detection (French, German)
- Test French phone number formats
- Test EU VAT numbers
- Test generic EU passport format
- Test French postal code pattern
* Add EU pattern loading and category validation tests
- Verify all 6 EU PII patterns are loaded correctly
- Verify patterns are categorized as 'EU PII Patterns'
- Ensure pattern loading consistency
* Add end-to-end tests for GDPR policy template
- 4 tests for PII that should be masked (NIR, IBAN, phone, VAT)
- 4 tests for text that should pass through (invalid patterns, no PII)
- 1 bonus test for multiple PII types in same message
- All tests verify correct masking behavior
* Add region field to policy templates
- Added region field to all 6 templates (EU, AU, Global)
- Updated both main and backup JSON files
- Enables region-based filtering in UI
* Add region filter to policy templates UI
- Added Radio.Group filter for regions (All, AU, EU, Global)
- Efficient filtering with useMemo hooks
- Clean button-based UI matching existing design
- Defaults missing regions to Global
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Address Greptile review: add contextual guards and negative tests
- Added keyword_pattern to eu_vat (VAT, tax number, fiscal code, etc.)
- Added keyword_pattern to eu_passport_generic (passport, travel document, etc.)
- Added 3 negative unit tests for false positive prevention
- Added 2 E2E tests verifying no masking without keyword context
- All patterns now require contextual keywords to prevent false positives
* address greptile review feedback (greploop iteration 1)
- Remove unused HTTPException import from test file
- Add keyword_pattern to eu_vat for contextual VAT matching
- Add allow_word_numbers: false to eu_passport_generic
- Add negative test cases for EU VAT false positives
- All 5 Greptile comments addressed
* Address Greptile feedback: fix patterns and sync backup
- Fix fr_phone pattern: use negative lookbehind (?<!\d) to prevent false matches in longer digit strings
- Add keyword_pattern to eu_passport_generic to reduce false positives on version strings/SKUs
- Sync policy_templates_backup.json with main file (add GDPR template)
- Add keyword_pattern to eu_vat (was auto-added by formatter)
All pattern tests passing
* address greptile review feedback (greploop iteration 2)
- Update test to document that eu_vat raw pattern is intentionally broad
- Test verifies pattern DOES match common words (by design)
- Documents that keyword_pattern guard prevents false positives in production
- Addresses Greptile's false positive risk concern
* address greptile review feedback (greploop iteration 3)
- Fix test_eu_vat_masked: change gap from 2 words to 1 word (VAT number: FR...)
- This ensures keyword matching works within MAX_KEYWORD_VALUE_GAP_WORDS=1 limit
- fr_phone pattern already works correctly (verified with tests)
- test_pattern_requires_keyword_context already updated in iteration 2
Addresses final issues from Greptile 2/5 review
* fix: remove country-specific passport patterns from GDPR template
- Remove passport_france, passport_germany, passport_netherlands from template
- These patterns lack keyword guards and cause false positives
- Only eu_passport_generic remains (has keyword_pattern guard)
- Sync policy_templates_backup.json with main file
- Update test setup to match template
All 11 E2E tests now passing ✅
* Update tests/test_litellm/proxy/guardrails/guardrail_hooks/content_filter/test_gdpr_policy_e2e.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/litellm_content_filter/content_filter.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(guardrail_hooks/): add guardrail logging to all unified guardrails
ensures unified guardrails use the 'log_guardrail_information' decorator for logging
* fix(custom_guardrail.py): don't log inputs on guardrail response - just emit state
* refactor: don't double log bedrock guardrail information
* feat: add in-product nudges for contributing + trying community custom code guardrails
allows users to contribute / share custom code guardrails
* added mcp guardrails doc in mcp.md
* add button to reload models
* Added button changes
* added button for scheduling reload
* add multi pod support to reloading the model price json
* fix ruff
* feat: Implement health check functionality for models
- Added individual model health check API call and integrated it into the dashboard.
- Introduced health check history and latest health checks retrieval.
- Enhanced model dashboard to display health status, last check time, and error details.
- Created a new health check columns component for better organization of health data.
- Added error pattern handling for meaningful error messages.
- Updated UI components to support health check actions and display loading states.
* refactor: extract health check functionality into a new component
- Removed individual health check logic from ModelDashboard and encapsulated it within HealthCheckComponent for better modularity and maintainability.
- Updated ModelDashboard to utilize the new HealthCheckComponent, streamlining the health check process and improving code organization.
- Cleaned up unused state variables and imports related to health checks in ModelDashboard.
- Enhanced error handling and status management within the new component, ensuring a more robust health check experience.
* fix: add access token check in HealthCheckComponent to prevent unauthorized API calls
- Implemented a check for the presence of an access token before making API calls to fetch the latest health checks, enhancing security and preventing potential errors when the token is missing.
