Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
* feat: implement guardrails usage dashboard backend
Add backend infrastructure for guardrails performance monitoring dashboard:
Database Schema:
- Add LiteLLM_DailyGuardrailMetrics table for daily aggregated metrics
- Track total_requests, success/intervened/failed/not_run counts per guardrail
- Store aggregated latency metrics in milliseconds
- Unique constraint on [guardrail_name, provider, mode, date, api_key]
Data Collection & Aggregation:
- Add DailyGuardrailMetricsTransaction type for queue transactions
- Implement guardrail metrics extraction from spend log metadata
- Add batch upsert logic with retry handling (60s commit interval)
- Process each guardrail separately with status-based counting
API Endpoints:
- GET /guardrail/metrics - List all guardrails with aggregated metrics
- GET /guardrail/{name}/metrics - Detail view with daily time-series
- GET /guardrail/{name}/logs - Request logs with status filtering
Type Definitions:
- Add Pydantic models for API request/response validation
- GuardrailSummary, GuardrailDetailMetrics, GuardrailLogsResponse
Key Features:
- Fail rate = (intervened_count / total_requests) * 100
- Avg latency measures guardrail execution overhead only
- Reuses LiteLLM_SpendLogs for per-request drill-down
- Follows existing daily spend tracking patterns
Testing Required:
- Run: poetry run prisma migrate dev --name add_guardrail_metrics
- Frontend implementation pending (Phase 5)
- See IMPLEMENTATION_STATUS.md for details
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: implement guardrails usage dashboard frontend
Add complete frontend UI for guardrails performance monitoring:
Components:
- GuardrailsTableView: List view with sortable metrics table
- GuardrailDetailView: Detail page with metric cards and tabs
- GuardrailLogsTab: Request logs with expandable details
- Types and mock data for development/testing
Pages:
- /guardrails/metrics: Main metrics dashboard page
- /guardrails/metrics/[name]: Individual guardrail detail page
Features:
- Date range picker for filtering metrics
- Color-coded fail rates (red >10%, yellow >5%)
- Clickable table rows for drill-down
- Expandable log entries with full guardrail response
- Status filter (All, Blocked, Passed)
- Area chart for fail rate trends
- Daily metrics table in detail view
Mock Data:
- USE_MOCK_DATA flag enabled for development
- Sample data for 5 guardrails with realistic metrics
- Toggle flag to false to use real API endpoints
Next Steps:
- Run Prisma migration to create database table
- Set USE_MOCK_DATA=false to connect to backend
- Test with real guardrail traffic
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add frontend completion summary
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Replace Zapier webhook with Google Form for survey submission
* Add back error logging for survey submission debugging
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
* feat: new response rejection detector
* fix: multiple fixes
* feat: add mcp tools support to agent builder
create an agent with access to llm's + mcp servers
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
Replace Click CliRunner with standalone_mode=False to avoid
"I/O operation on closed file" errors caused by Click's stream
isolation in CI environments.
* fix(logging): preserve pass-through endpoint response_cost in async_success_handler
Two places in the logging pipeline were overwriting response_cost that
pass-through handlers (Gemini/Vertex) had already calculated:
1. _process_hidden_params_and_response_cost fell through to
_response_cost_calculator which returns None for pass-through calls
2. async_success_handler pass-through branch unconditionally set
response_cost = None (introduced in PR #19887)
Now both places check if response_cost is already set before overwriting.
* test: add regression test for pass-through endpoint response_cost preservation
