- Prefix credential name tags with "Credential: " to distinguish them
from user-defined tags when litellm_credential_name is injected
- Remove stale "new feature" banners from Organization, Customer, and
A2A usage views
- Add closable info banner to Tag usage view noting that reusable
credentials are automatically tracked and appear as "Credential: <name>"
* feat: add competitor name blocker guardrail
* fix: fix batch test endpoint for compliance playground
* fix(airline.py): add list of all known airlines to airline competitor name detector
prevent competitor discussion on company chatbot
* feat: ui tweaks for prod
* ui: redesign guardrail creation form with inline vertical stepper
Replace horizontal Ant Design Steps with an inline vertical stepper.
Completed steps collapse to a single line, active step expands.
Switch to Tremor buttons, rename steps for clarity.
* ui: rename Content Categories to Blocked topics and fix overflow
Update heading and description text, add flexWrap to prevent
text from going off-screen, fix YAML preview overflow with
pre-wrap and word-break.
* feat: support explicit display_name in content filter category YAML
Check for a display_name field before auto-generating from
category_name. Lets categories have human-friendly names
without changing their API identifier.
* fix: update denied_financial_advice display name
Add display_name field so it shows as
"Denied Financial / Investment Advice" in the UI.
* Add keyword-based topic blocker implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add embedding-based topic blocker using MiniLM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add topic blocker package init with exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add synthetic engine eval set (34 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment questions eval set (207 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine eval synthetic policy config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MiniLM eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MPNet eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment TF-IDF eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add unified eval runner with confusion matrix reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add benchmarks comparison table in markdown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Clean up topic blocker: remove unused blockers, add phrase_patterns to content filter
- Remove embedding_blocker.py, api_embedding_blocker.py, nli_blocker.py,
tfidf_blocker.py, onnx_blocker.py (heavy deps not in Docker, inferior accuracy)
- Remove airline_off_topic_restriction policy template and its test
- Fix __init__.py to only export DeniedTopic and TopicBlocker (no eager import crash)
- Add phrase_patterns support to ContentFilterGuardrail for regex-based paraphrase detection
- Rewrite denied_financial_advice.yaml with conditional matching (identifier + block word),
always-block keywords, phrase patterns, and exception phrases
- Clean up test_eval.py: only keyword blocker + content filter tests remain (no network calls)
- All 207 eval cases pass at 100% F1, 0 FP, 0 FN, <0.1ms latency
Addresses all Greptile review comments:
- Eager import crash (embedding deps) → fixed
- Undeclared dependencies → fixed (files deleted)
- lru_cache memory leak → fixed (file deleted)
- Real network calls in tests → fixed (embedding tests removed)
- Unused Dict import → already fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add LLM-as-judge eval and update BENCHMARKS.md
- Add TestInvestmentLlmJudgeGpt4oMini and TestInvestmentLlmJudgeClaude
test classes that use litellm.completion() to classify messages
- System prompt instructs LLM to act as airline chatbot content moderator
- Tests skip gracefully when API keys aren't set
- Update BENCHMARKS.md with production results table, historical comparison,
and instructions for running LLM judge evals
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move evals and benchmarks to guardrail_benchmarks folder
Move eval runner, eval data (JSONL), and results from
tests/test_litellm/.../topic_blocker/ into the guardrail implementation
folder at litellm/.../litellm_content_filter/guardrail_benchmarks/.
This keeps benchmarks co-located with the guardrail code they test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove standalone topic_blocker package, consolidate into content_filter
The standalone keyword_blocker.py was redundant with content_filter.py +
denied_financial_advice.yaml. Removed the entire topic_blocker/ package,
engine eval files, and old keyword blocker results. Simplified test_eval.py
to only test ContentFilter + LLM judge baselines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix compliance playground batch scoring bug, add display_name support
The compliance playground was sending all texts in a single batch API call,
but the content filter raises HTTPException on the first blocked text. This
caused a single blocked/allowed result to be applied to all rows, producing
incorrect scores (e.g. 41% instead of 100%). Fix by sending each text
individually to get per-text results with progressive UI updates.
Also add display_name field support for category YAML files so
denied_financial_advice shows as "Denied Financial / Investment Advice"
in the UI dropdown.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add block_investment CSV eval set and update benchmark result JSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
Fix stale test path in denied_financial_advice.yaml comment.
Other comments were on files already deleted in prior commits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
GuardrailViewer was rewritten from ant-design Collapse to a custom card
layout. Tests now match the new component: updated header text, ms-based
duration, expand-to-reveal provider details, and removed ant-collapse
references.
AllModelsTab tests failed because ModelSettingsModal now uses useMutation
via useStoreModelInDB. Switched from bare render() to renderWithProviders()
which wraps in QueryClientProvider.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When backend filters (Key Alias, Key Hash, etc.) were active, the main
logs query still refetched whenever startTime/endTime/sort/page changed,
firing a redundant unfiltered server request whose result was discarded.
Expose hasBackendFilters from useLogFilterLogic and use it to gate the
main query's enabled condition.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove fetchKeyHashForAlias: Key Alias filtering is handled server-side
by performSearch via key_alias; translating the alias to api_key hash
caused a duplicate main-query request alongside performSearch's request.
The effect now sets selectedKeyHash = filters["Key Hash"] || "" directly.
- Add setCurrentPage(1) to quick select time range handler so the page
resets to 1 when the user picks a preset time window (was keeping
the previous page number, e.g. page=4, in the API request).
- Add comments explaining the intentionally omitted react-hooks/exhaustive-deps
in the performSearch effect per Greptile review feedback.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Fix two bugs in the logs table with backend filters (e.g., Key Alias):
1. Bug 1 - Table doesn't update with custom time range: When Key Alias filter was active and user selected a custom time range, the main query would refetch (network request visible) but backendFilteredLogs would stay stale because the performSearch effect only watched [sortBy, sortOrder, currentPage]. Added startTime, endTime, isCustomDate to the effect deps.
2. Bug 2 - Pagination shows wrong results: fetchKeyHashForAlias incorrectly had currentPage (log page) in its deps, causing it to search the wrong page of the key list and trigger unnecessary effect re-runs. Removed currentPage from deps and always pass page 1 for key alias lookup.
Also added debouncedSearch.cancel() in the effect to prevent race conditions when pagination happens within 300ms of filter application.
Added tests verifying that time range changes trigger refetch when backend filters are active.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
- Show "-" when retry info is absent (older logs)
- Show green "None" tag when not retried (attempted_retries === 0)
- Update max_retries after deployment/retry-policy overrides (greptile feedback)
- Update tests to match new display behavior
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add attempted_retries and max_retries fields to SpendLogsMetadata so the
Logs page can display how many retries occurred for each request. The
router now injects retry tracking metadata before each make_call, which
flows through the logging pipeline into the spend logs metadata JSON.
The UI shows "Not Retried" when the first attempt succeeded, and
"N / M" (attempted / max) when retries occurred. The field is hidden
for requests that did not go through the router.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Prefix all 62 test names with 'should' per AGENTS.md convention
- Wrap fireEvent.click() calls in act() in ModelsCell.test.tsx
- Replace querySelector('.bg-blue-500') with within()+getByTestId in
TeamsFilters.test.tsx; add data-testid="active-filter-indicator" to source
- Add aria-label="Close" to X button in DeleteTeamModal.tsx; update test
to use getByRole('button', { name: /close/i }) instead of fragile index
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(policy): add estimated_latency_ms to all policy templates
* feat(policy): add estimated_latency_ms to backup templates
* feat(ui): show latency overhead badge in AI policy suggestions
* add spec_path column to LiteLLM_MCPServerTable schema
* add spec_path to MCP request types and table model
* wire spec_path through build_mcp_server_from_table
* add openapi transport type constant
* add OpenAPI Spec as first-class transport option in create form
* add OpenAPI transport support to edit form with auto-detection
* support spec_path in connection status component
* support spec_path in tool configuration component
* support OpenAPI transport in test connection hook
* register OpenAPI tools on server add/update/reload
* preview OpenAPI tools in test/tools/list endpoint
* fix aviation safety topic filter: remove overly broad exceptions, add cockpit access block words
* fix airline brand protection filter: add identifier words, competitor/ops block words, tighten exceptions
* add example_sentences to all policy templates + topic-filtering and prompt-injection templates
* add policy_endpoints package with AI policy suggester
* update test patch targets for policy_endpoints package move
* add unit tests for AI policy suggester
* add suggestPolicyTemplates networking function
* add AI suggestion modal component
* add Use AI button and template loading callback to PolicyTemplates
* wire up AI suggestion modal in policies page
* fix policy_templates_backup.json path after package move
* add estimated_latency field to all policy templates
* use llm_router and accept model parameter in ai_policy_suggester
* add model param to suggest templates endpoint
* pass model param in suggestPolicyTemplates
* polish ai suggestion modal: model selector, auto-growing textareas, latency badges
* add template queue for processing multiple AI-suggested templates
* show template progress badge in guardrail selection modal
Virtual keys only track created_at and updated_at, which don't indicate
when a key was last used. This adds a last_active field that gets updated
during the async batch spend update, giving admins visibility into which
keys are actively being used.
Changes:
- Add last_active DateTime? to VerificationToken and
DeletedVerificationToken in all 3 schema files and Python types
- Set last_active in the batch key spend update alongside spend increment
- Add Last Active column to virtual keys UI table with info popover
and hover tooltip showing full date/time with timezone
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix aviation safety topic filter: remove overly broad exceptions, add cockpit access block words
* fix airline brand protection filter: add identifier words, competitor/ops block words, tighten exceptions
* update policy templates with competitor pre/post guardrails and streaming enrichment
* sync policy_templates_backup.json with policy_templates.json
* add streaming enrichment endpoint, competitor variations, and model selection
* add streaming enrichPolicyTemplate networking function
* rewrite template parameter modal with streaming tags, AI/manual toggle, model selector
* update policies panel to pass enrichment options and show discovered competitors
* show discovered competitors as tags in guardrail selection modal
* use llm_router instead of litellm.acompletion, extract helpers, move constants
* validate competitors list size, cap variation prompt input
* add refinement instruction support for competitor discovery
* add instruction and existingCompetitors params to streaming enrichment
* add refinement input for iterating on competitor list with AI
* emit status events during variation generation so UI shows progress
* add onStatus callback to streaming enrichment
* show status spinner during variation generation, widen modal to 700px
* add tests for competitor enrichment helper functions
Add a new team member permission `/team/daily/activity` that allows
non-admin team members to see all team usage data, not just their own.
## Changes
- Add `TEAM_DAILY_ACTIVITY` to `KeyManagementRoutes` enum and available
team member permissions
- Add `_team_member_has_permission` helper in `common_utils.py`
- Modify `/team/daily/activity` endpoint to skip API key filtering when
the member has this permission
- Add permission description and method detection in the UI
- Add backend and frontend tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(ui/): initial commit adding a compliance testing playground
allow proxy admins to test policies and guardrails against datasets
* feat(ui/): make score more friendly
* feat(policy_endpoints.py): new helper function for testing policies
* feat(policy_endpoints.py): expose new endpoint for testing policies and guardrails
enables compliance playground to work as expected
* feat(complianceui.tsx): show returned text
