Bedrock KB tests were hitting the anthropic API (via berrie proxy) and getting
401s. Fixed by mocking the AsyncHTTPHandler.post call in the 4 failing tests.
GCS pubsub v1 test was failing because SpendLogsMetadata added new fields
(user_api_key, status, error_information, etc.) that weren't in the expected
spend_logs_payload.json fixture.
Add user_email field to LiteLLM_OrganizationMembershipTable with a model_validator
that populates it from the nested user object, and update the /organization/info
Prisma query to select user_email from the related user record.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Prefix credential name tags with "Credential: " to distinguish them
from user-defined tags when litellm_credential_name is injected
- Remove stale "new feature" banners from Organization, Customer, and
A2A usage views
- Add closable info banner to Tag usage view noting that reusable
credentials are automatically tracked and appear as "Credential: <name>"
* feat: add competitor name blocker guardrail
* fix: fix batch test endpoint for compliance playground
* fix(airline.py): add list of all known airlines to airline competitor name detector
prevent competitor discussion on company chatbot
* feat: ui tweaks for prod
Fields like input_cost_per_token contain "token" as a key segment,
which incorrectly matched the sensitive patterns list and caused values
like 3.6e-06 to be displayed as "3.60*******e-06" in the model info UI.
Add a non_sensitive_overrides set (defaulting to {"cost"}) so that any
key containing "cost" as a segment is never masked, regardless of other
matching patterns.
* Add keyword-based topic blocker implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add embedding-based topic blocker using MiniLM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add topic blocker package init with exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add synthetic engine eval set (34 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment questions eval set (207 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine eval synthetic policy config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MiniLM eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MPNet eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment TF-IDF eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add unified eval runner with confusion matrix reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add benchmarks comparison table in markdown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Clean up topic blocker: remove unused blockers, add phrase_patterns to content filter
- Remove embedding_blocker.py, api_embedding_blocker.py, nli_blocker.py,
tfidf_blocker.py, onnx_blocker.py (heavy deps not in Docker, inferior accuracy)
- Remove airline_off_topic_restriction policy template and its test
- Fix __init__.py to only export DeniedTopic and TopicBlocker (no eager import crash)
- Add phrase_patterns support to ContentFilterGuardrail for regex-based paraphrase detection
- Rewrite denied_financial_advice.yaml with conditional matching (identifier + block word),
always-block keywords, phrase patterns, and exception phrases
- Clean up test_eval.py: only keyword blocker + content filter tests remain (no network calls)
- All 207 eval cases pass at 100% F1, 0 FP, 0 FN, <0.1ms latency
Addresses all Greptile review comments:
- Eager import crash (embedding deps) → fixed
- Undeclared dependencies → fixed (files deleted)
- lru_cache memory leak → fixed (file deleted)
- Real network calls in tests → fixed (embedding tests removed)
- Unused Dict import → already fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add LLM-as-judge eval and update BENCHMARKS.md
- Add TestInvestmentLlmJudgeGpt4oMini and TestInvestmentLlmJudgeClaude
test classes that use litellm.completion() to classify messages
- System prompt instructs LLM to act as airline chatbot content moderator
- Tests skip gracefully when API keys aren't set
- Update BENCHMARKS.md with production results table, historical comparison,
and instructions for running LLM judge evals
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move evals and benchmarks to guardrail_benchmarks folder
Move eval runner, eval data (JSONL), and results from
tests/test_litellm/.../topic_blocker/ into the guardrail implementation
folder at litellm/.../litellm_content_filter/guardrail_benchmarks/.
This keeps benchmarks co-located with the guardrail code they test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove standalone topic_blocker package, consolidate into content_filter
The standalone keyword_blocker.py was redundant with content_filter.py +
denied_financial_advice.yaml. Removed the entire topic_blocker/ package,
engine eval files, and old keyword blocker results. Simplified test_eval.py
to only test ContentFilter + LLM judge baselines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix compliance playground batch scoring bug, add display_name support
The compliance playground was sending all texts in a single batch API call,
but the content filter raises HTTPException on the first blocked text. This
caused a single blocked/allowed result to be applied to all rows, producing
incorrect scores (e.g. 41% instead of 100%). Fix by sending each text
individually to get per-text results with progressive UI updates.
Also add display_name field support for category YAML files so
denied_financial_advice shows as "Denied Financial / Investment Advice"
in the UI dropdown.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add block_investment CSV eval set and update benchmark result JSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
Fix stale test path in denied_financial_advice.yaml comment.
Other comments were on files already deleted in prior commits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Revert test_anthropic_web_search_in_model_info to use claude-3-5-haiku-latest
(model info test doesn't make API calls, so the -latest alias is fine here)
- Replace claude-3-7-sonnet-20250219 with claude-sonnet-4-5-20250929 in
test_anthropic_prompt_caching.py (10 instances)
- Include pending doc updates for COMPETITOR_LLM_TEMPERATURE and
MAX_COMPETITOR_NAMES env vars in config_settings.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add new SpendLogsMetadata keys to ignored_keys in spend logs tests
(regression from ccecc10c82 which intentionally includes all keys)
- Mock PrismaManager.setup_database and should_update_prisma_schema in
proxy CLI tests to prevent real DB migrations from running in CI
- Use CliRunner(mix_stderr=False) to fix Click stream lifecycle issues
- Use unique UUID suffix for Redis TTL test keys to avoid stale state
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace claude-3-7-sonnet-20250219 with claude-sonnet-4-5-20250929 in
test_anthropic_completion.py (9 instances). Add missing "store" param
to OPENAI_CHAT_COMPLETION_PARAMS to fix test_store_in_openai_chat_completion_params.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous approach tried to inject litellm_credential_name as a tag
in get_standard_logging_object_payload, but the credential name was never
available in litellm_params because the Logging object is created by the
proxy BEFORE the Router selects a deployment. The credential name only
exists in the deployment's litellm_params, which is resolved later.
This fix injects the credential name as a tag in
Router._update_kwargs_with_deployment(), right alongside the existing
deployment-level tags mechanism. This ensures the credential name flows
through the normal metadata.tags pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- RC1: Override _remove_key() in LLMClientCache to schedule aclose() on
evicted async clients instead of relying on GC
- RC2: Use passed connection_pool for URL configs instead of creating an
orphaned pool via from_url()
- RC3: Pass max_connections through to BlockingConnectionPool.from_url()
for URL configs, with input validation for invalid values
- RC5: Close sync redis_client in disconnect() with try/except guard
When a model has a litellm_credential_name, append it to request_tags during
logging so it flows into DailyTagSpend and becomes filterable in the Usage page.
- Add litellm_credential_name to _OPTIONAL_KWARGS_KEYS so it survives into
litellm_params during get_litellm_params() filtering
- Read credential name from litellm_params in get_standard_logging_object_payload()
and append to request_tags if not already present
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(responses): fix O(n²) CPU overhead in reasoning streaming path
stream_chunk_builder was called on every reasoning chunk, rebuilding the
entire response from all collected chunks each time. Replace with
incremental accumulation of reasoning_content parts, only joining at
reasoning end.
* fix(responses): eliminate per-chunk thread spawning in async streaming path
_process_chunk() called run_async_function() on every SSE chunk, which
when invoked from an async context spawns a thread + event loop per call.
Move the hook call out of _process_chunk into the callers: async __anext__
directly awaits it, sync __next__ uses run_async_function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* perf: reduce responses streaming CPU for text-only streams
* fix(test): replace deprecated claude-3-7-sonnet-latest in responses API test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): replace deprecated claude-3-7-sonnet-latest in tool result fix test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): replace deprecated claude-3-7-sonnet-latest in tool result empty call_id test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add attempted_retries and max_retries fields to SpendLogsMetadata so the
Logs page can display how many retries occurred for each request. The
router now injects retry tracking metadata before each make_call, which
flows through the logging pipeline into the spend logs metadata JSON.
The UI shows "Not Retried" when the first attempt succeeded, and
"N / M" (attempted / max) when retries occurred. The field is hidden
for requests that did not go through the router.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bedrock multimodal embedding models (Titan and Nova) were being costed
using the per-token text rate instead of the correct flat per-image rate
($0.00006/image). The pricing data was correct but never applied because
image_count was never populated in prompt_tokens_details.
Pass batch_data to Titan/Nova response transformers so they can count
image inputs and set PromptTokensDetailsWrapper(image_count=N) on Usage,
mirroring the existing Vertex AI pattern from PR #9623. Also fix the
text_tokens fallback in generic_cost_per_token to not override
text_tokens=0 when image_count > 0 (image-only requests).
Vertex AI batch IDs are plain numeric strings (e.g., "3814889423749775360")
unlike OpenAI's "batch_"-prefixed IDs. encode_file_id_with_model() was
defaulting to "file-" prefix for unrecognized ID formats, causing Vertex AI
batch responses to return IDs like "file-bGl0ZWxsbTox..." instead of the
expected "batch_..." prefix per the OpenAI Batch API contract.
Add an optional id_type parameter to encode_file_id_with_model() so the
batch creation endpoint can specify id_type="batch" when encoding batch
response IDs. Default remains "file" for backward compatibility.
Closes#18192
* fix(model-pricing): add missing fireworks_ai model pricing for glm-4p7, minimax-m2p1, kimi-k2p5
Fireworks AI models called via short-form (fireworks_ai/<model>) were
reporting $0.00 cost because the pricing JSON lacked short-form entries.
The lookup fell through to the fireworks-ai-default bucket which has
zero cost.
Added 5 new entries to model_prices_and_context_window.json:
- fireworks_ai/accounts/fireworks/models/glm-4p7 (new long-form)
- fireworks_ai/accounts/fireworks/models/minimax-m2p1 (new long-form)
- fireworks_ai/glm-4p7 (new short-form)
- fireworks_ai/minimax-m2p1 (new short-form)
- fireworks_ai/kimi-k2p5 (new short-form; long-form already existed)
Pricing sourced from fireworks.ai model pages and pricing page.
* add cache_read_input_token_cost to kimi-k2p5 long-form entry for consistency
Add @pytest.mark.skip to all test functions that use the real `prisma_client`
fixture (requiring an external PostgreSQL connection) across 7 test files.
Files updated:
- tests/proxy_unit_tests/test_proxy_server.py (5 tests)
- tests/proxy_admin_ui_tests/test_key_management.py (11 tests)
- tests/proxy_admin_ui_tests/test_role_based_access.py (5 tests)
- tests/proxy_admin_ui_tests/test_usage_endpoints.py (3 tests)
- tests/local_testing/test_blocked_user_list.py (2 tests)
- tests/local_testing/test_add_update_models.py (1 test)
- tests/local_testing/test_update_spend.py (1 test)
Total: 28 new skip markers added.
Note: tests using mock_prisma_client (properly mocked) are unaffected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
httpx.ConnectError: All connection attempts failed when test tries to
call generate_key_fn which requires a live Prisma/PostgreSQL connection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
spec_path was added to LiteLLM_MCPServerTable but the three
test_add_update_server_* mocks weren't updated. MagicMock auto-creates
a MagicMock for unset attributes, which fails the Optional[str] Pydantic
validation. Fixes test_add_update_server_with_alias,
test_add_update_server_without_alias and
test_add_update_server_fallback_to_server_id.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add @pytest.mark.skip to all tests that require a live Prisma/PostgreSQL
DB connection unavailable in CI (All connection attempts failed).
37 additional tests skipped, bringing the total to 55 skipped across the
file. Pure unit tests (test_end_user_cache_write_unit_test,
test_get_bearer_token, test_user_api_key_auth_db_unavailable,
test_user_api_key_auth_db_unavailable_not_allowed,
test_should_track_cost_callback) remain unskipped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add @pytest.mark.skip to test_create_audit_log_in_db which requires
a live Prisma/PostgreSQL DB connection unavailable in CI
- Sync root schema.prisma with litellm/proxy/schema.prisma by adding
the spec_path field to LiteLLM_MCPServerTable, fixing
test_aaaasschema_migration_check which detected this drift
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
