* fix(test): remove deprecated Click mix_stderr param in test_use_prisma_db_push_flag_behavior
Click 8.2+ removed the mix_stderr parameter from CliRunner. Use CliRunner() without it.
* fix(test): use app.dependency_overrides for auth mock in test_role_mappings_stored_and_retrieved
monkeypatch.setattr doesn't affect FastAPI's Depends() resolution in parallel
test execution. Use app.dependency_overrides which is the proper FastAPI pattern.
Any PrismaError should be treated as a DB connection error for the
allow_requests_on_db_unavailable feature and 503 responses. The narrow
keyword-based check is now in is_database_transport_error, which is
what the reconnect logic in auth_checks.py should use.
Fixes test_delete_access_group_503_on_db_connection_error and
test_handle_authentication_error_db_unavailable failures caused by
PR #21706 narrowing is_database_connection_error.
- Add early return in _get_assembled_streaming_response for non-streaming
requests, preventing duplicate standard_logging_object computation
- Move emit_standard_logging_payload into _process_hidden_params_and_response_cost
so non-streaming requests still emit the debug payload
- Add emit_standard_logging_payload for dict/list result edge cases
* fix(tests): isolate flaky files endpoint tests from global proxy state
* test(secret_managers): add mocked unit test for write/read JSON secret cycle
* fix(tests): restore litellm.callbacks in TestSpendLogsPayload setup/teardown
* fix(tests): clear app.openapi_schema in TestSwaggerChatCompletions setup/teardown
* fix(tests): add flaky marker to test_async_increment_tokens_with_ttl_preservation
* fix(tests): isolate flaky files endpoint tests from global proxy state
* test(secret_managers): add mocked unit test for write/read JSON secret cycle
Clients like OpenCode's @ai-sdk/openai-compatible send budgetTokens
(camelCase) instead of budget_tokens in the thinking parameter, causing
validation errors. Add early normalization in completion().
- Restore preserve_fields param for backward compatibility (deprecated)
- Use zip() instead of index-based iteration to prevent IndexError
- Add backward compatibility test
* fix: reset db_health_cache in source module to prevent stale cache hits
The test was reassigning db_health_cache via `global` in the test module,
which doesn't affect the _health_endpoints module's variable. When a prior
test set the cache to "connected" within 2 minutes, _db_health_readiness_check
returned early without calling health_check(), causing assert_called_once to fail.
Also use PrismaError with a connection message so it's properly recognized
as a connection error by PrismaDBExceptionHandler.is_database_connection_error.
* fix: replace asyncio.sleep with polling loop in spend logs tests
The GLOBAL_LOGGING_WORKER processes callbacks via an async queue, so
asyncio.sleep(1) is a race condition - under CI load the worker may not
have processed the queued task within 1 second. Replace with a polling
helper that waits up to 10 seconds for the mock to be called.
Also add metadata.attempted_retries and metadata.max_retries to
ignored_keys since these are new fields.
* fix: isolate test_skip_server_startup from CI environment
Remove mix_stderr=False (unsupported in some Click versions). Strip
DATABASE_URL/DIRECT_URL from environment during the test to prevent
real prisma operations when these are set in CI.
Add vector_size parameter to QdrantSemanticCache and expose it through
the Cache facade as qdrant_semantic_cache_vector_size. This allows users
to use embedding models with dimensions other than the default 1536,
enabling cheaper/stronger models like Stella (1024d), bge-en-icl (4096d),
voyage, cohere, etc.
The parameter defaults to QDRANT_VECTOR_SIZE (env var or 1536) for
backward compatibility. When creating new collections, the configured
vector_size is used instead of the hardcoded constant.
Closes#9377
POST /access_group/new and PUT /access_group/{name}/update now accept an
optional model_ids list that targets specific deployments by their unique
model_id, instead of tagging every deployment that shares a model_name.
When model_ids is provided it takes priority over model_names, giving
API callers the same single-deployment precision that the UI already has
via PATCH /model/{model_id}/update.
Backward compatible: model_names continues to work as before.
Closes#21544
Add user_email field to LiteLLM_OrganizationMembershipTable with a model_validator
that populates it from the nested user object, and update the /organization/info
Prisma query to select user_email from the related user record.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Prefix credential name tags with "Credential: " to distinguish them
from user-defined tags when litellm_credential_name is injected
- Remove stale "new feature" banners from Organization, Customer, and
A2A usage views
- Add closable info banner to Tag usage view noting that reusable
credentials are automatically tracked and appear as "Credential: <name>"
* feat: add competitor name blocker guardrail
* fix: fix batch test endpoint for compliance playground
* fix(airline.py): add list of all known airlines to airline competitor name detector
prevent competitor discussion on company chatbot
* feat: ui tweaks for prod
Fields like input_cost_per_token contain "token" as a key segment,
which incorrectly matched the sensitive patterns list and caused values
like 3.6e-06 to be displayed as "3.60*******e-06" in the model info UI.
Add a non_sensitive_overrides set (defaulting to {"cost"}) so that any
key containing "cost" as a segment is never masked, regardless of other
matching patterns.
* Add keyword-based topic blocker implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add embedding-based topic blocker using MiniLM
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add topic blocker package init with exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add synthetic engine eval set (34 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment questions eval set (207 cases)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine eval synthetic policy config
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add engine keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment keyword blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding blocker eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MiniLM eval results
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment embedding MPNet eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add investment TF-IDF eval results (historical)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add unified eval runner with confusion matrix reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add benchmarks comparison table in markdown
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Clean up topic blocker: remove unused blockers, add phrase_patterns to content filter
- Remove embedding_blocker.py, api_embedding_blocker.py, nli_blocker.py,
tfidf_blocker.py, onnx_blocker.py (heavy deps not in Docker, inferior accuracy)
- Remove airline_off_topic_restriction policy template and its test
- Fix __init__.py to only export DeniedTopic and TopicBlocker (no eager import crash)
- Add phrase_patterns support to ContentFilterGuardrail for regex-based paraphrase detection
- Rewrite denied_financial_advice.yaml with conditional matching (identifier + block word),
always-block keywords, phrase patterns, and exception phrases
- Clean up test_eval.py: only keyword blocker + content filter tests remain (no network calls)
- All 207 eval cases pass at 100% F1, 0 FP, 0 FN, <0.1ms latency
Addresses all Greptile review comments:
- Eager import crash (embedding deps) → fixed
- Undeclared dependencies → fixed (files deleted)
- lru_cache memory leak → fixed (file deleted)
- Real network calls in tests → fixed (embedding tests removed)
- Unused Dict import → already fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add LLM-as-judge eval and update BENCHMARKS.md
- Add TestInvestmentLlmJudgeGpt4oMini and TestInvestmentLlmJudgeClaude
test classes that use litellm.completion() to classify messages
- System prompt instructs LLM to act as airline chatbot content moderator
- Tests skip gracefully when API keys aren't set
- Update BENCHMARKS.md with production results table, historical comparison,
and instructions for running LLM judge evals
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Move evals and benchmarks to guardrail_benchmarks folder
Move eval runner, eval data (JSONL), and results from
tests/test_litellm/.../topic_blocker/ into the guardrail implementation
folder at litellm/.../litellm_content_filter/guardrail_benchmarks/.
This keeps benchmarks co-located with the guardrail code they test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove standalone topic_blocker package, consolidate into content_filter
The standalone keyword_blocker.py was redundant with content_filter.py +
denied_financial_advice.yaml. Removed the entire topic_blocker/ package,
engine eval files, and old keyword blocker results. Simplified test_eval.py
to only test ContentFilter + LLM judge baselines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix compliance playground batch scoring bug, add display_name support
The compliance playground was sending all texts in a single batch API call,
but the content filter raises HTTPException on the first blocked text. This
caused a single blocked/allowed result to be applied to all rows, producing
incorrect scores (e.g. 41% instead of 100%). Fix by sending each text
individually to get per-text results with progressive UI updates.
Also add display_name field support for category YAML files so
denied_financial_advice shows as "Denied Financial / Investment Advice"
in the UI dropdown.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add block_investment CSV eval set and update benchmark result JSON
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* address greptile review feedback (greploop iteration 1)
Fix stale test path in denied_financial_advice.yaml comment.
Other comments were on files already deleted in prior commits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Add new SpendLogsMetadata keys to ignored_keys in spend logs tests
(regression from ccecc10c82 which intentionally includes all keys)
- Mock PrismaManager.setup_database and should_update_prisma_schema in
proxy CLI tests to prevent real DB migrations from running in CI
- Use CliRunner(mix_stderr=False) to fix Click stream lifecycle issues
- Use unique UUID suffix for Redis TTL test keys to avoid stale state
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous approach tried to inject litellm_credential_name as a tag
in get_standard_logging_object_payload, but the credential name was never
available in litellm_params because the Logging object is created by the
proxy BEFORE the Router selects a deployment. The credential name only
exists in the deployment's litellm_params, which is resolved later.
This fix injects the credential name as a tag in
Router._update_kwargs_with_deployment(), right alongside the existing
deployment-level tags mechanism. This ensures the credential name flows
through the normal metadata.tags pipeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- RC1: Override _remove_key() in LLMClientCache to schedule aclose() on
evicted async clients instead of relying on GC
- RC2: Use passed connection_pool for URL configs instead of creating an
orphaned pool via from_url()
- RC3: Pass max_connections through to BlockingConnectionPool.from_url()
for URL configs, with input validation for invalid values
- RC5: Close sync redis_client in disconnect() with try/except guard
When a model has a litellm_credential_name, append it to request_tags during
logging so it flows into DailyTagSpend and becomes filterable in the Usage page.
- Add litellm_credential_name to _OPTIONAL_KWARGS_KEYS so it survives into
litellm_params during get_litellm_params() filtering
- Read credential name from litellm_params in get_standard_logging_object_payload()
and append to request_tags if not already present
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(responses): fix O(n²) CPU overhead in reasoning streaming path
stream_chunk_builder was called on every reasoning chunk, rebuilding the
entire response from all collected chunks each time. Replace with
incremental accumulation of reasoning_content parts, only joining at
reasoning end.
* fix(responses): eliminate per-chunk thread spawning in async streaming path
_process_chunk() called run_async_function() on every SSE chunk, which
when invoked from an async context spawns a thread + event loop per call.
Move the hook call out of _process_chunk into the callers: async __anext__
directly awaits it, sync __next__ uses run_async_function.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* perf: reduce responses streaming CPU for text-only streams
* fix(test): replace deprecated claude-3-7-sonnet-latest in responses API test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): replace deprecated claude-3-7-sonnet-latest in tool result fix test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): replace deprecated claude-3-7-sonnet-latest in tool result empty call_id test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add attempted_retries and max_retries fields to SpendLogsMetadata so the
Logs page can display how many retries occurred for each request. The
router now injects retry tracking metadata before each make_call, which
flows through the logging pipeline into the spend logs metadata JSON.
The UI shows "Not Retried" when the first attempt succeeded, and
"N / M" (attempted / max) when retries occurred. The field is hidden
for requests that did not go through the router.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ModelResponse.choices was typed as List[Union[Choices, StreamingChoices]] which
caused Pydantic serialization warnings and false linting errors. Now that
ModelResponseStream exists for streaming, narrow ModelResponse.choices to
List[Choices] and migrate all ModelResponse(stream=True) call sites to use
ModelResponseStream() instead.
Bedrock multimodal embedding models (Titan and Nova) were being costed
using the per-token text rate instead of the correct flat per-image rate
($0.00006/image). The pricing data was correct but never applied because
image_count was never populated in prompt_tokens_details.
Pass batch_data to Titan/Nova response transformers so they can count
image inputs and set PromptTokensDetailsWrapper(image_count=N) on Usage,
mirroring the existing Vertex AI pattern from PR #9623. Also fix the
text_tokens fallback in generic_cost_per_token to not override
text_tokens=0 when image_count > 0 (image-only requests).
* fix(model-pricing): add missing fireworks_ai model pricing for glm-4p7, minimax-m2p1, kimi-k2p5
Fireworks AI models called via short-form (fireworks_ai/<model>) were
reporting $0.00 cost because the pricing JSON lacked short-form entries.
The lookup fell through to the fireworks-ai-default bucket which has
zero cost.
Added 5 new entries to model_prices_and_context_window.json:
- fireworks_ai/accounts/fireworks/models/glm-4p7 (new long-form)
- fireworks_ai/accounts/fireworks/models/minimax-m2p1 (new long-form)
- fireworks_ai/glm-4p7 (new short-form)
- fireworks_ai/minimax-m2p1 (new short-form)
- fireworks_ai/kimi-k2p5 (new short-form; long-form already existed)
Pricing sourced from fireworks.ai model pages and pricing page.
* add cache_read_input_token_cost to kimi-k2p5 long-form entry for consistency
For gemini-3-pro-preview (not gemini-3-flash or gemini-3.1-pro-preview),
reasoning_effort="medium" maps to thinkingLevel="high" because the "medium"
thinking level is not available on that model variant.
Both test_reasoning_effort_maps_to_thinking_level_gemini_3 and
test_reasoning_effort_dict_format_gemini_3 had the correct comment
("medium -> high") but the wrong assertion (== "medium"). Fixed to
match the production code and the comments.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The production error message was expanded when Sonnet 4.6 was also added as
a supported model for effort='max'. The test's match regex still referenced
the old "Claude Opus 4.6"-only message; update it to match the new
"Claude 4.6 models" wording.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
