Yuneng Jiang
6796fa0668
chore: fixes
2026-04-04 23:41:58 -07:00
Ishaan Jaffer
18cdf567df
fix(logging): zero out response_cost for cache hits in _process_hidden_params_and_response_cost
...
When a cached response had `response_cost` pre-calculated in its `_hidden_params`
(from the original response), `_process_hidden_params_and_response_cost` was using
it directly without checking `cache_hit`, so cached responses were reported with
non-zero cost instead of 0.
Fixes test_cost_tracking_with_caching.
2026-02-21 14:51:00 -08:00
Shivam Rawat
698c46830b
remove prompt from getting logged ( #21831 )
2026-02-21 14:49:11 -08:00
yuneng-jiang
3476452774
Merge pull request #21822 from dkindlund/fix/admin-ui-logging-metadata
...
fix(ui): preserve logging_settings in key metadata on update
2026-02-21 14:46:41 -08:00
Ishaan Jaff
8afeaf8da4
fix(tests): fix flaky test_create_vertex_fine_tune_jobs_mocked - handle background Datadog flush ( #21838 )
2026-02-21 14:44:01 -08:00
Ishaan Jaff
d7b22d340b
fix(tests): move test_router_azure_acompletion to llm_translation testing ( #21837 )
2026-02-21 14:41:53 -08:00
Ishaan Jaff
59e5b7e8c6
fix(tests): use monkeypatch.setenv for Redis pool max_connections tests ( #21834 )
...
Replace patch('litellm._redis._get_redis_client_logic') with monkeypatch.setenv
in test_max_connections_url_config and test_max_connections_url_config_string_value.
The mock was unreliable in CI (REDIS_URL is set to the real Redis Cloud server),
causing the pool to silently use the real config instead of the test config.
Using monkeypatch.setenv tests the full env-var→pool chain more robustly and
matches the actual production code path.
2026-02-21 14:38:28 -08:00
Ishaan Jaff
235a47c576
fix(tests): mock test_claude_tool_use_with_gemini to fix flaky CI ( #21832 )
...
* ui fixes
* fix(tests): mock test_claude_tool_use_with_gemini to avoid MALFORMED_FUNCTION_CALL flakiness
2026-02-21 14:34:54 -08:00
ryan-crabbe
3da77f310b
Merge pull request #21828 from BerriAI/fix-duplicate-issues-workflow
...
fix: pass prompt as env var in duplicate detection workflows
2026-02-21 14:30:03 -08:00
Ryan Crabbe
c7d3198d9a
fix: pass prompt as env var in duplicate detection workflows
...
Fixes "Input must be provided either through stdin or as a prompt
argument" error by moving the prompt to a PROMPT env variable
instead of inline multiline shell string.
2026-02-21 14:29:08 -08:00
Ishaan Jaff
fb4249005e
fix(tests): add atexit.register mock to prevent Click isolation stream closure in test_use_prisma_db_push_flag_behavior ( #21829 )
2026-02-21 14:28:02 -08:00
yuneng-jiang
f61dc66a02
Merge pull request #21827 from BerriAI/revert-21707-add-watchdog-prisma
...
Revert "fix(proxy): recover from prisma-query-engine zombie process"
2026-02-21 14:21:19 -08:00
yuneng-jiang
8c5be4cb62
Revert "fix(proxy): recover from prisma-query-engine zombie process ( #21707 )"
...
This reverts commit 977ad015ca .
2026-02-21 14:20:06 -08:00
Ishaan Jaff
56a095a079
fix(tests): update deprecated Anthropic model in test_user_model_access ( #21826 )
...
* fix(tests): clear _async_success_callback in vertex fine-tune mocked tests to prevent Datadog interference
* fix(tests): update deprecated claude-3-5-haiku-20241022 to claude-haiku-4-5-20251001
2026-02-21 14:18:24 -08:00
Ishaan Jaff
a1ead765ed
fix(tests): clear _async_success_callback in vertex fine-tune mocked tests to prevent Datadog interference ( #21825 )
2026-02-21 14:16:18 -08:00
Ishaan Jaff
494aad4a68
fix(tests): isolate auth in vertex passthrough and spend logs date range tests ( #21824 )
...
test_vertex_passthrough_with_default_credentials and
test_view_spend_logs_with_date_range_summarized fail intermittently when a
prior xdist worker sets master_key — auth then rejects the unauthenticated
test requests before the code under test is reached.
- mock user_api_key_auth in test_vertex_passthrough_with_default_credentials
(same pattern used for test_vertex_passthrough_with_no_default_credentials
in #21810 )
- wrap test_view_spend_logs_with_date_range_summarized in
app.dependency_overrides[ps.user_api_key_auth] with try/finally cleanup
(same pattern used for the other spend log tests in #21810 )
2026-02-21 14:14:50 -08:00
Ishaan Jaff
dd6a74da63
fix(tests): isolate litellm.cache and CLI env vars in flaky tests ( #21821 )
...
- TestSpendLogsPayload: save/restore litellm.cache in setup_method/teardown_method
so tests that run after a cache-setting test don't see a non-None cache and get
a hash instead of "Cache OFF" in the cache_key field
- test_use_prisma_db_push_flag_behavior: apply clean_env pattern (strip DATABASE_URL/DIRECT_URL,
then set DATABASE_URL to test value) inside the with block instead of using @patch.dict
decorator, matching the pattern from test_skip_server_startup to avoid Click 8.3.x
StreamMixer stream lifecycle issues in CI
2026-02-21 14:11:48 -08:00
Ishaan Jaff
6acfa1c71d
fix(tests): clear ANTHROPIC_BASE_URL/ANTHROPIC_API_BASE in spend log api_base tests ( #21820 )
...
Tests hardcode expected api_base as https://api.anthropic.com/v1/messages but
if ANTHROPIC_BASE_URL is set in the environment the recorded api_base changes,
causing a mismatch. Clear both env vars via monkeypatch at the start of each test.
2026-02-21 14:11:18 -08:00
Ishaan Jaff
6ec16d583d
fix(test): add timeout to flush() call to prevent 300s hang in CI ( #21819 )
...
GLOBAL_LOGGING_WORKER.flush() calls queue.join() which blocks until all
items are task_done(). In CI with pytest-asyncio, each test gets a fresh
event loop so the worker reinitializes its queue - items from a previous
test never get task_done(), causing an infinite hang.
Fix: wrap flush() with asyncio.wait_for(..., timeout=10.0).
2026-02-21 14:10:57 -08:00
Darien Kindlund
5f0bef3133
fix(ui): preserve logging_settings in key metadata on update
...
The logging_settings condition used a bare truthiness check which
failed when the form field was undefined or not properly synced
from the EditLoggingSettings component. Changed to explicit
Array.isArray() check consistent with the tags field pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-02-21 16:51:56 -05:00
ryan-crabbe
b17d37eceb
Merge pull request #21815 from BerriAI/litellm_fix_openai_init_params_immutable
...
fix: make cached OpenAI init params immutable and fix import ordering
2026-02-21 13:33:03 -08:00
Ryan Crabbe
dcbac4a4af
style: add missing PEP 8 blank line before top-level function
2026-02-21 13:31:19 -08:00
Ishaan Jaff
c810f5cd63
fix(tests): replace fake France Azure endpoint in test_router_azure_acompletion ( #21818 )
2026-02-21 13:26:59 -08:00
Ryan Crabbe
9e1d83e3de
fix: make LITELLM_CLIENT_SPECIFIC_PARAMS a tuple to prevent TypeError
...
tuple + list raises TypeError in get_openai_client_cache_key. Also add
test coverage for get_openai_client_cache_key to catch type mismatches.
2026-02-21 13:25:23 -08:00
shin-bot-litellm
1be30f5129
feat(router): Add complexity-based auto routing strategy ( #21789 )
...
* feat(router): Add complexity-based auto routing strategy
Adds a rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - with zero API calls and sub-millisecond
latency.
## Features
- **Zero external API calls** - all scoring is local
- **Sub-millisecond latency** - typically <1ms per classification
- **Weighted multi-dimensional scoring** across 7 dimensions:
- Token count (short=simple, long=complex)
- Code presence (code keywords → complex)
- Reasoning markers ("step by step" → reasoning tier)
- Technical terms (domain complexity)
- Simple indicators ("what is" → simple, negative weight)
- Multi-step patterns (numbered steps)
- Question complexity (multiple questions)
- **Configurable tier boundaries** and model mappings
- **Reasoning override** - 2+ reasoning markers force REASONING tier
## Usage
```yaml
model_list:
- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gpt-4o-mini
MEDIUM: gpt-4o
COMPLEX: claude-sonnet-4
REASONING: o1-preview
```
Inspired by ClawRouter: https://github.com/BlockRunAI/ClawRouter
## Files Added
- litellm/router_strategy/complexity_router/complexity_router.py - Main router class
- litellm/router_strategy/complexity_router/config.py - Configuration and defaults
- litellm/router_strategy/complexity_router/__init__.py - Package exports
- litellm/router_strategy/complexity_router/README.md - Documentation
- tests/test_litellm/router_strategy/test_complexity_router.py - Test suite (37 tests)
## Files Modified
- litellm/router.py - Integration with pre_routing_hook
- litellm/types/router.py - New config params
* feat(router): Add complexity-based auto routing strategy
Adds a new rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - without any external API calls.
## Features
- Weighted scoring across 7 dimensions: token count, code presence, reasoning
markers, technical terms, simple indicators, multi-step patterns, questions
- Maps to 4 tiers: SIMPLE, MEDIUM, COMPLEX, REASONING
- Each tier configurable to a different model
- Zero API calls, <1ms latency
- Inspired by ClawRouter
## Configuration
```yaml
model_list:
- model_name: smart_router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gemini-2.0-flash
MEDIUM: gpt-4o-mini
COMPLEX: claude-sonnet-4
REASONING: claude-opus-4
```
## Use Cases
- Cost optimization: route simple queries to cheaper models
- Quality optimization: route complex queries to capable models
- Zero configuration: works out of the box with sensible defaults
* feat(router): Add complexity-based auto routing strategy
Adds a new rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - without any external API calls.
- Weighted scoring across 7 dimensions: token count, code presence, reasoning
markers, technical terms, simple indicators, multi-step patterns, questions
- Maps to 4 tiers: SIMPLE, MEDIUM, COMPLEX, REASONING
- Each tier configurable to a different model
- Zero API calls, <1ms latency
- Inspired by ClawRouter
```yaml
model_list:
- model_name: smart_router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gemini-2.0-flash
MEDIUM: gpt-4o-mini
COMPLEX: claude-sonnet-4
REASONING: claude-opus-4
```
- Cost optimization: route simple queries to cheaper models
- Quality optimization: route complex queries to capable models
- Zero configuration: works out of the box with sensible defaults
* feat: add enterprise presets for complexity router
Adds preset configurations for different cloud providers:
- bedrock: AWS Bedrock (Claude models)
- vertex: Google Vertex AI (Gemini models)
- azure: Azure OpenAI (GPT + o1)
- standard: Direct API (OpenAI + Anthropic)
- cost_optimized: Maximum savings (Gemini Flash + cheaper models)
Usage:
```yaml
complexity_router_config:
preset: bedrock # or vertex, azure, standard, cost_optimized
```
* feat(ui): update auto router submit handler for complexity router
- Handle complexity_router model type in submit handler
- Generate correct litellm_params for complexity router:
- model: auto_router/complexity_router
- complexity_router_config: { tiers: { SIMPLE, MEDIUM, COMPLEX, REASONING } }
- Keep existing semantic router handling intact
- Add success notification with router type name
* docs: update PR description with UI changes
* chore: remove preset feature, keep simple tier config
* fix: exclude complexity_router from auto_router check
The _is_auto_router_deployment() was matching all auto_router/* models,
causing complexity_router to fail initialization. Now it explicitly
excludes auto_router/complexity_router which has its own handler.
* fix(complexity_router): Address Greptile review feedback
Fixes 5 issues flagged in code review:
1. **Mutable singleton mutation bug** - Now always creates a new
ComplexityRouterConfig instance instead of reusing DEFAULT_COMPLEXITY_CONFIG
singleton, preventing cross-instance config pollution.
2. **Substring matching false positives** - Added word boundaries (spaces)
to short keywords like 'ok', 'try', 'api', 'git', 'node', 'java', 'vue'
to prevent matching within longer words (e.g., 'capital' matching 'api').
3. **Redundant message extraction** - Simplified to single reverse loop that
extracts both last user message and last system prompt efficiently.
4. **Unused imports** - Removed unused DEFAULT_CREATIVE_KEYWORDS and
DEFAULT_MULTI_STEP_PATTERNS imports.
5. **Missing async_pre_routing_hook tests** - Added comprehensive tests for:
- Multi-turn conversations
- List-type content handling
- No user message case
- Empty string content
- Message preservation
- Singleton mutation prevention
* fix(complexity_router): Address Greptile review feedback
- Use word boundary matching for short keywords (<5 chars) to avoid
false positives (e.g., 'api' matching 'capital', 'git' matching 'digital')
- Remove 'ok' from simple keywords (too many false positives)
- Add tests for keyword false positive prevention
- Fix test expectations for edge cases (empty string content, list content)
Addresses: 2/5 Greptile score feedback on PR #21789
* docs(auto_routing): Add complexity router documentation
- Add Complexity Router section to auto_routing.md
- Include comparison table with semantic auto router
- Add Python SDK and Proxy Server configuration examples
- Document all configuration options (tier boundaries, token thresholds, dimension weights)
- Explain how complexity scoring works
* feat(complexity_router): Add eval suite + tune scoring parameters
Added comprehensive evaluation suite with 29 test cases covering:
- SIMPLE tier: greetings, definitions, factual questions
- MEDIUM tier: technical explanations, comparisons, debugging
- COMPLEX tier: architecture design, complex coding
- REASONING tier: explicit reasoning requests
- Regression tests: substring false positive prevention
Tuned scoring parameters based on eval results:
- Lowered tier boundaries (0.15/0.35/0.60) for better tier distribution
- Increased code/technical weights (0.30/0.25) for complex prompts
- Reduced simple indicator weight (0.05) to avoid over-penalizing
- Fixed 'hey'/'hi' keywords to require leading space
Eval results: 29/29 passed (100%)
* fix(complexity_router): Address Greptile review round 2
1. **Empty user message handling** - Changed from falsy check to None check
to properly distinguish 'no user message' from 'empty string message'
2. **ReDoS prevention** - Changed 'first.*then' to 'first.*?then' (non-greedy)
to prevent regex backtracking on pathological inputs
3. **Documentation sync** - Updated README.md to match actual config values:
- Tier boundaries: 0.15/0.35/0.60 (not 0.25/0.50/0.75)
- Dimension weights: tokenCount=0.10, codePresence=0.30, technicalTerms=0.25,
simpleIndicators=0.05, multiStepPatterns=0.03, questionComplexity=0.02
4. **Missing UI component** - Added ComplexityRouterConfig.tsx with:
- Tier-to-model dropdown selectors
- Descriptions and examples for each tier
- How classification works explanation
5. **Inline import comment** - Added explanation for why ComplexityRouter
import is inline (matches AutoRouter pattern, avoids circular imports)
* docs(auto_routing): fix dimension weights and tier boundaries to match config.py defaults
* fix(complexity_router): skip empty string content in async_pre_routing_hook
* fix(router): remove or {} masking None complexity_router_config
* fix(config): remove unused DEFAULT_MULTI_STEP_PATTERNS and DEFAULT_CREATIVE_KEYWORDS exports
* fix(complexity_router): use word boundary matching for all single-word keywords, avoid double-scanning reasoning keywords
* fix(router): clarify circular import comment for ComplexityRouter
* docs(README): fix token thresholds to match config.py defaults
* test(complexity_router): add false positive tests for error/class/merge keyword matching
* fix(complexity_router): align .get() fallbacks with config.py defaults, document system prompt scoring
* fix(config): deduplicate keywords across code and technical lists
---------
Co-authored-by: OpenClaw Assistant <assistant@openclaw.ai>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
2026-02-21 13:23:37 -08:00
Ryan Crabbe
bbbec23c8b
fix: update tests to match tuple return type for cached init params
2026-02-21 13:18:03 -08:00
Ishaan Jaff
886d168154
fix(logging): resolve cache_hit before hidden_params short-circuit in _response_cost_calculator ( #21816 )
...
Cached responses carry response_cost in _hidden_params from the original call.
_response_cost_calculator was returning that pre-computed cost before checking
cache_hit, so cached responses were billed instead of returning 0.0.
Fix: move cache_hit resolution and early-return to top of the function.
Regression introduced in bdf01fa283 (fix mypy error).
2026-02-21 13:17:02 -08:00
Ryan Crabbe
72e75c1122
Merge origin/main into litellm_fix_openai_init_params_immutable
...
Resolve conflict: keep Tuple types, incorporate type: ignore comment
and alphabetical typing import order from main.
2026-02-21 13:15:30 -08:00
Ryan Crabbe
20a685fe7f
fix: make cached OpenAI init params immutable and fix import ordering
...
- Move `import inspect` to stdlib import group
- Change _OPENAI_INIT_PARAMS and _AZURE_OPENAI_INIT_PARAMS from
mutable lists to immutable tuples to prevent accidental mutation
- Update return type and helper to use Tuple[str, ...]
2026-02-21 13:10:49 -08:00
Ishaan Jaff
8483477512
fix(test): add asyncio.sleep(0) before flush() to prevent hang in test_async_no_duplicate_spend_logs ( #21813 )
2026-02-21 13:09:36 -08:00
Ishaan Jaff
0a0768b3df
fix(ci): resolve mypy and check_code_and_doc_quality CI failures ( #21812 )
...
- fix(mypy): suppress [misc] type error in common_utils.py for cls.__init__ access
- fix(mypy): move type: ignore comment to correct line in test_eval.py (line 232 not 231)
- fix(mypy): suppress [misc] and pre-existing pyright errors in vertex_ai_non_gemini.py
- fix(check_licenses): strip inline comments before parsing requirements.txt lines so CVE comments don't break packaging.requirements.Requirement()
- fix(router_coverage): add _merge_tools_from_deployment and _invalidate_access_groups_cache to ignored list (private helpers tested indirectly)
2026-02-21 13:08:47 -08:00
github-actions[bot]
22704b0176
chore: regenerate poetry.lock to match pyproject.toml ( #21811 )
...
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-02-21 21:07:29 +00:00
ryan-crabbe
336ecd6267
Merge pull request #21003 from BerriAI/litellm_perf_skip_usage_roundtrip
...
perf: skip Usage Pydantic round-trip in logging payload
2026-02-21 12:56:33 -08:00
Ryan Crabbe
75bc8329e2
Merge origin/main into litellm_perf_skip_usage_roundtrip
...
Resolve conflict in litellm_logging.py: take main's version and
re-apply get_usage_as_dict optimization on top.
2026-02-21 12:55:55 -08:00
Ishaan Jaff
b281181448
fix(tests): isolate auth in spend logs and vertex passthrough tests ( #21810 )
...
* fix(tests): add app.dependency_overrides for auth in spend logs tests
test_ui_view_spend_logs_with_status, test_ui_view_spend_logs_with_model,
test_ui_view_spend_logs_with_model_id, and test_view_spend_logs_summarize_parameter
all send Bearer sk-test without mocking user_api_key_auth. When a prior test
in the same xdist worker sets master_key, the auth check fails for sk-test
and the test fails intermittently.
Fix: use app.dependency_overrides[ps.user_api_key_auth] to bypass auth,
same pattern as other tests in the same file.
* fix(tests): mock user_api_key_auth in test_vertex_passthrough_with_no_default_credentials
vertex_proxy_route calls user_api_key_auth internally. When a prior test in the
same xdist worker sets master_key, the auth check fails for the test request
and create_pass_through_route is never called, causing assert_called_once_with to fail.
Fix: patch user_api_key_auth as an AsyncMock in the with mock.patch() block.
2026-02-21 12:51:38 -08:00
Ishaan Jaff
1ed529092f
fix(test): replace flaky test_vertex_ai_gemini_audio_ogg with mocked version ( #21807 )
...
Previously made a real Vertex AI call with a Wikimedia URL that intermittently
failed with URL_REJECTED-REJECTED_FC_TIMEOUT.
Now mocks HTTPHandler.post and VertexBase._ensure_access_token so the test
verifies the translation (OGG -> file_data with audio/ogg mime_type) without
any real network calls. Runs in ~0.36s instead of ~60s.
2026-02-21 12:49:06 -08:00
ryan-crabbe
da8f038178
Merge pull request #20789 from ryan-crabbe/perf/cache-openai-init-params
...
perf: pre-compute OpenAI client __init__ params at module load
2026-02-21 12:47:54 -08:00
Ishaan Jaff
d928588de0
docs: mark v1.81.12 as stable ( #21809 )
...
* docs: mark v1.81.12 as stable, point to stable docker image and pip
* docs: fix v1.81.12 docker image to point to stable
2026-02-21 12:45:41 -08:00
ryan-crabbe
4c393de216
Merge pull request #21213 from BerriAI/litellm_fix_streaming_connection_pool_leak
...
fix: close streaming connections to prevent connection pool exhaustion
2026-02-21 12:45:18 -08:00
Ryan Crabbe
5bcaeabfd8
Merge origin/main into litellm_fix_streaming_connection_pool_leak
...
Resolve conflict in test_proxy_server.py: keep both async_data_generator
cleanup tests and store_model_in_db DB config override tests.
2026-02-21 12:44:50 -08:00
ryan-crabbe
53f3dfa989
Merge pull request #20354 from ryan-crabbe/perf/callback-registration-routing
...
perf: move async/sync callback separation from per-request to registration
2026-02-21 12:41:28 -08:00
Ishaan Jaff
086ced1f6a
add missing sql_injection.yaml policy template ( #21806 )
2026-02-21 12:40:44 -08:00
Ryan Crabbe
ea32ad72c6
Merge origin/main into perf/callback-registration-routing
...
Resolve conflicts:
- logging_callback_manager.py: keep PR's MAX_CALLBACKS, _is_async_callable, Callable type
- test_utils.py: keep both TestCallbackAsyncSyncSeparation and TestMetadataNoneHandling
2026-02-21 12:40:23 -08:00
Ishaan Jaff
02670582c5
fix(tests): replace asyncio.sleep(1) with event-based wait in metadata callback tests ( #21805 )
2026-02-21 12:40:06 -08:00
Miguel Armenta
0fe3145e83
Fix/presidio controls ( #21798 )
...
* check should_run_guardrail in sync logging hook path
* Add tests for CustomGuardrail logging behavior
Added tests to ensure CustomGuardrail logging behavior based on the guardrail execution state.
---------
Co-authored-by: Miguel Armenta <ma826r@att.com>
2026-02-21 12:39:17 -08:00
ryan-crabbe
f5139716a1
Merge pull request #20882 from ryan-crabbe/perf/optimize-model-dump-preserved-fields
...
perf: optimize model_dump_with_preserved_fields
2026-02-21 12:36:02 -08:00
Ishaan Jaff
daa682e125
fix(tests): add missing start_db_health_watchdog_task mock ( #21804 )
...
* fix(tests): add missing start_db_health_watchdog_task mock in test_proxy_server_prisma_setup
* fix(tests): add missing start_db_health_watchdog_task mock in test_health_check_not_called_when_disabled
2026-02-21 12:31:52 -08:00
Ishaan Jaffer
0b69c21eca
Revert "fix(http_handler): bypass cache when shared_session is provided for aiohttp tracing ( #20630 )"
...
This reverts commit 7ee36c2a3a .
2026-02-21 12:27:46 -08:00
ryan-crabbe
d7b3b2eb9e
Merge pull request #20374 from ryan-crabbe/perf/cache-access-groups
...
perf: cache get_model_access_groups() no-args result on Router
2026-02-21 12:25:45 -08:00
ryan-crabbe
469951466e
Merge pull request #20541 from ryan-crabbe/perf/cost-calculator-optimizations
...
Perf/cost calculator optimizations
2026-02-21 12:18:38 -08:00