Harshit Jain
9fc3c77c42
fix: ensure arrival_time is set before calculating queue time
2026-02-23 17:04:47 +05:30
Sameer Kankute
3561bfb96c
Merge pull request #21640 from ta-stripe/feat/regional-sts-endpoint-for-auth_with_role_name
...
feat(bedrock): support optional regional STS endpoint in role assumption
2026-02-23 13:28:54 +05:30
Sameer Kankute
55ee8cdd56
Merge pull request #21701 from ta-stripe/fix/bedrock-openai-imported-model-name-encoding
...
fix(bedrock): encode model arns for OpenAI compatible bedrock imported models
2026-02-23 13:25:02 +05:30
Sameer Kankute
f54fb9aeb1
Add tests for fast and us
2026-02-23 11:25:47 +05:30
Sameer Kankute
22bccc4f61
Fix entries with fast and us/
2026-02-23 11:23:24 +05:30
Harshit Jain
304862175e
Merge branch 'main' into litellm_fix_langfuse_otel_trace_v2
2026-02-22 19:54:05 +05:30
Harshit28j
1cc185eb16
fix: update opentelemetry with greptile review
2026-02-22 19:53:18 +05:30
Krish Dholakia
76ccc9e844
Guardrail Policy Versioning ( #21862 )
...
* feat: initial commit, adding support for policy versioning on litellm
* fix(policy_registry): support policy versioning
* fix: multiple QA fixes for policy flow builder with guardrail versioning on litellm
* feat: ui improvements
* feat: add prisma migration
* fix: address greptile fixes
2026-02-21 20:14:31 -08:00
Krish Dholakia
52585eb2d7
Revert "fix(vertex_ai): enable context-1m-2025-08-07 beta header ( #21870 )" ( #21876 )
...
This reverts commit bce078a796
2026-02-21 20:12:01 -08:00
Edwin Isac
bce078a796
fix(vertex_ai): enable context-1m-2025-08-07 beta header ( #21870 )
...
* server root path regression doc
* fixing syntax
* fix: replace Zapier webhook with Google Form for survey submission (#21621 )
* Replace Zapier webhook with Google Form for survey submission
* Add back error logging for survey submission debugging
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* Revert "Merge pull request #21140 from BerriAI/litellm_perf_user_api_key_auth"
This reverts commit 0e1db3f7e47e2d6f2355Fixes #21861
---------
Co-authored-by: yuneng-jiang <yuneng.jiang@gmail.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2026-02-21 20:11:13 -08:00
LeeJuOh
50f36d9ca6
fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo ( #21754 )
...
* fix(budget): fix timezone config lookup and replace hardcoded timezone map with ZoneInfo
* fix(budget): update stale docstring on get_budget_reset_time
2026-02-21 19:35:06 -08:00
Ryan Crabbe
94b76ea9ad
feat: add network_mock transport for benchmarking proxy overhead without real API calls
...
Intercepts at httpx transport layer so the full proxy path (auth, routing,
OpenAI SDK, response transformation) is exercised with zero-latency responses.
Activated via `litellm_settings: { network_mock: true }` in proxy config.
2026-02-21 17:52:39 -08:00
yuneng-jiang
ca9111ea31
feat: add disable_show_blog to UISettings
...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 17:34:08 -08:00
yuneng-jiang
4b1ce1ff54
fix: log fallback warning in blog posts endpoint and tighten test
2026-02-21 17:34:08 -08:00
yuneng-jiang
021a9097c4
feat: add GET /public/litellm_blog_posts endpoint
...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 17:34:08 -08:00
yuneng-jiang
7e5c63c134
test: add cache reset fixture and LITELLM_LOCAL_BLOG_POSTS test
...
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-21 17:34:08 -08:00
yuneng-jiang
be1a543b55
feat: add GetBlogPosts utility with GitHub fetch and local fallback
...
Adds GetBlogPosts class that fetches blog posts from GitHub with a 1-hour
in-process TTL cache, validates the response, and falls back to the bundled
blog_posts_backup.json on any network or validation failure.
2026-02-21 17:34:08 -08:00
Ishaan Jaffer
52294029a0
test_vertex_ai_gemini_2_5_pro_streaming
2026-02-21 16:59:22 -08:00
Ishaan Jaffer
2270a3aaf3
Revert "Merge pull request #21140 from BerriAI/litellm_perf_user_api_key_auth"
...
This reverts commit 0e1db3f7e47e2d6f2355
2026-02-21 16:57:42 -08:00
Ryan Crabbe
643c9b6c04
Merge remote-tracking branch 'origin/main' into litellm_perf_user_api_key_auth
2026-02-21 16:03:31 -08:00
Ishaan Jaffer
d31d5b8486
fix failing tests
2026-02-21 15:48:26 -08:00
Ishaan Jaffer
26ea29afd3
test_get_usage_as_dict
2026-02-21 15:39:06 -08:00
Krish Dholakia
1f7eeb274c
Agent Builder - improve rejected response detection based on agent response ( #21850 )
...
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
* feat: new response rejection detector
* fix: multiple fixes
* feat: add mcp tools support to agent builder
create an agent with access to llm's + mcp servers
2026-02-21 15:34:42 -08:00
Krish Dholakia
9fc6fd647c
Agent Builder - support new experimental agent builder, to ensure agents pass compliance checks ( #21817 )
...
* fix: feat: add litellm_system_prompt support
* feat: support new 'litellm_agent' model provider
* feat: ui/ - new agent builder ui
* fix(anthropic/chat/transformation.py): normalize max_tokens if decimal
* feat(agentbuilderview.tsx): run compliance datasets against litellm agent
2026-02-21 15:32:47 -08:00
Ishaan Jaff
bab4127cae
fix(tests): fix flaky test_use_prisma_db_push_flag_behavior ( #21849 )
...
Replace Click CliRunner with standalone_mode=False to avoid
"I/O operation on closed file" errors caused by Click's stream
isolation in CI environments.
2026-02-21 15:23:55 -08:00
Ishaan Jaff
f74a1c94df
test(router): add coverage tests for _is_complexity_router_deployment and init_complexity_router_deployment ( #21848 )
2026-02-21 15:21:10 -08:00
Darien Kindlund
ca5c109a92
feat: add optional digest mode for Slack alert types ( #21683 )
...
Adds per-alert-type digest mode that aggregates duplicate alerts
within a configurable time window and emits a single summary message
with count, start/end timestamps.
Configuration via general_settings.alert_type_config:
alert_type_config:
llm_requests_hanging:
digest: true
digest_interval: 86400
Digest key: (alert_type, request_model, api_base)
Default interval: 24 hours
Window type: fixed interval
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-02-21 15:19:17 -08:00
Ryan Crabbe
c7ad8053b1
Merge origin/main into litellm_perf_user_api_key_auth
...
Resolve conflicts:
- pass_through_endpoints.py: take main's version, re-apply
MAPPED_PASS_THROUGH_PREFIXES startswith(tuple) optimization
- test_user_api_key_auth.py: keep both auth optimization regression
tests and JWT admin identity field tests
2026-02-21 15:14:20 -08:00
Ishaan Jaff
9f459c5c57
fix(logging): preserve pass-through endpoint response_cost ( #21844 )
...
* fix(logging): preserve pass-through endpoint response_cost in async_success_handler
Two places in the logging pipeline were overwriting response_cost that
pass-through handlers (Gemini/Vertex) had already calculated:
1. _process_hidden_params_and_response_cost fell through to
_response_cost_calculator which returns None for pass-through calls
2. async_success_handler pass-through branch unconditionally set
response_cost = None (introduced in PR #19887 )
Now both places check if response_cost is already set before overwriting.
* test: add regression test for pass-through endpoint response_cost preservation
2026-02-21 15:09:45 -08:00
Ryan Crabbe
bb24ebebd9
Merge origin/main into litellm_perf_convert_model_response_frozensets
...
Resolve conflict: keep main's provider_specific_fields passthrough
preservation while using frozenset set-difference optimization.
2026-02-21 14:52:43 -08:00
Ishaan Jaff
8afeaf8da4
fix(tests): fix flaky test_create_vertex_fine_tune_jobs_mocked - handle background Datadog flush ( #21838 )
2026-02-21 14:44:01 -08:00
Ishaan Jaff
d7b22d340b
fix(tests): move test_router_azure_acompletion to llm_translation testing ( #21837 )
2026-02-21 14:41:53 -08:00
Ishaan Jaff
59e5b7e8c6
fix(tests): use monkeypatch.setenv for Redis pool max_connections tests ( #21834 )
...
Replace patch('litellm._redis._get_redis_client_logic') with monkeypatch.setenv
in test_max_connections_url_config and test_max_connections_url_config_string_value.
The mock was unreliable in CI (REDIS_URL is set to the real Redis Cloud server),
causing the pool to silently use the real config instead of the test config.
Using monkeypatch.setenv tests the full env-var→pool chain more robustly and
matches the actual production code path.
2026-02-21 14:38:28 -08:00
Ishaan Jaff
235a47c576
fix(tests): mock test_claude_tool_use_with_gemini to fix flaky CI ( #21832 )
...
* ui fixes
* fix(tests): mock test_claude_tool_use_with_gemini to avoid MALFORMED_FUNCTION_CALL flakiness
2026-02-21 14:34:54 -08:00
Ishaan Jaff
fb4249005e
fix(tests): add atexit.register mock to prevent Click isolation stream closure in test_use_prisma_db_push_flag_behavior ( #21829 )
2026-02-21 14:28:02 -08:00
yuneng-jiang
8c5be4cb62
Revert "fix(proxy): recover from prisma-query-engine zombie process ( #21707 )"
...
This reverts commit 977ad015ca
2026-02-21 14:20:06 -08:00
Ishaan Jaff
56a095a079
fix(tests): update deprecated Anthropic model in test_user_model_access ( #21826 )
...
* fix(tests): clear _async_success_callback in vertex fine-tune mocked tests to prevent Datadog interference
* fix(tests): update deprecated claude-3-5-haiku-20241022 to claude-haiku-4-5-20251001
2026-02-21 14:18:24 -08:00
Ishaan Jaff
a1ead765ed
fix(tests): clear _async_success_callback in vertex fine-tune mocked tests to prevent Datadog interference ( #21825 )
2026-02-21 14:16:18 -08:00
Ishaan Jaff
494aad4a68
fix(tests): isolate auth in vertex passthrough and spend logs date range tests ( #21824 )
...
test_vertex_passthrough_with_default_credentials and
test_view_spend_logs_with_date_range_summarized fail intermittently when a
prior xdist worker sets master_key — auth then rejects the unauthenticated
test requests before the code under test is reached.
- mock user_api_key_auth in test_vertex_passthrough_with_default_credentials
(same pattern used for test_vertex_passthrough_with_no_default_credentials
in #21810 )
- wrap test_view_spend_logs_with_date_range_summarized in
app.dependency_overrides[ps.user_api_key_auth] with try/finally cleanup
(same pattern used for the other spend log tests in #21810 )
2026-02-21 14:14:50 -08:00
Ishaan Jaff
dd6a74da63
fix(tests): isolate litellm.cache and CLI env vars in flaky tests ( #21821 )
...
- TestSpendLogsPayload: save/restore litellm.cache in setup_method/teardown_method
so tests that run after a cache-setting test don't see a non-None cache and get
a hash instead of "Cache OFF" in the cache_key field
- test_use_prisma_db_push_flag_behavior: apply clean_env pattern (strip DATABASE_URL/DIRECT_URL,
then set DATABASE_URL to test value) inside the with block instead of using @patch.dict
decorator, matching the pattern from test_skip_server_startup to avoid Click 8.3.x
StreamMixer stream lifecycle issues in CI
2026-02-21 14:11:48 -08:00
Ishaan Jaff
6acfa1c71d
fix(tests): clear ANTHROPIC_BASE_URL/ANTHROPIC_API_BASE in spend log api_base tests ( #21820 )
...
Tests hardcode expected api_base as https://api.anthropic.com/v1/messages but
if ANTHROPIC_BASE_URL is set in the environment the recorded api_base changes,
causing a mismatch. Clear both env vars via monkeypatch at the start of each test.
2026-02-21 14:11:18 -08:00
Ishaan Jaff
6ec16d583d
fix(test): add timeout to flush() call to prevent 300s hang in CI ( #21819 )
...
GLOBAL_LOGGING_WORKER.flush() calls queue.join() which blocks until all
items are task_done(). In CI with pytest-asyncio, each test gets a fresh
event loop so the worker reinitializes its queue - items from a previous
test never get task_done(), causing an infinite hang.
Fix: wrap flush() with asyncio.wait_for(..., timeout=10.0).
2026-02-21 14:10:57 -08:00
ryan-crabbe
b17d37eceb
Merge pull request #21815 from BerriAI/litellm_fix_openai_init_params_immutable
...
fix: make cached OpenAI init params immutable and fix import ordering
2026-02-21 13:33:03 -08:00
Ishaan Jaff
c810f5cd63
fix(tests): replace fake France Azure endpoint in test_router_azure_acompletion ( #21818 )
2026-02-21 13:26:59 -08:00
Ryan Crabbe
9e1d83e3de
fix: make LITELLM_CLIENT_SPECIFIC_PARAMS a tuple to prevent TypeError
...
tuple + list raises TypeError in get_openai_client_cache_key. Also add
test coverage for get_openai_client_cache_key to catch type mismatches.
2026-02-21 13:25:23 -08:00
shin-bot-litellm
1be30f5129
feat(router): Add complexity-based auto routing strategy ( #21789 )
...
* feat(router): Add complexity-based auto routing strategy
Adds a rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - with zero API calls and sub-millisecond
latency.
## Features
- **Zero external API calls** - all scoring is local
- **Sub-millisecond latency** - typically <1ms per classification
- **Weighted multi-dimensional scoring** across 7 dimensions:
- Token count (short=simple, long=complex)
- Code presence (code keywords → complex)
- Reasoning markers ("step by step" → reasoning tier)
- Technical terms (domain complexity)
- Simple indicators ("what is" → simple, negative weight)
- Multi-step patterns (numbered steps)
- Question complexity (multiple questions)
- **Configurable tier boundaries** and model mappings
- **Reasoning override** - 2+ reasoning markers force REASONING tier
## Usage
```yaml
model_list:
- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gpt-4o-mini
MEDIUM: gpt-4o
COMPLEX: claude-sonnet-4
REASONING: o1-preview
```
Inspired by ClawRouter: https://github.com/BlockRunAI/ClawRouter
## Files Added
- litellm/router_strategy/complexity_router/complexity_router.py - Main router class
- litellm/router_strategy/complexity_router/config.py - Configuration and defaults
- litellm/router_strategy/complexity_router/__init__.py - Package exports
- litellm/router_strategy/complexity_router/README.md - Documentation
- tests/test_litellm/router_strategy/test_complexity_router.py - Test suite (37 tests)
## Files Modified
- litellm/router.py - Integration with pre_routing_hook
- litellm/types/router.py - New config params
* feat(router): Add complexity-based auto routing strategy
Adds a new rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - without any external API calls.
## Features
- Weighted scoring across 7 dimensions: token count, code presence, reasoning
markers, technical terms, simple indicators, multi-step patterns, questions
- Maps to 4 tiers: SIMPLE, MEDIUM, COMPLEX, REASONING
- Each tier configurable to a different model
- Zero API calls, <1ms latency
- Inspired by ClawRouter
## Configuration
```yaml
model_list:
- model_name: smart_router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gemini-2.0-flash
MEDIUM: gpt-4o-mini
COMPLEX: claude-sonnet-4
REASONING: claude-opus-4
```
## Use Cases
- Cost optimization: route simple queries to cheaper models
- Quality optimization: route complex queries to capable models
- Zero configuration: works out of the box with sensible defaults
* feat(router): Add complexity-based auto routing strategy
Adds a new rule-based routing strategy that classifies requests by complexity
and routes them to appropriate models - without any external API calls.
- Weighted scoring across 7 dimensions: token count, code presence, reasoning
markers, technical terms, simple indicators, multi-step patterns, questions
- Maps to 4 tiers: SIMPLE, MEDIUM, COMPLEX, REASONING
- Each tier configurable to a different model
- Zero API calls, <1ms latency
- Inspired by ClawRouter
```yaml
model_list:
- model_name: smart_router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gemini-2.0-flash
MEDIUM: gpt-4o-mini
COMPLEX: claude-sonnet-4
REASONING: claude-opus-4
```
- Cost optimization: route simple queries to cheaper models
- Quality optimization: route complex queries to capable models
- Zero configuration: works out of the box with sensible defaults
* feat: add enterprise presets for complexity router
Adds preset configurations for different cloud providers:
- bedrock: AWS Bedrock (Claude models)
- vertex: Google Vertex AI (Gemini models)
- azure: Azure OpenAI (GPT + o1)
- standard: Direct API (OpenAI + Anthropic)
- cost_optimized: Maximum savings (Gemini Flash + cheaper models)
Usage:
```yaml
complexity_router_config:
preset: bedrock # or vertex, azure, standard, cost_optimized
```
* feat(ui): update auto router submit handler for complexity router
- Handle complexity_router model type in submit handler
- Generate correct litellm_params for complexity router:
- model: auto_router/complexity_router
- complexity_router_config: { tiers: { SIMPLE, MEDIUM, COMPLEX, REASONING } }
- Keep existing semantic router handling intact
- Add success notification with router type name
* docs: update PR description with UI changes
* chore: remove preset feature, keep simple tier config
* fix: exclude complexity_router from auto_router check
The _is_auto_router_deployment() was matching all auto_router/* models,
causing complexity_router to fail initialization. Now it explicitly
excludes auto_router/complexity_router which has its own handler.
* fix(complexity_router): Address Greptile review feedback
Fixes 5 issues flagged in code review:
1. **Mutable singleton mutation bug** - Now always creates a new
ComplexityRouterConfig instance instead of reusing DEFAULT_COMPLEXITY_CONFIG
singleton, preventing cross-instance config pollution.
2. **Substring matching false positives** - Added word boundaries (spaces)
to short keywords like 'ok', 'try', 'api', 'git', 'node', 'java', 'vue'
to prevent matching within longer words (e.g., 'capital' matching 'api').
3. **Redundant message extraction** - Simplified to single reverse loop that
extracts both last user message and last system prompt efficiently.
4. **Unused imports** - Removed unused DEFAULT_CREATIVE_KEYWORDS and
DEFAULT_MULTI_STEP_PATTERNS imports.
5. **Missing async_pre_routing_hook tests** - Added comprehensive tests for:
- Multi-turn conversations
- List-type content handling
- No user message case
- Empty string content
- Message preservation
- Singleton mutation prevention
* fix(complexity_router): Address Greptile review feedback
- Use word boundary matching for short keywords (<5 chars) to avoid
false positives (e.g., 'api' matching 'capital', 'git' matching 'digital')
- Remove 'ok' from simple keywords (too many false positives)
- Add tests for keyword false positive prevention
- Fix test expectations for edge cases (empty string content, list content)
Addresses: 2/5 Greptile score feedback on PR #21789
* docs(auto_routing): Add complexity router documentation
- Add Complexity Router section to auto_routing.md
- Include comparison table with semantic auto router
- Add Python SDK and Proxy Server configuration examples
- Document all configuration options (tier boundaries, token thresholds, dimension weights)
- Explain how complexity scoring works
* feat(complexity_router): Add eval suite + tune scoring parameters
Added comprehensive evaluation suite with 29 test cases covering:
- SIMPLE tier: greetings, definitions, factual questions
- MEDIUM tier: technical explanations, comparisons, debugging
- COMPLEX tier: architecture design, complex coding
- REASONING tier: explicit reasoning requests
- Regression tests: substring false positive prevention
Tuned scoring parameters based on eval results:
- Lowered tier boundaries (0.15/0.35/0.60) for better tier distribution
- Increased code/technical weights (0.30/0.25) for complex prompts
- Reduced simple indicator weight (0.05) to avoid over-penalizing
- Fixed 'hey'/'hi' keywords to require leading space
Eval results: 29/29 passed (100%)
* fix(complexity_router): Address Greptile review round 2
1. **Empty user message handling** - Changed from falsy check to None check
to properly distinguish 'no user message' from 'empty string message'
2. **ReDoS prevention** - Changed 'first.*then' to 'first.*?then' (non-greedy)
to prevent regex backtracking on pathological inputs
3. **Documentation sync** - Updated README.md to match actual config values:
- Tier boundaries: 0.15/0.35/0.60 (not 0.25/0.50/0.75)
- Dimension weights: tokenCount=0.10, codePresence=0.30, technicalTerms=0.25,
simpleIndicators=0.05, multiStepPatterns=0.03, questionComplexity=0.02
4. **Missing UI component** - Added ComplexityRouterConfig.tsx with:
- Tier-to-model dropdown selectors
- Descriptions and examples for each tier
- How classification works explanation
5. **Inline import comment** - Added explanation for why ComplexityRouter
import is inline (matches AutoRouter pattern, avoids circular imports)
* docs(auto_routing): fix dimension weights and tier boundaries to match config.py defaults
* fix(complexity_router): skip empty string content in async_pre_routing_hook
* fix(router): remove or {} masking None complexity_router_config
* fix(config): remove unused DEFAULT_MULTI_STEP_PATTERNS and DEFAULT_CREATIVE_KEYWORDS exports
* fix(complexity_router): use word boundary matching for all single-word keywords, avoid double-scanning reasoning keywords
* fix(router): clarify circular import comment for ComplexityRouter
* docs(README): fix token thresholds to match config.py defaults
* test(complexity_router): add false positive tests for error/class/merge keyword matching
* fix(complexity_router): align .get() fallbacks with config.py defaults, document system prompt scoring
* fix(config): deduplicate keywords across code and technical lists
---------
Co-authored-by: OpenClaw Assistant <assistant@openclaw.ai>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
2026-02-21 13:23:37 -08:00
Ryan Crabbe
bbbec23c8b
fix: update tests to match tuple return type for cached init params
2026-02-21 13:18:03 -08:00
Ishaan Jaff
8483477512
fix(test): add asyncio.sleep(0) before flush() to prevent hang in test_async_no_duplicate_spend_logs ( #21813 )
2026-02-21 13:09:36 -08:00
Ishaan Jaff
0a0768b3df
fix(ci): resolve mypy and check_code_and_doc_quality CI failures ( #21812 )
...
- fix(mypy): suppress [misc] type error in common_utils.py for cls.__init__ access
- fix(mypy): move type: ignore comment to correct line in test_eval.py (line 232 not 231)
- fix(mypy): suppress [misc] and pre-existing pyright errors in vertex_ai_non_gemini.py
- fix(check_licenses): strip inline comments before parsing requirements.txt lines so CVE comments don't break packaging.requirements.Requirement()
- fix(router_coverage): add _merge_tools_from_deployment and _invalidate_access_groups_cache to ignored list (private helpers tested indirectly)
2026-02-21 13:08:47 -08:00
Ryan Crabbe
75bc8329e2
Merge origin/main into litellm_perf_skip_usage_roundtrip
...
Resolve conflict in litellm_logging.py: take main's version and
re-apply get_usage_as_dict optimization on top.
2026-02-21 12:55:55 -08:00
