Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
P1 review: adaptive_router.py had a top-level import of
AdaptiveRouterUpdateQueue from litellm.proxy.db, which broke the
SDK/proxy boundary that every other router strategy respects. No
other router_strategy module imports from litellm.proxy at module
level.
The queue only depends on litellm._logging — it never needed to
live under litellm.proxy. Moved:
litellm/proxy/db/db_transaction_queue/adaptive_router_update_queue.py
→ litellm/router_strategy/adaptive_router/update_queue.py
tests/test_litellm/proxy/db/db_transaction_queue/
test_adaptive_router_update_queue.py
→ tests/test_litellm/router_strategy/adaptive_router/test_update_queue.py
Also switched the queue's logger from verbose_proxy_logger to
verbose_router_logger to match the new module's ownership.
P2 review: drop unused constant STAGNATION_JACCARD_EXACT from
config.py — it was defined but never referenced.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
P1: start the adaptive-router flusher loop unconditionally at proxy boot
instead of gating on 'adaptive_routers is non-empty'. Adaptive routers
added via /config/reload after boot now have their queues drained.
State is lazy-loaded per router on first flush tick (new _state_loaded
flag on AdaptiveRouter) so hot-reloaded routers still get their
persisted priors.
P2: _finalize_adaptive_router_if_configured now prunes stale
AdaptiveRouterPostCallHook callbacks from every litellm callback list
before registering new ones. Without this, every Router replacement
left the old hooks wired up in litellm.callbacks and double-fired
signal recording for every request. Uses
logging_callback_manager.remove_callbacks_by_type (same pattern as the
semantic tool filter).
CI fixes:
- black --check failure: reformatted litellm/router.py
- schema migration diff: aligned @@index with the explicit index name
('idx_adaptive_router_session_activity') from the original migration
by adding 'map:' to all three schema.prisma copies. No new migration
needed.
Tests: 1 new covering the prune-on-hot-reload path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Mark last_updated_at (AdaptiveRouterState) and last_activity_at
(AdaptiveRouterSession) with @updatedAt so Prisma refreshes the
timestamps on every write. Without this the fields stayed frozen at
INSERT time and the last_activity_at index was misleading for any
future TTL/eviction logic. Applied to all three schema.prisma copies;
no migration SQL change needed (Prisma @updatedAt is a client-side
annotation that doesn't touch DDL).
- get_state_snapshot: report cell.total_samples instead of alpha+beta
for the 'samples' field. The previous value inflated every cell by
the COLD_START_MASS prior (e.g. showed 10.0 before any real traffic
arrived), which confused operators reading /adaptive_router/.../state.
Updated docs + the snapshot test to match.
Also fixes two pre-existing merge-break syntax errors in router.py
(missing ')' on the AdaptiveRouter TYPE_CHECKING import; truncated
async_pre_routing_hook dispatch call for the adaptive router branch)
that were masking the rest of the file from the interpreter.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
- _owner_cache now opportunistically sweeps expired entries past
_OWNER_CACHE_SWEEP_THRESHOLD live entries. Previously sessions that never
came back piled up forever.
- flush_session_to_db strips session_id/router_name/model_name from the update
payload. Prisma rejects writes to @@id fields.
- record_turn no longer persists last_user_content / last_assistant_content /
tool_call_history / pending_tool_calls. Those are needed only in-memory for
the next turn's signal detection; writing user prompts and tool payloads to
the DB would store PII for every conversation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- SessionState now carries clean_credit_awarded + last_processed_turn (matching
the DB schema). Satisfaction only fires once per session AND only after
MIN_TURNS_FOR_CLEAN_CREDIT turns of context — early "thanks" no longer
inflates alpha.
- _detect_failure no longer treats empty content as failure. Many tools
legitimately return empty output (zero-result searches, silent bash);
penalizing those corrupted the bandit posterior. Only is_error fires now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(router): add auto_router/quality_router for quality-tier routing (#25987)
* feat(router): add auto_router/quality_router for quality-tier routing
Adds a new auto-router type that routes a request to a model at a target
quality tier. The quality tier is inferred by re-using the existing
ComplexityRouter's classification, then mapped through an admin-configured
complexity_to_quality table. Each candidate model declares its own
quality_tier in model_info.litellm_routing_preferences.
Resolution strategy: exact tier match, else round up to the next higher
tier, else fall back to default_model.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): add capability-based filtering
Each deployment can declare a `capabilities: List[str]` field in
`model_info.litellm_routing_preferences` (e.g. ["vision",
"function_calling"]). Requests can pass `litellm_capabilities` in
`request_kwargs` to require specific capabilities — the router will only
route to deployments whose declared capabilities are a superset.
Resolution still walks tier (exact → round up), but at each tier filters
by capability before picking. Falls back to default_model only when it
also satisfies the required capabilities; otherwise raises rather than
silently routing to a model that lacks a required capability.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): expose routing decision in response headers
For transparency, expose the QualityRouter's routing decision in the
proxy response headers:
x-litellm-quality-router-model → picked model_name (e.g. "haiku-vision")
x-litellm-quality-router-tier → resolved quality tier (e.g. "1")
x-litellm-quality-router-complexity → ComplexityTier name (e.g. "SIMPLE")
Mechanism: the pre-routing hook stashes the decision in
request_kwargs["metadata"]["quality_router_decision"]. After the call
returns, Router.set_response_headers lifts the decision into
response._hidden_params["additional_headers"] alongside the existing
x-litellm-model-group / x-litellm-model-id headers. Existing metadata
keys (trace_id, user_id, etc.) are preserved.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): replace capabilities with keyword override
Drops the capability-based filtering in favor of a keyword-based override
for v0:
- RoutingPreferences.keywords: List[str] (replaces capabilities) — each
deployment can declare substring keywords.
- If any declared keyword (case-insensitive) appears in the user message,
the router short-circuits the complexity-classification flow and routes
to the matching deployment.
- Tiebreaker for overlapping keyword matches: quality_tier DESC, then
cheapest model_info.input_cost_per_token ASC. Unpriced models lose ties
to priced ones.
Decision metadata + headers now expose the override:
x-litellm-quality-router-via → "keyword" | "quality_tier"
x-litellm-quality-router-keyword → matched keyword (only on keyword route)
x-litellm-quality-router-complexity → complexity tier (only on tier route)
Removes:
- request_kwargs["litellm_capabilities"] reading
- _model_capabilities, _model_supports_capabilities,
_first_capable_model_at_tier, capability filter in
_resolve_model_for_quality_tier
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): add explicit `order` to RoutingPreferences
Adds an explicit priority field to RoutingPreferences for resolving
collisions deterministically:
RoutingPreferences.order: Optional[int] # lower wins; unset = +inf
Used as the PRIMARY tiebreaker in two places:
1. Keyword overlap: when multiple deployments declare the same matching
keyword, sort by (order ASC, quality_tier DESC, input_cost_per_token
ASC, model_name ASC). Explicit always beats implicit.
2. Tier resolution: when multiple deployments share a quality tier,
`_resolve_model_for_quality_tier` picks the one with the lowest
order. The tier list is now sorted at index-build time.
This lets admins make routing decisions explicit when the natural
quality-and-price ordering would pick the wrong model.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): reorder tiebreak to (quality, order, price)
Changes the tiebreak ordering so quality_tier always wins first, then
explicit `order` is used to break ties within the same tier, then price
breaks the rest:
1. quality_tier DESC ← best model wins first
2. order ASC ← explicit priority within a tier
3. input_cost_per_token ASC
4. model_name ASC
Previously `order` was the primary key — that meant a tier-2 model with
`order=1` would beat a tier-3 model with no `order`, which is the wrong
default. Now `order` only resolves collisions among same-tier candidates.
Tier resolution (within a single tier) keeps the same key minus quality:
(order ASC, cost ASC, name).
Test renames + flips:
- test_explicit_order_overrides_quality_tier → test_quality_wins_over_explicit_order
- new: test_order_breaks_tie_within_same_quality_tier
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* fix(quality_router): resolve Greptile review feedback
Addresses four P1 findings from PR review plus test coverage:
1. set_model_list missing quality_routers reset
- Hot-reloading the Router would leave stale QualityRouter instances
pointing at the old model_list. `set_model_list` now clears
`self.quality_routers` alongside the other indices.
2. Round-down fallback before default_model
- `_resolve_model_for_quality_tier` now rounds DOWN to the closest
lower tier after round-up fails, before falling back to
`default_model`. Degrades gracefully rather than jumping straight
off-tier.
3. RoutingPreferences validation bypass
- `_build_tier_index` now instantiates `RoutingPreferences(**prefs)`
so invalid shapes (e.g. non-int quality_tier) raise a clear
ValueError instead of silently succeeding.
4. Config-ordering dependency
- `_tier_to_models` is now built lazily on first access. Previously,
eager construction in `__init__` meant a QualityRouter deployment
had to appear AFTER all its referenced models in config.yaml,
because `Router._create_deployment` populates `model_list`
incrementally. Any `available_models` defined after the router
entry would silently be reported as missing.
Also adds 6 new tests covering each fix:
- test_invalid_quality_tier_type_raises_clear_error
- test_router_can_be_instantiated_before_its_targets_exist
- test_set_model_list_clears_quality_routers_registry
- test_rounds_down_when_no_higher_tier_exists
- test_rounds_down_prefers_closest_lower_tier
- test_prefers_round_up_over_round_down
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* style: apply black 24.10.0 formatting to pre-existing offenders
Unblocks the LiteLLM Linting check for this PR — these 12 files are already
failing `black --check` on main (the lint workflow only runs on PRs, so main
drifts). No behavior changes; formatting-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update litellm/router.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Support /v1/responses in complexity router (#26137)
* feat(proxy): add --reload flag for uvicorn hot reload (dev only)
Opt-in CLI flag, off by default, no env var. Only affects the uvicorn
run path; gunicorn/hypercorn paths and prod (which doesn't pass the
flag) are unaffected.
* Feature/add audio support for scaleway (#26110)
* feat(scaleway): add SCALEWAY to LlmProviders enum
* feat(scaleway): add audio transcription config and dispatch wiring
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(scaleway): add behavior tests for audio transcription config
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(scaleway): advertise audio_transcriptions in endpoint-support JSON
* docs(scaleway): document audio transcription support
* fix(scaleway): address PR review — plain-text response_format + missing-key fail-fast
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(scaleway): cover new response paths, drop gettysburg.wav coupling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Prompt Compression - add it to the proxy (#25729)
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Support /v1/responses in complexity router
Adds cross-format support to the complexity router via the guardrail
translation handler dispatch. Adds get_structured_messages to base
translation plus OpenAI chat, Responses, and Anthropic handlers.
Auto-router helper _extract_text_from_messages handles tool-call and
multimodal messages. Widens async_pre_routing_hook messages type to
Dict[str, Any].
Fixes https://github.com/BerriAI/litellm/issues/25134
* chore: apply black formatting
* fix: fallback to trying each handler when route inference fails
---------
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* test: cover _is_quality_router_deployment and init_quality_router_deployment
* fix: reset auto_routers on set_model_list to prevent hot-reload ValueError
* style: apply black formatting to websearch_interception and agentic_streaming_iterator
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
Six tests in test_hooks.py were written against an older API and had been
failing in CI. Updated:
- test_resolve_session_key_* (4 tests): _resolve_session_key now requires
at least SIGNAL_GATE_MIN_MESSAGES messages before deriving a hash (it
returns None on shorter convos to match the signal-processing gate).
Switched the tests to use _long_messages() so they hit the hash path.
- test_post_call_success_hook_* (2 tests): the hook was migrated from
async_post_call_success_hook (mutates response._hidden_params) to
async_post_call_response_headers_hook (returns a headers dict) because
the former fires too late for streaming responses. Rewrote the tests
against the new API; added a metadata-not-dict noop case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The post-call hook was hardcoding tool_results=[] on every Turn, so the
failure detector never saw tool errors and the bandit only learned from
satisfaction — never from negative tool outcomes.
Added _recent_tool_results(messages): walks the request messages from the
tail and collects the contiguous run of role=='tool' entries — those are
the results from the most recent assistant tool_calls round. Normalizes
each to {content, is_error}, the only fields signals._detect_failure /
_detect_exhaustion read.
Tests: 6 new covering empty input, trailing-run extraction, is_error
propagation, boundary at first non-tool message, no-trailing-tool case,
and the end-to-end path from hook -> Turn.tool_results.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use 'auto_router/adaptive_router' prefix in example yaml, docs, and
README — the old 'adaptive_router/...' and 'openai/gpt-4o-mini' values
silently skipped adaptive-router init because detection requires the
'auto_router/adaptive_router' prefix.
- Read x-litellm-min-quality-tier from request headers (and the
'min_quality_tier' metadata key as fallback) in async_pre_routing_hook.
Previously the documented header was defined but never extracted, so
the quality-floor feature was inert.
- Evict expired entries from _session_states. The cache grew without
bound — added a parallel expiry map (same TTL as _owner_cache) and an
opportunistic bulk sweep when the cache crosses a size threshold.
- Align adaptive-router migration SQL with Prisma schema: all count
columns and the 'clean_credit_awarded' / 'last_processed_turn' fields
are NOT NULL in the data model, so the migration now declares them
NOT NULL. Fixes test_aaaasschema_migration_check.
Tests: 8 new covering header/metadata/precedence/invalid-value paths for
min_quality_tier and TTL-based eviction of _session_states.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Router coverage check flagged this method as untested. Adds two cases:
- initializes AdaptiveRouter from model_list and is idempotent on re-entry
- no-op when no adaptive deployments are configured
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(pricing): add unversioned vertex_ai/claude-haiku-4-5 entry
Missing unversioned entry causes cost tracking to return $0.00 for
all requests using vertex_ai/claude-haiku-4-5. All other Vertex AI
Claude models have both versioned and unversioned entries.
* fix(router): skip misleading tags error when no candidates (e.g. cooldown)
Return early from get_deployments_for_tag when healthy_deployments is empty so
tag-based routing does not raise no_deployments_with_tag_routing after cooldown
filters all deployments. Adds regression test.
Made-with: Cursor
* feat(oci): add embedding support and update model catalog
- Add OCIEmbeddingConfig for OCI GenAI embedding models
- Add 16 new chat models (Cohere, Meta Llama, xAI Grok, Google Gemini)
- Add 8 embedding models (Cohere embed v3.0, v4.0)
- Update documentation with embedding examples
- Update pricing for all new models
* test(oci): add unit tests for OCI embedding support
- 17 unit tests covering OCIEmbeddingConfig
- Tests for URL generation, param mapping, request/response transform
- Tests for model pricing JSON completeness
* style(oci): format with black and ruff
* fix(oci): correct embedding request body format
OCI embedText API expects inputs, truncate, and inputType at the
top level of the request body, not nested under embedTextDetails.
Fixed transformation and updated tests accordingly.
Verified with real OCI API: 3/3 embedding models working.
* docs: clarify tag routing early return and test intent
Made-with: Cursor
* fix(oci): address code review findings from Greptile
- P1: Fix signing URL mismatch with custom api_base by accepting
api_base parameter in transform_embedding_request
- P2: Remove encoding_format from supported params (OCI does not
support it, was silently dropped)
- P2: Raise ValueError for token-array inputs instead of silently
converting to string representation
- Add test for token-list rejection
* fix(mcp): add STS AssumeRole support for MCP SigV4 authentication
MCPSigV4Auth only supported static AWS credentials or the boto3 default
credential chain. Production Kubernetes environments typically authenticate
via IAM role assumption (sts:AssumeRole), which was not possible.
Add aws_role_name and aws_session_name parameters to the MCP SigV4 auth
stack. When aws_role_name is provided, MCPSigV4Auth calls sts:AssumeRole
to obtain temporary credentials before signing requests. Explicit keys,
if also provided, are used as the source identity for the STS call;
otherwise ambient credentials (pod role, instance profile) are used.
* fix: stop logging credential values and add missing redaction patterns
Replaces raw credential values in debug/error log messages with
boolean presence checks or type names. Adds PEM block, GCP token,
JWT, SAS token, and service-account blob patterns to the redaction
filter. Fixes private_key pattern to capture full PEM blocks instead
of stopping at the first whitespace.
Addresses: Vertex AI credential JSON (including RSA private key)
being logged to stderr on health check failures.
* fix: log only field names for UserAPIKeyAuth, not full object
* style: apply black formatting to experimental_mcp_client/client.py
* style: fix black/isort formatting and mypy error in proxy_server.py
- Fix black formatting in experimental_mcp_client/client.py (done in prev commit)
- Fix black/isort formatting in key_management_endpoints.py, proxy_server.py, transformation.py
- Fix mypy: iterate over optional list safely (access_group_ids or []) in proxy_server.py
* fix(test): patch check_migration.verbose_logger directly to fix xdist ordering issue
When test_proxy_cli.py tests run before test_check_migration.py in the same
xdist worker, litellm.proxy.db.check_migration is already in sys.modules.
Patching litellm._logging.verbose_logger has no effect on the already-bound
reference. Patch the correct target (check_migration.verbose_logger) and
import the module before patching so the order doesn't matter.
* fix(mypy): make api_base Optional in PydanticAIProviderConfig to match base class signature
---------
Co-authored-by: Ihsan Soydemir <soydemir.ihsan@gmail.com>
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: Daniel Gandolfi <danielgandolfi@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
pytest-asyncio 1.x no longer provides an implicit event loop in sync
fixtures/tests. Make async-dependent fixtures and tests async, and
replace deprecated asyncio.get_event_loop() in tests. Switch
Dockerfile.build_from_pip from Alpine to Debian slim since
pyroscope-io 0.8.x has no musl wheels.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(bedrock): respect s3_region_name for batch file uploads (#23569)
* fix(bedrock): respect s3_region_name for batch file uploads (GovCloud fix)
* fix: s3_region_name always wins over aws_region_name for S3 signing (Greptile feedback)
* fix: _filter_headers_for_aws_signature - Bedrock KB (#23571)
* fix: _filter_headers_for_aws_signature
* fix: filter None header values in all post-signing re-merge paths
Addresses Greptile feedback: None-valued headers were being filtered
during SigV4 signing but re-merged back into the final headers dict
afterward, which would cause downstream HTTP client failures.
Made-with: Cursor
* feat(router): tag_regex routing — route by User-Agent regex without per-developer tag config (#23594)
* feat(router): add tag_regex support for header-based routing
Adds a new `tag_regex` field to litellm_params that lets operators route
requests based on regex patterns matched against request headers — primarily
User-Agent — without requiring per-developer tag configuration.
Use case: route all Claude Code traffic (User-Agent: claude-code/x.y.z) to
a dedicated deployment by setting:
tag_regex:
- "^User-Agent: claude-code\\/"
in the deployment's litellm_params. Works alongside existing `tags` routing;
exact tag match takes precedence over regex match. Unmatched requests fall
through to deployments tagged `default`.
The matched deployment, pattern, and user_agent are recorded in
`metadata["tag_routing"]` so they flow through to SpendLogs automatically.
* fix(tag_regex): address backwards-compat, metadata overwrite, and warning noise
Three issues from code review:
1. Backwards-compat: `has_tag_filter` was widened to activate on any non-empty
User-Agent, which would raise ValueError for existing deployments using plain
tags without a `default` fallback. Fix: only activate header-based regex
filtering when at least one candidate deployment has `tag_regex` configured.
2. Metadata overwrite: `metadata["tag_routing"]` was overwritten for every
matching deployment in the loop, leaving inaccurate provenance when multiple
deployments match. Fix: write only for the first match.
3. Warning noise: an invalid regex pattern logged one warning per header string
rather than once per pattern. Fix: compile first (catching re.error once),
then iterate over header strings.
Also adds two new tests covering these cases, and adds docs page for
tag_regex routing with a Claude Code walk-through.
* refactor(tag_regex): remove unnecessary _healthy_list copy
* docs: merge tag_regex section into tag_routing.md, remove standalone page
- Add ## Regex-based tag routing (tag_regex) section to existing
tag_routing.md instead of a separate page
- Remove tag_regex_routing.md standalone doc (odd UX to have a separate
page for a sub-feature)
- Remove proxy/tag_regex_routing from sidebars.js
- Add match_any=False debug warning in tag_based_routing.py when regex
routing fires under strict mode (regex always uses OR semantics)
* fix(tag_regex): address greptile review - security docs, strict-mode enforcement, validation order
- Strengthen security note in tag_routing.md: explicitly state User-Agent
is client-supplied and can be set to any value; frame tag_regex as a
traffic classification hint, not an access-control mechanism
- Move tag_regex startup validation before _add_deployment() so an invalid
pattern never leaves partial router state
- Enforce match_any=False strict-tag policy: when a deployment has both
tags and tag_regex and the strict tag check fails, skip the regex fallback
rather than silently bypassing the operator's intent
- Extract per-deployment match logic into _match_deployment() helper to
keep get_deployments_for_tag() readable
- Add two new tests: strict-mode blocks regex fallback, regex-only
deployment still matches under match_any=False
* fix(ci): apply Black formatting to 14 files and stabilize flaky caplog tests
- Run Black formatter on 14 files that were failing the lint check
- Replace caplog-based assertions in TestAliasConflicts with
unittest.mock.patch on verbose_logger.warning for xdist compatibility
- The caplog fixture can produce empty text in pytest-xdist workers
in certain CI environments, causing flaky test failures
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: complexity_router fails on list-format message content (OpenAI multi-part messages)
When a client sends messages with list-format content
(e.g. [{"type": "text", "text": "..."}] as used by the OpenAI JS SDK
and other clients), the complexity_router's async_pre_routing_hook
skipped those messages because it only handled str content. This caused
user_message to be None, the hook returned None, and the router fell
through to selecting the complexity_router deployment itself
(model="auto_router/complexity_router") which litellm cannot dispatch,
resulting in LiteLLMUnknownProvider.
Fixes:
- Extract text from list-format content parts (type=text) before
classifying
- Return default_model instead of None when no user message can be
extracted, preventing the crash fallthrough
- Loosen PreRoutingHookResponse.messages type from Dict[str, str] to
Dict[str, Any] to accommodate list-format content values
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: update messages type annotation in async_pre_routing_hook to Dict[str, Any]
Consistent with PreRoutingHookResponse.messages type change and the
list-format content support added in the previous commit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: normalize None content to empty string in complexity_router message parsing
msg.get("content", "") returns None when the key exists with value None
(e.g. assistant messages with tool calls). Use `or ""` to normalize
None to an empty string explicitly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: strip whitespace from joined list content parts in complexity_router
Prevents leading/trailing spaces when some content parts have empty
text values (e.g. " ".join(["", "hello"]) → " hello").
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix: avoid NoneType AttributeError when extracting tags
I've been running into this error:
```
21:47:08 - LiteLLM:ERROR: litellm_logging.py:2396 - LiteLLM.LoggingError: [Non-Blocking] Exception occurred while success logging Traceback (most recent call last):
File "/usr/lib/python3.13/site-packages/litellm/litellm_core_utils/litellm_logging.py", line 2312, in async_success_handler
await callback.async_log_success_event(
...<6 lines>...
)
File "/usr/lib/python3.13/site-packages/litellm/router_strategy/budget_limiter.py", line 396, in async_log_success_event
request_tags = _get_tags_from_request_kwargs(kwargs)
File "/usr/lib/python3.13/site-packages/litellm/router_strategy/tag_based_routing.py", line 144, in _get_tags_from_request_kwargs
return _metadata.get("tags", [])
^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'
```
This makes the function more resilient without resorting to try catch.
* add tests
Signed-off-by: Max Falk <gmdfalk@gmail.com>
---------
Signed-off-by: Max Falk <gmdfalk@gmail.com>
* refactor: comment out circuit breaker
causes incorrect rate limiting in high traffic
* fix(base_routing_strategy.py): don't reset value if redis val is lower than current in-memory value
Fixes issue where redis might be trailing in-memory value
* fix(parallel_request_limiter_v2.py): if in-memory higher than redis, don't reset value; add previous slot keys to redis increment to correctly 'get' them
* fix(parallel_request_limiter_v3.py): v3 implementation of parallel request limiter
does not use background redis syncing - increments redis in call
simplify rate limiting logic, to improve accuracy
* fix: fix ruff errors
* fix(parallel_request_limiter_v3.py): don't decrement limit on post call success - causes double decrements
* fix(parallel_request_limiter_v3.py): working accurate multi-instance logic
ensured just 100 requests allowed on 100 users, 10 ramp up, 100 rpm limit key, 2 instances
* fix(parallel_request_limiter_v3.py): working accurate rate limiting with time window resets
allows rate limiting to work across multiple windows
* test: add unit tests for v3 rate limiter
* fix(parallel_request_limiter_v3.py): return window value into in-memory cache
allows in-memory cache checks to be used correctly
* refactor(parallel_request_limiter_v3.py): refactor rate limiting to work for multiple window/counter key pairs
enables using for user/team/model rate limiting
* feat(parallel_request_limiter_v3.py): working rate limiting, across key/user/team/end-user
* fix(parallel_request_limiter_v3.py): add model specific rate limiting
* fix(parallel_request_limiter_v3.py): ignore if no rate limits set
skip unecessary rate limit checks - if no limits set
* fix(parallel_request_limiter_v3.py): initial commit bringing token rate limits back
* fix(parallel_request_limiter_v3.py): increment by value in list + update assertions to handle tokens + max parallel requests
* test(parallel_request_limiter_v3.py): more testing
* fix(parallel_request_limiter.py): working in-memory cache limiter
* fix(redis_cache.py): ignore linting error - use safe hasattr
* fix(parallel_request_limiter_v3.py): fix linting error
* refactor: remove redundant parallel_Request_limiter_v2.py
old / inaccurate implementation
* test: update tests
* style: cleanup
* test: update test
* docs(config_settings.md): document new env var
* test(test_base_routing_strategy.py): update test
* fix(base_routing_strategy.py): compress increments to redis - reduces write ops
* fix(base_routing_strategy.py): make get and reset in memory keys atomic
* fix(base_routing_strategy.py): don't reset keys - causes discrepency on subsequent requests to instance
* fix(parallel_request_limiter.py): retrieve values of previous slots from cache
more accurate rate limiting with sliding window
* fix: fix test
* fix: fix linting error
* feat(parallel_request_limiter_v2.py): add sliding window logic
allows rate limiting to work across minutes
* fix(parallel_request_limiter_v2.py): decrement usage on rate limit error
* fix(base_routing_strategy.py): fix merge from redis - preserve values in in-memory cache during gap b/w push to redis and read from redis
* fix(base_routing_strategy.py): catch the delta change during redis sync
ensures values are kept in sync
* fix(parallel_request_limiter_v2.py): update tpm tracking to use slot key logic
* fix: fix linting error
* test: update testing
* test: update tests
* test: skip on rate limit or internal server errors
* test: use pytest fixture instead
* test: bump mistral model
