Two fail-safes for the /v1/messages → Bedrock Invoke pass-through so new
Anthropic-only extensions Claude Code starts sending can't reach Bedrock
and trigger a 400 "Extra inputs are not permitted":
1. Top-level body fields are filtered to a typed allowlist. New
`BedrockInvokeAnthropicMessagesRequest` TypedDict (in
`litellm/types/llms/bedrock.py`) captures the Bedrock Invoke Anthropic
Messages body schema; the runtime allowlist is derived from its
`__annotations__` so the type and the filter can't drift. Anchored to
the AWS reference page in docstrings + transform comment. An
exact-set test pins the resolved allowlist so any future edit forces
conscious review.
Drops context_management, output_config, speed, mcp_servers,
container, inference_geo, internal litellm_metadata, and any future
Anthropic addition. output_format stays as an active inline-schema
conversion (not just a strip).
2. The anthropic-beta header list is filtered + transformed against the
bedrock mapping for ALL betas, not just auto-injected ones. The
previous code union'd user-provided betas back in unfiltered, so a
client on a new Anthropic-direct beta (e.g. advisor-tool-…,
context-management-…) could still pin the request to fail. In a proxy
context the client can't know the backend is Bedrock; the provider
mapping is authoritative. User-provided drops are logged at WARNING
so intentional overrides leave a breadcrumb.
Updates one existing test that happened to assert on the old buggy
pass-through (it used output-128k-2025-02-19, which is null in the
bedrock mapping and would 400 at runtime); rewrote it against a
bedrock-supported beta.
Scope: messages/invoke only. The same user-beta bypass exists in
chat/invoke but that's a different code path with different
user-expectation trade-offs — follow-up.
Add a router utils unit test that directly exercises _filter_deployments_by_model_access_groups for access-group-only key permissions.
Made-with: Cursor
Filter router candidate deployments by caller-authorized model access groups when access is granted via group membership, preventing cross-group load balancing for shared public model names.
Made-with: Cursor
VertexAIImagenImageEditConfig.get_complete_url was resolving vertex_project
and vertex_location only from env vars and global settings, ignoring
litellm_params. Users supplying project/location exclusively via YAML
config would get a ValueError or wrong URL even after auth headers were fixed.
Mirrors the pattern already used by VertexAIGeminiImageEditConfig and
image_generation counterpart (safe_get_vertex_ai_project/location).
Also fixes api_key type hint in MockImageEditConfig (str -> Optional[str])
and adds a test covering get_complete_url credential resolution.
Made-with: Cursor
Adds three test cases to prevent regression of the Vertex AI image_edit
credentials bug:
1. test_validate_environment_signature_includes_litellm_params: ensures
all image-edit configs accept litellm_params (contract for the handler)
2. test_vertex_gemini_image_edit_reads_credentials_from_litellm_params:
verifies Gemini config reads from litellm_params first
3. test_vertex_imagen_image_edit_reads_credentials_from_litellm_params:
verifies Imagen config reads from litellm_params first
These tests catch if the fix is accidentally reverted or if new image-edit
configs are added without the litellm_params parameter.
Made-with: Cursor
When aimage_edit or image_edit was called with Vertex AI Gemini/Imagen models
via YAML-style config (vertex_project / vertex_credentials in proxy YAML),
the credentials were dropped during handler-to-config plumbing, causing
fallback to Application Default Credentials and DefaultCredentialsError.
Root cause: image_edit_handler and async_image_edit_handler did not pass
litellm_params to validate_environment, unlike image_generation_handler.
Fixes:
1. Widen BaseImageEditConfig.validate_environment signature to accept
litellm_params and api_base (optional kwargs).
2. Forward dict(litellm_params) and litellm_params.api_base from both
sync and async image_edit handlers to validate_environment.
3. Update VertexAIImagenImageEditConfig.validate_environment to read
vertex_ai_project/vertex_ai_credentials from litellm_params first,
matching Gemini config pattern (secondary latent bug fix).
4. Widen all image-edit config override signatures to match base.
Made-with: Cursor
Addresses review feedback on the snapshot approach:
1. Class-instance mutable state
The snapshot only covers primitives + collections + None. Class
instances (DualCache, LLMClientCache) weren't reset between tests,
so in-place cache mutations could leak. Can't deepcopy these — they
hold thread locks — but they expose flush_cache(). Collect every
module attribute whose value implements flush_cache() at conftest
import, and invoke it per-test alongside the snapshot restore.
2. Silent skips are now warnings
_snapshot_mutable_state and _restore_mutable_state previously
swallowed exceptions, so if a future attr gained a property without
a setter (or other non-round-trippable state), an isolation gap
would have no signal. Emit warnings.warn on each failure path.
3. Docstring
Explicitly documents what IS and IS NOT reset, and tells authors to
use monkeypatch.setattr() for in-place mutations of instances
without flush_cache() (ProxyLogging, JWTHandler, etc.).
The previous snapshot only tracked list/dict/set values. Tests mutate
scalar module attrs too — master_key, premium_user, prisma_client — and
importlib.reload used to reset those implicitly. Under the snapshot
approach they were leaking between tests, so test_active_callbacks
failed in CI with "No api key passed in." once an earlier test left
master_key set to sk-1234.
Expand the snapshot to cover primitives (str/int/float/bool/bytes/tuple)
and None-valued attributes. Complex object instances are still skipped
to avoid deepcopy issues.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Add regression tests that mock make_bedrock_api_request and verify
input_type=request uses source=INPUT with user messages, and
input_type=response uses source=OUTPUT with synthetic ModelResponse.
Made-with: Cursor
tests/proxy_unit_tests/conftest.py was calling importlib.reload(litellm) in an
autouse function-scoped fixture, which cost ~17s per test because it re-ran
the full litellm __init__ import chain. With 400+ proxy unit tests, this was
the single biggest driver of CI wall time — 18 of the top 20 slowest durations
in a typical run were just the 17s fixture setup.
Replace the reload with a snapshot-and-restore approach: snapshot the mutable
lists/dicts/sets on litellm and litellm.proxy.proxy_server once at conftest
import, then deep-copy that snapshot back before each test. Callback lists,
caches, router state, etc. still get reset between tests, but the expensive
import chain only runs once per worker.
Local measurement on test_proxy_utils.py: 188 tests in 3.50s (previously took
~15 minutes of CI wall time on a single worker).
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
- _owner_cache now opportunistically sweeps expired entries past
_OWNER_CACHE_SWEEP_THRESHOLD live entries. Previously sessions that never
came back piled up forever.
- flush_session_to_db strips session_id/router_name/model_name from the update
payload. Prisma rejects writes to @@id fields.
- record_turn no longer persists last_user_content / last_assistant_content /
tool_call_history / pending_tool_calls. Those are needed only in-memory for
the next turn's signal detection; writing user prompts and tool payloads to
the DB would store PII for every conversation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- SessionState now carries clean_credit_awarded + last_processed_turn (matching
the DB schema). Satisfaction only fires once per session AND only after
MIN_TURNS_FOR_CLEAN_CREDIT turns of context — early "thanks" no longer
inflates alpha.
- _detect_failure no longer treats empty content as failure. Many tools
legitimately return empty output (zero-result searches, silent bash);
penalizing those corrupted the bandit posterior. Only is_error fires now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(router): add auto_router/quality_router for quality-tier routing (#25987)
* feat(router): add auto_router/quality_router for quality-tier routing
Adds a new auto-router type that routes a request to a model at a target
quality tier. The quality tier is inferred by re-using the existing
ComplexityRouter's classification, then mapped through an admin-configured
complexity_to_quality table. Each candidate model declares its own
quality_tier in model_info.litellm_routing_preferences.
Resolution strategy: exact tier match, else round up to the next higher
tier, else fall back to default_model.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): add capability-based filtering
Each deployment can declare a `capabilities: List[str]` field in
`model_info.litellm_routing_preferences` (e.g. ["vision",
"function_calling"]). Requests can pass `litellm_capabilities` in
`request_kwargs` to require specific capabilities — the router will only
route to deployments whose declared capabilities are a superset.
Resolution still walks tier (exact → round up), but at each tier filters
by capability before picking. Falls back to default_model only when it
also satisfies the required capabilities; otherwise raises rather than
silently routing to a model that lacks a required capability.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): expose routing decision in response headers
For transparency, expose the QualityRouter's routing decision in the
proxy response headers:
x-litellm-quality-router-model → picked model_name (e.g. "haiku-vision")
x-litellm-quality-router-tier → resolved quality tier (e.g. "1")
x-litellm-quality-router-complexity → ComplexityTier name (e.g. "SIMPLE")
Mechanism: the pre-routing hook stashes the decision in
request_kwargs["metadata"]["quality_router_decision"]. After the call
returns, Router.set_response_headers lifts the decision into
response._hidden_params["additional_headers"] alongside the existing
x-litellm-model-group / x-litellm-model-id headers. Existing metadata
keys (trace_id, user_id, etc.) are preserved.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): replace capabilities with keyword override
Drops the capability-based filtering in favor of a keyword-based override
for v0:
- RoutingPreferences.keywords: List[str] (replaces capabilities) — each
deployment can declare substring keywords.
- If any declared keyword (case-insensitive) appears in the user message,
the router short-circuits the complexity-classification flow and routes
to the matching deployment.
- Tiebreaker for overlapping keyword matches: quality_tier DESC, then
cheapest model_info.input_cost_per_token ASC. Unpriced models lose ties
to priced ones.
Decision metadata + headers now expose the override:
x-litellm-quality-router-via → "keyword" | "quality_tier"
x-litellm-quality-router-keyword → matched keyword (only on keyword route)
x-litellm-quality-router-complexity → complexity tier (only on tier route)
Removes:
- request_kwargs["litellm_capabilities"] reading
- _model_capabilities, _model_supports_capabilities,
_first_capable_model_at_tier, capability filter in
_resolve_model_for_quality_tier
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): add explicit `order` to RoutingPreferences
Adds an explicit priority field to RoutingPreferences for resolving
collisions deterministically:
RoutingPreferences.order: Optional[int] # lower wins; unset = +inf
Used as the PRIMARY tiebreaker in two places:
1. Keyword overlap: when multiple deployments declare the same matching
keyword, sort by (order ASC, quality_tier DESC, input_cost_per_token
ASC, model_name ASC). Explicit always beats implicit.
2. Tier resolution: when multiple deployments share a quality tier,
`_resolve_model_for_quality_tier` picks the one with the lowest
order. The tier list is now sorted at index-build time.
This lets admins make routing decisions explicit when the natural
quality-and-price ordering would pick the wrong model.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(quality_router): reorder tiebreak to (quality, order, price)
Changes the tiebreak ordering so quality_tier always wins first, then
explicit `order` is used to break ties within the same tier, then price
breaks the rest:
1. quality_tier DESC ← best model wins first
2. order ASC ← explicit priority within a tier
3. input_cost_per_token ASC
4. model_name ASC
Previously `order` was the primary key — that meant a tier-2 model with
`order=1` would beat a tier-3 model with no `order`, which is the wrong
default. Now `order` only resolves collisions among same-tier candidates.
Tier resolution (within a single tier) keeps the same key minus quality:
(order ASC, cost ASC, name).
Test renames + flips:
- test_explicit_order_overrides_quality_tier → test_quality_wins_over_explicit_order
- new: test_order_breaks_tie_within_same_quality_tier
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* fix(quality_router): resolve Greptile review feedback
Addresses four P1 findings from PR review plus test coverage:
1. set_model_list missing quality_routers reset
- Hot-reloading the Router would leave stale QualityRouter instances
pointing at the old model_list. `set_model_list` now clears
`self.quality_routers` alongside the other indices.
2. Round-down fallback before default_model
- `_resolve_model_for_quality_tier` now rounds DOWN to the closest
lower tier after round-up fails, before falling back to
`default_model`. Degrades gracefully rather than jumping straight
off-tier.
3. RoutingPreferences validation bypass
- `_build_tier_index` now instantiates `RoutingPreferences(**prefs)`
so invalid shapes (e.g. non-int quality_tier) raise a clear
ValueError instead of silently succeeding.
4. Config-ordering dependency
- `_tier_to_models` is now built lazily on first access. Previously,
eager construction in `__init__` meant a QualityRouter deployment
had to appear AFTER all its referenced models in config.yaml,
because `Router._create_deployment` populates `model_list`
incrementally. Any `available_models` defined after the router
entry would silently be reported as missing.
Also adds 6 new tests covering each fix:
- test_invalid_quality_tier_type_raises_clear_error
- test_router_can_be_instantiated_before_its_targets_exist
- test_set_model_list_clears_quality_routers_registry
- test_rounds_down_when_no_higher_tier_exists
- test_rounds_down_prefers_closest_lower_tier
- test_prefers_round_up_over_round_down
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* style: apply black 24.10.0 formatting to pre-existing offenders
Unblocks the LiteLLM Linting check for this PR — these 12 files are already
failing `black --check` on main (the lint workflow only runs on PRs, so main
drifts). No behavior changes; formatting-only.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Update litellm/router.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Support /v1/responses in complexity router (#26137)
* feat(proxy): add --reload flag for uvicorn hot reload (dev only)
Opt-in CLI flag, off by default, no env var. Only affects the uvicorn
run path; gunicorn/hypercorn paths and prod (which doesn't pass the
flag) are unaffected.
* Feature/add audio support for scaleway (#26110)
* feat(scaleway): add SCALEWAY to LlmProviders enum
* feat(scaleway): add audio transcription config and dispatch wiring
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(scaleway): add behavior tests for audio transcription config
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* chore(scaleway): advertise audio_transcriptions in endpoint-support JSON
* docs(scaleway): document audio transcription support
* fix(scaleway): address PR review — plain-text response_format + missing-key fail-fast
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(scaleway): cover new response paths, drop gettysburg.wav coupling
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Prompt Compression - add it to the proxy (#25729)
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Support /v1/responses in complexity router
Adds cross-format support to the complexity router via the guardrail
translation handler dispatch. Adds get_structured_messages to base
translation plus OpenAI chat, Responses, and Anthropic handlers.
Auto-router helper _extract_text_from_messages handles tool-call and
multimodal messages. Widens async_pre_routing_hook messages type to
Dict[str, Any].
Fixes https://github.com/BerriAI/litellm/issues/25134
* chore: apply black formatting
* fix: fallback to trying each handler when route inference fails
---------
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* test: cover _is_quality_router_deployment and init_quality_router_deployment
* fix: reset auto_routers on set_model_list to prevent hot-reload ValueError
* style: apply black formatting to websearch_interception and agentic_streaming_iterator
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
Six tests in test_hooks.py were written against an older API and had been
failing in CI. Updated:
- test_resolve_session_key_* (4 tests): _resolve_session_key now requires
at least SIGNAL_GATE_MIN_MESSAGES messages before deriving a hash (it
returns None on shorter convos to match the signal-processing gate).
Switched the tests to use _long_messages() so they hit the hash path.
- test_post_call_success_hook_* (2 tests): the hook was migrated from
async_post_call_success_hook (mutates response._hidden_params) to
async_post_call_response_headers_hook (returns a headers dict) because
the former fires too late for streaming responses. Rewrote the tests
against the new API; added a metadata-not-dict noop case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Anthropic retired claude-3-haiku-20240307 on 2026-04-20, causing the
test_anthropic_messages_litellm_router_non_streaming_with_logging
test to 404. Update the model references in this file to the current
pinned haiku version.
* fix: /health/readiness returns 503 when DB is unreachable due to handle_db_exception re-raising
handle_db_exception() re-raises the Prisma exception inside _db_health_readiness_check's
except block, which propagates out to health_readiness() and gets wrapped in a 503.
The health endpoint never reached the reconnect path and the service never recovered.
Fix:
- Remove handle_db_exception() call from _db_health_readiness_check — that helper is
for API request handlers (allow_requests_on_db_unavailable flag), not health checks
- Replace raw disconnect()+connect() with attempt_db_reconnect(), which uses the proper
lock, cooldown, escalation, and heavy-reconnect (recreate_prisma_client) machinery
* test: update health readiness tests for handle_db_exception removal
- Remove tests that expected handle_db_exception to re-raise (old buggy behaviour)
- Remove tests asserting disconnect()/connect() calls (replaced by attempt_db_reconnect)
- Add regression tests covering the 503 loop fix:
- transport errors never raise (ClientNotConnectedError, httpx.ConnectError, etc.)
- reconnect success path returns 'connected'
- reconnect failure path returns 'disconnected' without raising
- non-transport errors return 'disconnected', skip reconnect
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
The post-call hook was hardcoding tool_results=[] on every Turn, so the
failure detector never saw tool errors and the bandit only learned from
satisfaction — never from negative tool outcomes.
Added _recent_tool_results(messages): walks the request messages from the
tail and collects the contiguous run of role=='tool' entries — those are
the results from the most recent assistant tool_calls round. Normalizes
each to {content, is_error}, the only fields signals._detect_failure /
_detect_exhaustion read.
Tests: 6 new covering empty input, trailing-run extraction, is_error
propagation, boundary at first non-tool message, no-trailing-tool case,
and the end-to-end path from hook -> Turn.tool_results.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Use 'auto_router/adaptive_router' prefix in example yaml, docs, and
README — the old 'adaptive_router/...' and 'openai/gpt-4o-mini' values
silently skipped adaptive-router init because detection requires the
'auto_router/adaptive_router' prefix.
- Read x-litellm-min-quality-tier from request headers (and the
'min_quality_tier' metadata key as fallback) in async_pre_routing_hook.
Previously the documented header was defined but never extracted, so
the quality-floor feature was inert.
- Evict expired entries from _session_states. The cache grew without
bound — added a parallel expiry map (same TTL as _owner_cache) and an
opportunistic bulk sweep when the cache crosses a size threshold.
- Align adaptive-router migration SQL with Prisma schema: all count
columns and the 'clean_credit_awarded' / 'last_processed_turn' fields
are NOT NULL in the data model, so the migration now declares them
NOT NULL. Fixes test_aaaasschema_migration_check.
Tests: 8 new covering header/metadata/precedence/invalid-value paths for
min_quality_tier and TTL-based eviction of _session_states.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Use LoggingCallbackManager.add_litellm_callback instead of
litellm.callbacks.append (required by callback_manager_test)
- init_adaptive_router_deployment now uses model_name_to_deployment_indices
for O(k) lookup instead of scanning model_list
- Rephrase comment in set_model_list to avoid the 'in self.model_list'
substring that the linear-scan test greps for
- Whitelist _finalize_adaptive_router_if_configured in
test_no_linear_scans_in_router — prefix match on 'auto_router/adaptive_router'
has no supporting index; runs once at init
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Router coverage check flagged this method as untested. Adds two cases:
- initializes AdaptiveRouter from model_list and is idempotent on re-entry
- no-op when no adaptive deployments are configured
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Vertex generateContent returns INVALID_ARGUMENT if cachedContent is sent
with system_instruction, tools, or toolConfig; those belong on CachedContent.
Fixes#26014
Made-with: Cursor
Per-user OAuth MCP requests now only skip pre-emptive 401 when a stored token is available, preserving token-reuse behavior while restoring fast PKCE kickoff for first-time or missing-token users.
Addresses Greptile P1: `metadata: null` on a key with a reserved field
was crashing inside prepare_metadata_fields because cast(dict, None) is
a runtime no-op, so the loop hit `reserved_field in None` and returned
500 instead of 400.
Extend the rejection condition to cover `casted_metadata is None`, so
attempts to clear an immutable field via `metadata: null` return 400
consistently with the overwrite path. Non-service-account keys still
fall through to the existing "clear all metadata" behavior.
Addresses Greptile review feedback:
- Clarify LiteLLM_Reserved_Metadata_Fields comment to describe both
preserve-on-omit and reject-on-change behaviors.
- Treat explicit null as a change attempt so callers trying to clear
service_account_id get a 400 instead of a silent no-op.
`should_create_missing_views()` had `and result[0]["reltuples"]` which is
falsy when reltuples=0. On a fresh empty PostgreSQL table, CREATE INDEX sets
reltuples=0, causing the guard to return False and skip view creation entirely.
Views like MonthlyGlobalSpendPerKey are never created, and the
/global/spend/logs endpoint returns 500.
Fix: change to `and result[0]["reltuples"] is not None` so reltuples=0
(empty table) and reltuples=-1 (unanalyzed table) both correctly return True.
Also harden test_vertex_ai.py to return None instead of crashing with
JSONDecodeError when the spend-logs endpoint returns a non-JSON 500 response,
and add unit tests covering all three reltuples branches (0, -1, positive).
