The previous revert left three files under
_experimental/out/_next/static/8Gn6tA2K4jsPxzCCMwObH/ still staged
(git checkout origin/main -- only restores files present in main, it
does not delete files that exist only on the PR branch). Remove them
so the PR no longer touches _experimental/out/ at all.
Anthropic retired claude-3-haiku-20240307 on 2026-04-20, causing the
test_anthropic_messages_litellm_router_non_streaming_with_logging
test to 404. Update the model references in this file to the current
pinned haiku version.
The UI rebuild bundled into this PR is not needed for the Dockerfile
change — the image simply copies whatever _experimental/out/ is in the
tree. Regenerating here conflicts with the release-time refresh policy
and adds ~100 files of review noise / merge-conflict risk for any
concurrent UI PR.
* fix: /health/readiness returns 503 when DB is unreachable due to handle_db_exception re-raising
handle_db_exception() re-raises the Prisma exception inside _db_health_readiness_check's
except block, which propagates out to health_readiness() and gets wrapped in a 503.
The health endpoint never reached the reconnect path and the service never recovered.
Fix:
- Remove handle_db_exception() call from _db_health_readiness_check — that helper is
for API request handlers (allow_requests_on_db_unavailable flag), not health checks
- Replace raw disconnect()+connect() with attempt_db_reconnect(), which uses the proper
lock, cooldown, escalation, and heavy-reconnect (recreate_prisma_client) machinery
* test: update health readiness tests for handle_db_exception removal
- Remove tests that expected handle_db_exception to re-raise (old buggy behaviour)
- Remove tests asserting disconnect()/connect() calls (replaced by attempt_db_reconnect)
- Add regression tests covering the 503 loop fix:
- transport errors never raise (ClientNotConnectedError, httpx.ConnectError, etc.)
- reconnect success path returns 'connected'
- reconnect failure path returns 'disconnected' without raising
- non-transport errors return 'disconnected', skip reconnect
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
The post-call hook was hardcoding tool_results=[] on every Turn, so the
failure detector never saw tool errors and the bandit only learned from
satisfaction — never from negative tool outcomes.
Added _recent_tool_results(messages): walks the request messages from the
tail and collects the contiguous run of role=='tool' entries — those are
the results from the most recent assistant tool_calls round. Normalizes
each to {content, is_error}, the only fields signals._detect_failure /
_detect_exhaustion read.
Tests: 6 new covering empty input, trailing-run extraction, is_error
propagation, boundary at first non-tool message, no-trailing-tool case,
and the end-to-end path from hook -> Turn.tool_results.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add supported providers to prompt caching doc
* Move Z.ai / GLM to cache_control marker list
* Mark xAI models as supporting prompt caching
* Narrow xAI prompt caching flag to models with documented cache pricing
* Add prompt caching flag to grok-4, grok-4-0709, grok-4-latest
---------
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local>
- Use 'auto_router/adaptive_router' prefix in example yaml, docs, and
README — the old 'adaptive_router/...' and 'openai/gpt-4o-mini' values
silently skipped adaptive-router init because detection requires the
'auto_router/adaptive_router' prefix.
- Read x-litellm-min-quality-tier from request headers (and the
'min_quality_tier' metadata key as fallback) in async_pre_routing_hook.
Previously the documented header was defined but never extracted, so
the quality-floor feature was inert.
- Evict expired entries from _session_states. The cache grew without
bound — added a parallel expiry map (same TTL as _owner_cache) and an
opportunistic bulk sweep when the cache crosses a size threshold.
- Align adaptive-router migration SQL with Prisma schema: all count
columns and the 'clean_credit_awarded' / 'last_processed_turn' fields
are NOT NULL in the data model, so the migration now declares them
NOT NULL. Fixes test_aaaasschema_migration_check.
Tests: 8 new covering header/metadata/precedence/invalid-value paths for
min_quality_tier and TTL-based eviction of _session_states.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Use LoggingCallbackManager.add_litellm_callback instead of
litellm.callbacks.append (required by callback_manager_test)
- init_adaptive_router_deployment now uses model_name_to_deployment_indices
for O(k) lookup instead of scanning model_list
- Rephrase comment in set_model_list to avoid the 'in self.model_list'
substring that the linear-scan test greps for
- Whitelist _finalize_adaptive_router_if_configured in
test_no_linear_scans_in_router — prefix match on 'auto_router/adaptive_router'
has no supporting index; runs once at init
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
prisma --version invokes the Schema Engine, which has no binary for the
Wolfi base image (only debian). In the baseline this was silenced by a
trailing || true wrapping the whole prisma chain; removing that wrapper
uncovered the failure on arm64 builds. The main Dockerfile does not
call prisma --version at all, so drop it here to match — prisma generate
is sufficient to validate the toolchain.
Revert the .dockerignore ui/ exclusion and remove the UI Drift Guard
workflow. _experimental/out/ refresh is already handled by the release
runbook; the global .dockerignore change also broke Dockerfile.custom_ui
(explicit COPY ./ui/litellm-dashboard) and the enterprise-colors inline
rebuild path in Dockerfile, Dockerfile.database, and Dockerfile.dev.
Dockerfile.non_root itself is unchanged functionally — still stages the
UI from the checked-in _experimental/out/. Only the companion workflow
and global dockerignore exclusion are dropped.
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Router coverage check flagged this method as untested. Adds two cases:
- initializes AdaptiveRouter from model_list and is idempotent on re-entry
- no-op when no adaptive deployments are configured
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unrelated timestamp and version drift was showing in the PR diff. This PR adds no new deps — keep uv.lock identical to main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove _experimental/out/ changes from this PR — these are auto-generated Next.js build outputs, not part of the adaptive router feature.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Vertex generateContent returns INVALID_ARGUMENT if cachedContent is sent
with system_instruction, tools, or toolConfig; those belong on CachedContent.
Fixes#26014
Made-with: Cursor
Five small, individually-verified cleanups collected into one commit:
- Drop 'prisma migrate diff --from-empty ... > /dev/null 2>&1 || true'
from the builder. Stdout/stderr/exit-status all discarded; nothing
reads the output. Dead line.
- Drop 'mkdir -p /app/.cache/npm' from the same RUN. npm is gone.
- Drop the runtime's redundant 'sed -i' + 'chmod +x' on the entrypoint
scripts. The builder already does the same three lines, and the
runtime copies /app from the builder via COPY --from=builder, so
the normalized files (and exec bits, which buildkit preserves) are
already in place.
- Drop NPM_CONFIG_CACHE and NPM_CONFIG_PREFER_OFFLINE from the runtime
ENV — nothing reads them after Task 2.2 removed npm.
- Drop '/.npm' and '/tmp/.npm' from the runtime's mkdir + chown. These
directories only existed as npm's writable dirs for the non-root
user; npm is gone.
.dockerignore: add 'ui/'. After Task 2.1 the non_root image sources
its UI bytes from litellm/proxy/_experimental/out/, so the whole
ui/litellm-dashboard/ source tree is dead weight when the blanket
'COPY . .' pulls it into /app. Verified (with ripgrep) that no Python
code under litellm/ opens any file under ui/. All string references to
'ui/...' are URL paths, not filesystem paths.
Final image size: 6.57GB baseline -> 1.96GB. API parity and UI visual
regression match baseline across all 12 API scenarios and 10 UI
routes. Trivy HIGH/CRITICAL: 6 -> 2, no new CVEs introduced.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Mount /app/.cache/uv as a BuildKit type=cache on both 'uv sync' steps.
The cache persists across builds on the same builder (and, when used
with type=gha in CI, across CI runs) so repeat builds don't re-download
every wheel.
Side-effect: because the cache lives outside the image layer, the
~742MB of downloaded wheel archives that were previously baked into
/app/.cache/uv drop out of the final image. Compressed image size
goes from ~5.0GB to ~3.7GB, and the 'USER nobody' prisma-generate
layer is 1.7GB vs 2.4GB.
Warm-build timing: a uv-sync-invalidating edit now takes ~1m30s vs
~2m39s without the cache mount, on this dev VM.
API parity and UI visual regression continue to match baseline.
Trivy HIGH/CRITICAL: 6 at baseline -> 2 now, no new CVEs.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
After Task 2.1 removed the in-image Next.js build, the builder stage no
longer needs a full C/C++ + Clang toolchain. Keep gcc + python3-dev
(required to compile ml-dtypes 0.4.1 from source — no wheel published
for Python 3.13 yet). Drop everything else.
Removed from apk: clang, llvm, lld, linux-headers, build-base,
openssl-dev, npm. Removed NVM_DIR env and /root/.nvm from PATH
(no nvm-based Node install anymore).
Kept: python3, python3-dev, gcc, bash, coreutils, curl, openssl,
libsndfile, nodejs. gcc (15.2) serves both C and C++; the separate
g++ package doesn't exist in Wolfi.
Image size unchanged (builder stage doesn't end up in the runtime);
cold builds slightly slower due to ml-dtypes source compile, but that
will be recovered in the next task via a BuildKit uv cache mount.
API parity and UI visual regression both match baseline, Trivy
HIGH/CRITICAL CVE count unchanged from opt-2 (4 CVEs, none new).
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
npm was installed in the runtime only to globally install vulnerability
patched versions of tar/glob/brace-expansion/minimatch/diff and to
in-place rewrite npm's own bundled package.json. Both were to silence
CVE scanners against modules that ship with npm itself.
Since we no longer run npm anywhere in the runtime (Prisma uses the
node binary directly for migrate deploy and generate), we can just
skip installing npm in the first place. This eliminates both the
~25-line CVE-patch shuffle AND the underlying CVE surface.
Kept: nodejs (needed by prisma-python's CLI and migrate deploy).
Removed: npm apk package, all 'npm install -g', all find+sed patching,
the redundant 'apk upgrade --no-cache nodejs' (already covered by the
preceding 'apk upgrade').
Image: 4.97GB (opt-1) -> 4.97GB (opt-2); the real win is that two
CVEs (CVE-2026-33671 and GHSA-q4gf-8mx6-v5v3) drop off the Trivy
HIGH/CRITICAL list. No new CVEs introduced. API parity and UI
visual regression both match baseline.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
The checked-in Next.js static export at litellm/proxy/_experimental/out/
is kept fresh by the UI Drift Guard CI workflow. Stage it directly
instead of re-running npm ci + npm run build inside the image.
This removes: nvm install, node 20.20.2 install, npm ci (801 pkgs),
next build, and the resulting intermediate node_modules/out tree.
Build time: ~6m25s -> ~2m (fuse-overlayfs DinD); image 6.57GB -> 5.0GB.
Behavior parity verified: API endpoints, UI screenshots (all 10 routes
pixel-perfect), and Trivy HIGH/CRITICAL CVE count (6 -> 5, one npm
GHSA removed) all match or improve over baseline.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Adds a CI job that rebuilds the admin UI from source and fails if the
committed static export at litellm/proxy/_experimental/out/ has drifted
from what npm run build produces. This prevents silently shipping stale
UI bytes and is a prerequisite for the non_root Dockerfile streamlining
work, which will stage the UI from _experimental/out/ directly instead
of rebuilding it inside the image.
Also regenerates litellm/proxy/_experimental/out/ to match a fresh
npm run build (Node 20.20.2) — the committed tree had drifted from
source prior to this commit.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
* bump litellm-proxy-extras version to 0.4.67
* bump litellm-proxy-extras pin to 0.4.67 in litellm pyproject
* regenerate uv.lock for litellm-proxy-extras 0.4.67
* bump litellm-enterprise version to 0.1.38
* bump litellm-enterprise pin to 0.1.38 in litellm pyproject
* regenerate uv.lock for litellm-enterprise 0.1.38
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
