* Add supported providers to prompt caching doc
* Move Z.ai / GLM to cache_control marker list
* Mark xAI models as supporting prompt caching
* Narrow xAI prompt caching flag to models with documented cache pricing
* Add prompt caching flag to grok-4, grok-4-0709, grok-4-latest
---------
Co-authored-by: Michael Riad Zaky <michaelr@Michaels-MacBook-Air.local>
- Use 'auto_router/adaptive_router' prefix in example yaml, docs, and
README — the old 'adaptive_router/...' and 'openai/gpt-4o-mini' values
silently skipped adaptive-router init because detection requires the
'auto_router/adaptive_router' prefix.
- Read x-litellm-min-quality-tier from request headers (and the
'min_quality_tier' metadata key as fallback) in async_pre_routing_hook.
Previously the documented header was defined but never extracted, so
the quality-floor feature was inert.
- Evict expired entries from _session_states. The cache grew without
bound — added a parallel expiry map (same TTL as _owner_cache) and an
opportunistic bulk sweep when the cache crosses a size threshold.
- Align adaptive-router migration SQL with Prisma schema: all count
columns and the 'clean_credit_awarded' / 'last_processed_turn' fields
are NOT NULL in the data model, so the migration now declares them
NOT NULL. Fixes test_aaaasschema_migration_check.
Tests: 8 new covering header/metadata/precedence/invalid-value paths for
min_quality_tier and TTL-based eviction of _session_states.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: new agentic loop event hook
simplifies how to create logic for tool based multi llm calls
* fix: compress - make it work on anthropic input as well
* fix(compress.py): working prompt compression for claude code
ensures claude code messages can run through proxy easily
* docs: add agentic loop hook guide
* docs: add agentic_loop_hook to sidebar
* fix: fix multiple arguments error
* fix: fix tool call loop for compression on streaming /v1/messages
* fix: fix linting errors
* fix: fix ci/cd errors
* feat(litellm_pre_call_utils.py): use claude code session for litellm session id
allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation
* fix: suppress incorrect mypy warning rE: module
* revert: drop PR's changes to litellm/proxy/_experimental/out/
Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.
* fix: address greptile review comments on PR #25729
- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
Anthropic Messages rejects empty tool arrays on requests that did not
originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
the per-callback try/except so they propagate instead of being swallowed
by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
preserve backwards compatibility; explicit x-litellm-* headers are
unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
module-level binding
(litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag
Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(compress): replace input_type with CallTypes call_type
Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead. ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.
Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).
Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Use LoggingCallbackManager.add_litellm_callback instead of
litellm.callbacks.append (required by callback_manager_test)
- init_adaptive_router_deployment now uses model_name_to_deployment_indices
for O(k) lookup instead of scanning model_list
- Rephrase comment in set_model_list to avoid the 'in self.model_list'
substring that the linear-scan test greps for
- Whitelist _finalize_adaptive_router_if_configured in
test_no_linear_scans_in_router — prefix match on 'auto_router/adaptive_router'
has no supporting index; runs once at init
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
prisma --version invokes the Schema Engine, which has no binary for the
Wolfi base image (only debian). In the baseline this was silenced by a
trailing || true wrapping the whole prisma chain; removing that wrapper
uncovered the failure on arm64 builds. The main Dockerfile does not
call prisma --version at all, so drop it here to match — prisma generate
is sufficient to validate the toolchain.
Revert the .dockerignore ui/ exclusion and remove the UI Drift Guard
workflow. _experimental/out/ refresh is already handled by the release
runbook; the global .dockerignore change also broke Dockerfile.custom_ui
(explicit COPY ./ui/litellm-dashboard) and the enterprise-colors inline
rebuild path in Dockerfile, Dockerfile.database, and Dockerfile.dev.
Dockerfile.non_root itself is unchanged functionally — still stages the
UI from the checked-in _experimental/out/. Only the companion workflow
and global dockerignore exclusion are dropped.
Router coverage check flagged this method as untested. Adds two cases:
- initializes AdaptiveRouter from model_list and is idempotent on re-entry
- no-op when no adaptive deployments are configured
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unrelated timestamp and version drift was showing in the PR diff. This PR adds no new deps — keep uv.lock identical to main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove _experimental/out/ changes from this PR — these are auto-generated Next.js build outputs, not part of the adaptive router feature.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Five small, individually-verified cleanups collected into one commit:
- Drop 'prisma migrate diff --from-empty ... > /dev/null 2>&1 || true'
from the builder. Stdout/stderr/exit-status all discarded; nothing
reads the output. Dead line.
- Drop 'mkdir -p /app/.cache/npm' from the same RUN. npm is gone.
- Drop the runtime's redundant 'sed -i' + 'chmod +x' on the entrypoint
scripts. The builder already does the same three lines, and the
runtime copies /app from the builder via COPY --from=builder, so
the normalized files (and exec bits, which buildkit preserves) are
already in place.
- Drop NPM_CONFIG_CACHE and NPM_CONFIG_PREFER_OFFLINE from the runtime
ENV — nothing reads them after Task 2.2 removed npm.
- Drop '/.npm' and '/tmp/.npm' from the runtime's mkdir + chown. These
directories only existed as npm's writable dirs for the non-root
user; npm is gone.
.dockerignore: add 'ui/'. After Task 2.1 the non_root image sources
its UI bytes from litellm/proxy/_experimental/out/, so the whole
ui/litellm-dashboard/ source tree is dead weight when the blanket
'COPY . .' pulls it into /app. Verified (with ripgrep) that no Python
code under litellm/ opens any file under ui/. All string references to
'ui/...' are URL paths, not filesystem paths.
Final image size: 6.57GB baseline -> 1.96GB. API parity and UI visual
regression match baseline across all 12 API scenarios and 10 UI
routes. Trivy HIGH/CRITICAL: 6 -> 2, no new CVEs introduced.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Mount /app/.cache/uv as a BuildKit type=cache on both 'uv sync' steps.
The cache persists across builds on the same builder (and, when used
with type=gha in CI, across CI runs) so repeat builds don't re-download
every wheel.
Side-effect: because the cache lives outside the image layer, the
~742MB of downloaded wheel archives that were previously baked into
/app/.cache/uv drop out of the final image. Compressed image size
goes from ~5.0GB to ~3.7GB, and the 'USER nobody' prisma-generate
layer is 1.7GB vs 2.4GB.
Warm-build timing: a uv-sync-invalidating edit now takes ~1m30s vs
~2m39s without the cache mount, on this dev VM.
API parity and UI visual regression continue to match baseline.
Trivy HIGH/CRITICAL: 6 at baseline -> 2 now, no new CVEs.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
After Task 2.1 removed the in-image Next.js build, the builder stage no
longer needs a full C/C++ + Clang toolchain. Keep gcc + python3-dev
(required to compile ml-dtypes 0.4.1 from source — no wheel published
for Python 3.13 yet). Drop everything else.
Removed from apk: clang, llvm, lld, linux-headers, build-base,
openssl-dev, npm. Removed NVM_DIR env and /root/.nvm from PATH
(no nvm-based Node install anymore).
Kept: python3, python3-dev, gcc, bash, coreutils, curl, openssl,
libsndfile, nodejs. gcc (15.2) serves both C and C++; the separate
g++ package doesn't exist in Wolfi.
Image size unchanged (builder stage doesn't end up in the runtime);
cold builds slightly slower due to ml-dtypes source compile, but that
will be recovered in the next task via a BuildKit uv cache mount.
API parity and UI visual regression both match baseline, Trivy
HIGH/CRITICAL CVE count unchanged from opt-2 (4 CVEs, none new).
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
npm was installed in the runtime only to globally install vulnerability
patched versions of tar/glob/brace-expansion/minimatch/diff and to
in-place rewrite npm's own bundled package.json. Both were to silence
CVE scanners against modules that ship with npm itself.
Since we no longer run npm anywhere in the runtime (Prisma uses the
node binary directly for migrate deploy and generate), we can just
skip installing npm in the first place. This eliminates both the
~25-line CVE-patch shuffle AND the underlying CVE surface.
Kept: nodejs (needed by prisma-python's CLI and migrate deploy).
Removed: npm apk package, all 'npm install -g', all find+sed patching,
the redundant 'apk upgrade --no-cache nodejs' (already covered by the
preceding 'apk upgrade').
Image: 4.97GB (opt-1) -> 4.97GB (opt-2); the real win is that two
CVEs (CVE-2026-33671 and GHSA-q4gf-8mx6-v5v3) drop off the Trivy
HIGH/CRITICAL list. No new CVEs introduced. API parity and UI
visual regression both match baseline.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
The checked-in Next.js static export at litellm/proxy/_experimental/out/
is kept fresh by the UI Drift Guard CI workflow. Stage it directly
instead of re-running npm ci + npm run build inside the image.
This removes: nvm install, node 20.20.2 install, npm ci (801 pkgs),
next build, and the resulting intermediate node_modules/out tree.
Build time: ~6m25s -> ~2m (fuse-overlayfs DinD); image 6.57GB -> 5.0GB.
Behavior parity verified: API endpoints, UI screenshots (all 10 routes
pixel-perfect), and Trivy HIGH/CRITICAL CVE count (6 -> 5, one npm
GHSA removed) all match or improve over baseline.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Adds a CI job that rebuilds the admin UI from source and fails if the
committed static export at litellm/proxy/_experimental/out/ has drifted
from what npm run build produces. This prevents silently shipping stale
UI bytes and is a prerequisite for the non_root Dockerfile streamlining
work, which will stage the UI from _experimental/out/ directly instead
of rebuilding it inside the image.
Also regenerates litellm/proxy/_experimental/out/ to match a fresh
npm run build (Node 20.20.2) — the committed tree had drifted from
source prior to this commit.
Co-authored-by: yuneng-jiang <yuneng-berri@users.noreply.github.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
* bump litellm-proxy-extras version to 0.4.67
* bump litellm-proxy-extras pin to 0.4.67 in litellm pyproject
* regenerate uv.lock for litellm-proxy-extras 0.4.67
* bump litellm-enterprise version to 0.1.38
* bump litellm-enterprise pin to 0.1.38 in litellm pyproject
* regenerate uv.lock for litellm-enterprise 0.1.38
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
send_max_budget_alert_email previously guarded with `is not None`, which
accepts `[]` and then crashes on `recipient_emails[0]` inside
_get_email_params. The current caller (_handle_multi_threshold_max_budget_alert)
already filters empty lists upstream, but the public method signature makes
no such guarantee — a future caller passing [] would hit IndexError.
Switch to truthiness so both None and [] fall through to the single-recipient
path.
Per-user OAuth MCP requests now only skip pre-emptive 401 when a stored token is available, preserving token-reuse behavior while restoring fast PKCE kickoff for first-time or missing-token users.
The lazy import from litellm_enterprise inside _normalize_alert_emails
coupled the core proxy auth path to an optional package. Core should not
depend on enterprise, even lazily — it hides the dependency from static
analysis and inverts the intended layering.
Duplicate the 7-line parser locally. It's pure and unlikely to drift; the
enterprise copy stays where it is for its own callers.
_merge_budget_alert_email_configs previously called list() directly on each
threshold's value, which raised TypeError on null YAML values and silently
split bare strings into single characters. Both are reachable from user-
supplied global config and per-key metadata, so the crash could fire on
every authenticated request once the metadata was in place.
Route both inputs through a _normalize_alert_emails helper that delegates
to the existing _parse_email_list parser (lazy-imported, matching the
enterprise import pattern used elsewhere in proxy/). The merge body keeps
its tight Dict[str, List[str]] contract.
The Union[str, List[str]] value type was speculative — _merge_budget_alert_email_configs
always returns List[str] values, and no caller produces bare strings. Narrowing to
match the runtime guarantee resolves a mypy invariance error at auth_checks.py:3021
without adding casts or Mapping covariance.
A previous refactor added `litellm/integrations/prometheus_helpers.py` as a
sibling to the existing `litellm/integrations/prometheus_helpers/` directory
(which contains `prometheus_api.py` and has no `__init__.py`). The file
shadowed the namespace-package directory, so any deferred
`from litellm.integrations.prometheus_helpers.prometheus_api import ...`
raised `ModuleNotFoundError: 'litellm.integrations.prometheus_helpers' is
not a package` at request time.
Two runtime call sites hit that path:
- /global/spend/logs (spend_management_endpoints.py) returned plain-text 500
"Internal Server Error" for every call, breaking the Admin UI Usage tab
and programmatic consumers.
- SlackAlerting.send_fallback_stats_from_prometheus silently failed inside
its own try/except.
Fix: move prometheus_helpers.py content into prometheus_helpers/__init__.py
and delete the stray .py. The directory becomes a regular package, so both
the package-root import (from ...prometheus_helpers import X) and the
submodule import (from ...prometheus_helpers.prometheus_api import X)
resolve correctly. No call sites change.
