The openai and azure_openai columns have working credentials in every suite runner, so only the bedrock_mantle column still needs an opt-in flag while the AWS account waits on the Mantle allowlist; COMPAT_GPT_CELLS becomes COMPAT_MANTLE_CELLS. The six tool_use cells now resolve the proxy through claude_code._env like the basic_messaging cells instead of hardcoding LITELLM_PROXY_BASE_URL/LITELLM_PROXY_API_KEY.
* fix(model_armor): add skip_unscannable_attachments to allow reference-only attachments through
* fix(model_armor): wire skip_unscannable_attachments through guardrail config
* fix(model_armor): make max_file_attachments configurable and scan overflow instead of dropping
* fix(model_armor): remove the per-request attachment count cap and scan all attachments
---------
Co-authored-by: yucheng <yucheng@berri.ai>
* feat(e2e): emit structured E2E_RESULT lines for package status history
Pytest progress logs only expose file basenames and collapse multi-test files
into one status-history row. Emit one logfmt E2E_RESULT per finished node with
package, file, outcome, duration_ms, node_id, and covers so Grafana can roll up
by package (stable cardinality) and drill down by node_id in Explore
* test(e2e): drop unit tests from the live e2e tree
tests/e2e is for live proxy suites only. Remove harness, coverage-registry,
and claude_code unit trees so the e2e run does not collect them
* fix(e2e): type E2E_RESULT hook for basedpyright zero-error gate
Protocol-typed covers extraction and pluggy Result typing on the
makereport hook so tests/e2e stays under the e2e basedpyright ceiling
* fix(e2e): drop unused xfailed/xpassed from E2E_RESULT Outcome
We never emit those states; xfail collapses to skipped/passed via pytest
report flags. Keep the literal honest so the dashboard only sees real outcomes
* fix(e2e): strip tests/e2e prefix when deriving E2E_RESULT package
Repo-root pytest nodeids are tests/e2e/<suite>/...; without stripping,
every line would package=tests and status history would be useless
* fix(e2e): use pytest wrapper=True instead of deprecated hookwrapper
pytest 8.1+ deprecates hookwrapper; yield returns the TestReport directly
so we return it for the outer chain and drop pluggy.Result
* fix(e2e): import e2e_result_reporter at module load
Surface a missing module as a collection-time ImportError instead of a
per-test hook failure mid-run
* test(e2e): restore coverage_registry/test_collector.py
Needed to validate registry coverage math and the checked-in cell
denominator; not a live proxy suite
With enable_redis_auth_cache and multiple replicas, /key/update and
/key/delete delete the Redis auth blob and the handling pod's in-memory
entry, but two read-path writers re-published the stale blob from any other
replica's per-pod memory back to Redis with a fresh 60s TTL on every
request: the post-auth re-cache in user_api_key_auth and the spend writeback
in update_cache. Replicas whose in-memory entries expired then re-primed
themselves from the poisoned Redis entry, so key limit and access changes
never took effect fleet-wide while traffic continued, and a deleted key kept
authenticating.
The auth object is now written only by the DB-load paths
(IdentityStore._resolve_key, get_key_object): the post-auth re-cache is
removed outright (even a local-only write could race an invalidation and
resurrect a revoked key on this worker) and spend tracking no longer writes
the auth object back at all; spend is tracked through the spend🔑*
counters. The remaining spend writebacks for user, team, end-user, and tag
objects become local-only so they cannot republish stale management objects
either, with one deliberate exception: the proxy-wide
{litellm_proxy_admin_name}:spend scalar keeps its shared Redis write because
the global max_budget check reads it between authoritative DB reloads, and
it carries no limits or permissions so sharing it cannot resurrect an
invalidated auth blob.
DualCache's redis-to-memory read backfill also ignored default_in_memory_ttl,
pinning backfilled entries for InMemoryCache's 600s default instead of the
configured 60s auth TTL; the backfill now injects the configured default like
every write path already does, so a replica primed from Redis converges
within the auth cache TTL as well.
Consolidates the sibling stale-auth-recache branch; the delete-propagation
case is the duplicate ticket LIT-4350.
Resolves LIT-4219
Team/key router_settings.enable_tag_filtering was stored and echoed by
/team/info but never applied at request time: the per-request override
whitelist in route_llm_request.py dropped it, tag filtering only read the
router-level flag, and UpdateRouterConfig silently discarded the field on
/key/generate and /config/update. Requests from teams with the toggle on
were load balanced across all deployments instead of tag-matched ones.
- add enable_tag_filtering to the router_settings_override whitelist and
strip any client-supplied copy from the request body first, so only the
key/team value reaches the router
- run tag filtering when the request carries enable_tag_filtering=True; a
request-level False cannot disable a router-level True, so per-request
settings can only scope down, never escape the global policy
- add the field to UpdateRouterConfig so key and config update paths stop
dropping it, and to all_litellm_params so it never leaks into provider
request bodies
- allow it through Router.update_settings/get_settings so the global UI
toggle persists across DB config reloads
Resolves LIT-4390
The hook returned early when the semantic filter selected no tools, which
restated a policy that SemanticMCPToolFilter.filter_tools already owns: it
returns the full tool set when nothing matches, so the selection is never
empty. The branch was unreachable, and reachable or not it changed nothing,
since the gateway reads the union of every reference's allowed_tools and
treats an empty union as unset. Its only effect was to suggest the reference
path and the plain tool path resolve a zero-match query differently.
Drop it so a single policy governs both paths, and pin that with a test
covering an unmatched query on each path. Flipping filter_tools to fail
closed now fails the test on both instead of quietly hard-limiting one
surface and not the other.
* fix(langfuse_otel): build per-request OTLP exporter from key/team dynamic Langfuse credentials
Key-scoped langfuse_otel callbacks only injected Authorization headers into the
init-time exporter, so a proxy without global LANGFUSE_* env vars kept its
fallback exporter and never exported traces to Langfuse. Dynamic params now
build a full per-request OTLP config (endpoint from the key's langfuse_host,
otlp_http, basic auth from the key's credentials).
Resolves LIT-3976
* fix(otel): log dynamic config endpoint in span processor debug output
* fix(otel): redact authorization headers in exporter debug logs
* fix(router): honor per-request routing_strategy from key/team router_settings
Key and team router_settings.routing_strategy was stored and shown in the
UI but never forwarded to the shared Router, so the global strategy always
won. Forward it through router_settings_override and resolve it in
_get_routing_context: a validated per-request strategy takes precedence
over routing groups and the top-level strategy, with lazily built cached
selectors for strategies that need one. Unknown or unsupported strategy
values are ignored with a warning instead of failing the request, and
routing_strategy is registered in all_litellm_params so it is stripped
before the provider call.
* fix(router): sweep override selectors on strategy re-init and cover coverage-gate helpers
routing_strategy_init now unregisters cached per-request override
selectors so a later update_settings strategy change cannot leave a
zombie selector receiving callback events. Adds direct tests for the
two new helpers so the router code coverage gate passes.
* docs(team): document mcp_rpm_limit in update_team docstring
The documentation CI job walks management_endpoints and requires every
UpdateTeamRequest field to appear in the update_team docstring;
mcp_rpm_limit was added to the model without a docstring line, failing
the job on unrelated PRs depending on walk order. Regenerates
schema.d.ts since the docstring feeds the OpenAPI spec.
* fix(proxy): coerce default_internal_user_params.max_budget to float on config load
* fix(proxy): log coerced default_internal_user_params and cover absent max_budget in tests
The semantic tool filter replaced each litellm_proxy MCP reference in
data["tools"] with the tools it expanded from that reference. The expansion
defaults to the Responses API tool shape, so a /chat/completions request came
out carrying flat {"type": "function", "name": ...} entries where the provider
transformations expect {"type": "function", "function": {...}}. Anthropic then
raised KeyError: 'function' and Bedrock dropped every MCP tool silently, so the
model answered as if no MCP server were connected.
Replacing the reference also removed the marker the MCP gateway matches on, so
acompletion_with_mcp never ran and tool calls were no longer auto-executed for
require_approval="never", on /responses as well as /chat/completions.
Narrow the reference through allowed_tools instead and leave it in place, so the
gateway still owns expansion and keeps both the per-endpoint tool shape and tool
auto-execution. Expansion already applies any caller-supplied allowed_tools, so
the selection can only narrow a reference further, never widen it.
* fix(proxy_cli): reap orphaned prisma query-engine processes when a worker dies
When the proxy runs multi-worker (uvicorn multiprocess supervisor or the
gunicorn arbiter), a worker that crashes or is force-killed never runs its
in-process atexit cleanup, so its prisma query-engine subprocess reparents
to PID 1 and keeps its database connection pool established forever while
the replacement worker opens a fresh pool. Active DB connections then grow
past database_connection_pool_limit with every worker death.
Run a reaper thread in the supervisor process that marks itself a child
subreaper on Linux, scans for adopted query-engine children whose worker
is gone, and terminates them with SIGTERM escalating to SIGKILL after a
bounded grace period. Engines owned by live workers are children of those
workers, never of the supervisor, so they are structurally out of reach.
Resolves LIT-4449
Fixes https://github.com/BerriAI/litellm/issues/33023
* fix(proxy_cli): address review findings on the query-engine reaper
Make start_query_engine_reaper idempotent, reap simultaneous orphans
under one shared grace period instead of serially, and log when a PID
survives SIGKILL. Also regenerate schema.d.ts for the update_team
docstring line that documents the existing mcp_rpm_limit param (fixes
the walk-order-dependent documentation CI failure) and avoid a cast in
the prctl wrapper
* test(proxy): fix reaper idempotency-test isolation and widen coverage
The daemon-thread startup test now stubs threading.enumerate so a
reaper thread left running by an earlier test in the same xdist worker
cannot satisfy the idempotency guard and skip the code under test. Add
coverage for stat-file truncation, non-numeric ppid, non-child reap,
signal-to-dead-pid, subreaper capability, and reaper-loop resilience
Both enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl are
now read from LITELLM_ENABLE_ANTHROPIC_PROMPT_CACHING and
LITELLM_ANTHROPIC_PROMPT_CACHING_TTL at import, so the flag can be turned on
without a config file. An unsupported ttl falls back to the provider default
rather than reaching the provider verbatim
Anthropic only caches a prompt when the request carries explicit cache_control
breakpoints, unlike OpenAI where prompt caching is automatic and needs no
configuration. Today litellm can inject those breakpoints server-side, but only
when an admin hand-writes cache_control_injection_points into a model's
litellm_params (or router_settings.default_litellm_params). Clients such as
Claude Code and Claude Desktop never set cache_control themselves, and the
admin recipe is easy to miss, so Anthropic traffic through the proxy silently
pays full price on every repeated prefix.
This adds an opt-in litellm_settings flag, enable_anthropic_prompt_caching. When
it is on and the request has no injection points configured and no
client-supplied cache_control, litellm synthesizes a default pair of breakpoints
(the system prompt and the trailing turn) so the stable prefix is cached while
the breakpoint advances with the conversation. It is wired into both surfaces:
/chat/completions seeds the points before the existing prompt-management gate, and
/v1/messages resolves them in maybe_inject_cache_control, so the existing
AnthropicCacheControlHook applies them unchanged and keeps its four-block cap and
its refusal to overwrite client breakpoints.
The default is off, so no existing deployment changes behavior. Injection is
gated to providers that actually consume cache_control markers (anthropic and
bedrock) and to models the cost map flags as supporting prompt caching; note that
supports_prompt_caching alone is not a sufficient gate, since OpenAI, Azure and
Gemini models report it as well but never take cache_control markers. The default
ttl is Anthropic's 5 minute ephemeral cache, with an optional
anthropic_prompt_caching_ttl of "5m" or "1h"; ttl is also added to
ChatCompletionCachedContent, which the bedrock and anthropic transforms already
read at runtime but the type never declared
Resolves LIT-4478
* chore(e2e): establish litellm_e2e_staging integration line
Long-lived berri branch for e2e suite recovery work (LIT-4479 through LIT-4486) before merge to litellm_internal_staging
* test(e2e): remove langfuse_otel logging e2e suite (#33558)
* test(e2e): remove langfuse_otel logging e2e suite
Removes the LIT-4483 dynamic per-team/key/org langfuse_otel logging e2e tests (tests/e2e/logging/test_langfuse_e2e.py, added in #32857). The shared logging_client harness and the langfuse coverage-registry cells are left in place; only the test module is removed. The otel and prometheus logging e2e suites are unaffected.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop orphaned langfuse coverage-registry cells
The three logging.langfuse.*.logs_spend P0 cells were only exercised by the deleted langfuse_otel e2e suite. Remove them so the coverage registry has no orphaned rows.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e): log into the react admin ui in the management browser fixture (#33562)
The management ui_page fixture drove the old server-rendered login form: it clicked input[type="submit"] and treated wait_for_url("**/ui/**") as the done signal. /ui/ now serves the react (antd) dashboard whose submit is a <button type="submit">, so the click waited out the full 30s timeout and errored every browser test in the suite. wait_for_url also matched instantly because the login page already lives at /ui/, so on the fast path the fixture navigated before the auth cookie landed and got bounced back to login.
Click the antd submit button and wait for the token cookie loginCall sets on document.cookie, the real post-login signal.
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e): make ui login readiness robust to httpOnly token cookies (#33564)
The login readiness check waited only on document.cookie including token=, which is empty when the token cookie is httpOnly. If the server ever sets it via a Set-Cookie header, the wait would spin to the 30s timeout and silently reproduce the original hang. Also accept the login form detaching (#username gone after the post-login redirect) so readiness holds regardless of how the cookie is delivered.
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Cost values read from deployment model_info can be strings when the
config YAML contains scientific notation with an integer mantissa
(e.g. 1e-05), which YAML 1.2 parsers such as PyYAML 6.x treat as a
string. Comparing that string against the running float aggregate in
_set_model_group_info raised TypeError and broke /model_group/info,
the prometheus remaining-usage callback, and the
x-litellm-response-cost header. Coerce input/output cost values to
float before comparing and storing them.
* feat(logging): add user and team level spend and budget to StandardLoggingPayload metadata
* fix(logging): include user and team budget fields in dummy standard logging payload
* refactor(e2e/claude_code): align proxy env names with the rest of tests/e2e
Every claude_code compat cell used to read its own `LITELLM_PROXY_BASE_URL` and `LITELLM_PROXY_API_KEY` and duplicate the same 12-line "missing env, hard fail" block. The rest of `tests/e2e/` reads `LITELLM_PROXY_URL` and `LITELLM_MASTER_KEY` from `e2e_config.py`, so anyone standing up a live proxy for one suite had to export a second spelling for claude_code, and every cell repeated the same boilerplate.
Centralize the resolution in `claude_code/_env.py`. `resolve_proxy()` prefers the suite-wide `LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY` names and falls back to the legacy pair so existing CI wiring on stage keeps working during the roll-out. `require_proxy(compat_result)` is the one-liner cells call to bind `(base_url, api_key)` or hard-fail with a message that names both spellings.
55 cell files, `_basic_messaging.py`, and the driver's own unit-test fixture now go through the helper. `run_compat.sh` accepts either spelling and normalizes to the primary names before invoking pytest. `cron_vm/run_daily.sh` exports the primary names when launching pytest.
`_pr_gate_unit_tests/test_env_resolution.py` pins the resolution rules so a future edit cannot silently reintroduce the drift: primary names win on tie, legacy names still resolve when primary is unset, mixed URL-primary key-legacy still resolves, empty-string exports are treated as unset, `require_proxy` names both spellings in its error message.
Net diff: 71 files, +370/-1240.
* fix(e2e): anchor claude_code Bash pin at parents[1] so container run collects
`test_bash_tool_restrictions.py` derived `REPO_ROOT = Path(__file__).resolve().parents[4]` and then joined `tests/e2e/claude_code/<feature>`. That works locally, but the stage container mounts tests/e2e/ at /app/e2e/, so parents[4] resolves to filesystem root and the `_bash_cells()` assertion looks for `/tests/e2e/claude_code/tool_use` — a path that doesn't exist. Collection interrupts before any test runs, so the entire e2e suite appears broken.
Fix: `CLAUDE_CODE_DIR = Path(__file__).resolve().parents[1]` resolves to the sibling `claude_code/` dir in either layout, and the `relative_to(REPO_ROOT)` calls become `relative_to(CLAUDE_CODE_DIR)` so test IDs and error messages read the same.
Adds `test_claude_code_dir_anchor_is_layout_independent` as a regression pin: it checks the anchor lands on a directory named `claude_code` that contains this test file, which would fail under the old parents[4] anchor when run from /app/e2e/.
* feat(e2e/claude_code): register compat deployments via /model/new from a session fixture
Every compat cell hardcodes a virtual model name like `claude-sonnet-4-6` or `claude-sonnet-4-6-bedrock-invoke` and hits the proxy expecting it to be routable. On stage those live in the deployed model_list; locally the `docker-config.yaml` under tests/e2e/ only declares one of them, so anything past haiku 400s with `Invalid model name`.
`claude_code/test_config.yaml` is the ground-truth compat matrix config the deployment already uses. `_compat_models.py` loads it, normalizes the yaml keys pydantic would silently drop (vertex_ai_* → vertex_*), and selects the subset whose provider credentials are present in the environment. An autouse session fixture in `conftest.py` POSTs each selected deployment to `/model/new`, blocks until it is servable on the data plane, and tears them all down on session exit. Skips silently when the proxy env is unset so pure-unit runs stay hermetic.
`test_compat_models.py` pins the invariants that keep this safe. Every cell-referenced name must have a yaml entry (drift check catches a cell probing a name the fixture never registered); the yaml has no unused declarations; the fixture registers exactly 15 deployments (3 tiers × 5 provider surfaces); vertex_ai_* yaml keys populate the pydantic body's vertex_* fields (they got silently dropped historically); Azure needs both AZURE_FOUNDRY_* env vars; Bedrock lifts creds from the ambient AWS chain; Vertex needs both the yaml refs AND ambient GCP credentials.
* refactor(e2e/claude_code): inject env + runner instead of monkeypatching
`require_proxy` and `_basic_messaging.run_basic_messaging_cell` now take the env mapping (and the CLI runner) as constructor-style arguments with `os.environ` and `run_claude_models_parallel` as defaults. Tests exercise the branching by passing dicts and callables directly, so `monkeypatch.setenv` and `monkeypatch.setattr(_basic_messaging, "run_claude_models_parallel", ...)` are gone from every unit test in this refactor's blast radius.
`test_env_resolution.py` drops the `monkeypatch.setenv`/`delenv` fixtures and passes `env={...}` dicts to `require_proxy`. Added a new pinned check that a successful resolution leaves `compat_result` untouched, and split the "unset env" test into three explicit shapes (empty, primary-only, legacy-only) so a regression that swaps the precedence rule can no longer hide behind a single monkeypatched fixture.
`test_basic_messaging.py` (driver) replaces the `_install_fake_runner(monkeypatch, ...)` helper with `_make_fake_runner(...)` that returns a `(callable, captured_dict)` pair the test passes in via the helper's new `runner=` kwarg. Also drops the autouse `_proxy_env` fixture in favor of a module-level `_PROXY_ENV` dict each test wires through the helper's new `env=` kwarg. Added a regression pin that a missing-env call hard-fails without ever invoking the runner (so the guard order stays correct).
`test_run_daily_pytest_scrubs_env.py` updates its pin to assert the new suite-wide env spellings (`LITELLM_PROXY_URL` / `LITELLM_MASTER_KEY`) instead of the legacy `LITELLM_PROXY_BASE_URL` / `LITELLM_PROXY_API_KEY` that `run_daily.sh` used to export.
* handwrote rules
With litellm_settings.enable_redis_auth_cache enabled, user_api_key_cache
is Redis-backed. _update_user_db performed a cache lookup with
key=user_id where user_id can be None; the in-memory cache tolerates a
None key but Redis raises redis.exceptions.DataError (Invalid input of
type: NoneType) on every spend update for requests without a user_id.
The looked-up value was never used by any subsequent code, so the lookup
is removed along with the user_api_key_cache parameter it existed for.
Spend updates for users, end users, and the global proxy budget are
unchanged
* fix(cli): force UTF-8 output so emoji don't crash the CLI on Windows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cli): drop dead flush calls flagged by review
* fix(cli): replace non-ASCII CLI output with ASCII so legacy Windows consoles don't crash
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): datadog log delivery for successful chat, messages, and responses
Covers logging.datadog.success.exports_metric on all three routes: one
successful non-streaming call must reach the DataDog logs intake as exactly
one log event whose StandardLoggingPayload message carries the model group,
real token counts, and a response cost equal to the x-litellm-response-cost
header of the same response. Delivery is judged at the intake: the compose
stack gains a dd-sink service recording every batch the datadog callback
ships via the DD_BASE_URL testing override, and a typed reader replays it.
Writing these caught a live product bug: /v1/messages double-logs every
success (two byte-identical events per call), filed as LIT-4447; the messages
test tolerates byte-identical duplicates of the one event until it lands,
while a second differing event still fails
* test(e2e): address review findings on the datadog delivery suite
Consolidates the fresh-key first_ok helper into logging_client now that the
otel PR it mirrored has merged (both test files use the shared copy), moves
intake batch parsing into a helper so no path can leave the batch unbound,
and gives the sink's /health endpoint a truthful text/plain content type
* test(e2e): tolerate same-logical-event duplicates by call id, not byte identity
A clean LIT-4447 repro showed the duplicated payload is built twice and can
mint a fresh synthetic completion id per emission, arriving as two separate
intake POSTs with the same litellm_call_id and identical substantive fields.
Byte-identity was therefore a flaky criterion; duplicates now qualify only
when they share the call id, call type, model group, tokens, and cost, and a
second differing event still fails
* test(e2e): assert the scenario strictly; the messages test is the LIT-4447 regression pin
Per review direction the tests now assert exactly what the scenario promises:
exactly one DataDog log event per successful call, on every route. The
/v1/messages test therefore fails on current code against the known
double-log (LIT-4447) and is its regression pin; it goes green when the fix
lands. The duplicate-tolerance machinery is removed
* Simplify docstrings for DataDog log tests
Removed redundant phrasing about cost cross-checking in docstrings.
* Update test_datadog_log_e2e.py
Making the in-memory issuer reflect a trust-on-first-use discovered value fixed
the registry/row token-identity drift, but it overloaded a single field: the
carry-forward gate keyed on issuer truthiness as a proxy for "endpoints are
anchored to a pinned issuer, fail-closed". A discovered issuer is truthy yet not
anchored, so a resource-rooted server that had learned its issuer would drop its
last-known-good endpoints on a transient discovery blip instead of carrying them
forward.
Anchoring is now a first-class property rather than a proxy. MCPServer carries
issuer_is_anchored, set at both build paths from the single _uses_issuer_anchor
definition (a pinned issuer on a discovery auth type). issuer stays the identity
value used by the token-identity tuple and the serializers; issuer_is_anchored is
the provenance value the carry-forward gate reads to decide fail-closed. The two
properties can no longer be conflated, so a discovered issuer keeps its
resource-rooted endpoints carrying forward while a pinned issuer still fails
closed.
Regression tests pin both directions: a discovered-but-not-anchored server
restores its endpoints on a discovery blip, an anchored server does not, and the
build sets issuer_is_anchored true only when the issuer is pinned
Pin a session's first-turn model for the rest of the session by default
instead of reclassifying every turn. Keeps multi-turn sessions on a single
model, preserving provider prompt caches and avoiding cross-model
conversation-history errors (e.g. Anthropic rejecting a thinking block
produced by a different model). Requests without a resolvable session_id are
unaffected. Set session_affinity: false to opt out.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cli): add `lite up`/`lite down` to ambiently route Claude Code through the proxy
Patches ~/.claude/settings.json in place (env.ANTHROPIC_BASE_URL + apiKeyHelper
via `lite auth print-token`) so any `claude` session started afterward, from
any terminal, routes through the local LiteLLM proxy with no wrapper command
needed, unlike the existing `lite claude` subprocess-exec approach. Backs up
the original file first and restores it on Ctrl-C/SIGTERM, or via `lite down`
after an unclean exit. Cursor is not supported: no equivalent file-based config
to patch.
* feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy (#33249)
* feat(cli): add lite autoroute to QA complexity-based auto-routing against a real proxy
Lets a customer try litellm's complexity_router against models they already
have on their existing, unmodified production proxy, with no config.yaml
edits and no new infra. lite autoroute configure discovers accessible
models via /model_group/info and walks through tier assignment (plus
optional LLM classifier / semantic matching / adaptive selection); every
referenced model becomes its own litellm_proxy/<name> deployment forwarding
back to the real proxy with the real key, so every actual call, routed
completions, classifier calls, embedding calls, still lands on their real
proxy. lite autoroute up launches that generated config as an ephemeral
local proxy, patches ~/.claude/settings.json to point Claude Code at it, and
streams routing decisions live; Ctrl-C/SIGTERM (or lite autoroute down
after an unclean exit) restores everything.
Also adds lite model-groups list (a thin CLI wrapper over the existing
ModelGroupsManagementClient), and generalizes up.py's settings-backup/restore
helpers to take explicit paths so this feature can reuse them instead of
duplicating the logic.
Depends on litellm_lite_up_down (#33231) for that generalization.
* feat(cli): allow multiple models per autoroute tier
complexity_router already supports a pool of models per tier (randomly
picked per request; adaptive mode specifically needs a pool to choose
within), but the configure wizard only ever let you assign one. Tiers are
now a tuple of model names; the wizard prompt accepts comma-separated
indices to pick more than one per tier.
* feat(cli): fuzzy model picker and auto-route Claude Code to autorouter
Numbered-index selection didn't scale past a handful of models, so switch
the tier picker to InquirerPy's fzf-style fuzzy search. Also set
ANTHROPIC_DEFAULT_{SONNET,HAIKU,OPUS}_MODEL to "autorouter" in Claude
Code's settings, since Router resolves auto-router deployments by literal
model name with no wildcard support, so a "*" catch-all model_name would
never match real traffic.
* feat(cli): allow installing lite CLI from source via LITELLM_CLI_REF
Lets testers try an unreleased branch's CLI changes with the same
curl-piped installer, instead of waiting for a PyPI release.
* fix(ci): modernize type hints to clear ruff strict-rule budget
* fix(ci): bump httplib2 and setuptools to patched versions
Clears osv-scan findings for PYSEC-2026-3444 and PYSEC-2026-3447.
* fix(cli): write autoroute's secret-bearing files with mode 0600
commands.py wrote config.yaml (embeds the real proxy key) and Claude
Code's settings.json (embeds the ephemeral proxy's master key) with
plain open(), landing at the umask-derived default (commonly 0644)
until a later chmod call caught up. That window, and the missed case
where settings.json already exists (chmod never ran at all there),
left a credential-bearing file readable by another local account.
secure_create() fixes the mode via fchmod on the fd before any
content is written, covering both the brand-new-file and
already-exists cases, and commands.py/wizard.py now route their
sensitive writes through it.
* docs(cli): warn that a stale Claude Code session can leak to a squatted port
lite autoroute up's master key is embedded statically (unlike lite up's
apiKeyHelper, resolved per request), so a Claude Code session still
running after teardown keeps sending it, along with prompt content, to
a now-unbound loopback port that another local account can bind. This
is the same one-time-patch tradeoff lite up already accepts, just with
a static secret instead of a re-resolved one -- document it in the
README's Caveats section and surface it in the teardown message itself.
* fix(cli): address greptile review feedback on autoroute PR
- terminate the ephemeral proxy child process when its health check
fails, instead of leaking an orphaned, unrecoverable process bound
to the port
- replace bare assert isinstance checks (no-ops under python -O) with
click.ClickException in the model-groups list and configure wizard
code paths
- close launch_proxy's log file handle once the child process has
inherited its fd, instead of leaking it
- add build_generated_proxy_config to config.py's __all__
* fix(cli): close TOCTOU window in lite up's settings backup write
write_backup wrote the backup (which can embed the original
apiKeyHelper/settings content) with plain open() + a chmod call after
the fact -- the same permissive-until-corrected window already fixed
for autoroute's config.yaml and Claude settings writes, and missed
entirely when the backup file already exists with broader permissions.
Moves secure_create (atomic-enough 0600 via fchmod before any content
is written) to up.py, the module both lite up and lite autoroute
share, and has autoroute/process.py import it from there instead of
keeping its own copy.
* fix(cli): refuse autoroute up when a stale backup exists from a crash
The pid-record check only catches a still-live duplicate process; a
SIGKILL'd `up` leaves no live pid but does leave AUTOROUTE_BACKUP_PATH
behind. Without this guard, a fresh `up` overwrote that backup with
the currently-patched Claude settings instead of the true originals,
so `down`/Ctrl-C would restore the wrong content permanently. up.py's
`lite up` already guards the analogous case; mirror it here.
* fix(cli): bind the ephemeral autoroute proxy to loopback only
proxy_cli.py defaults --host to 0.0.0.0 when not passed explicitly.
launch_proxy never passed it, so the ephemeral proxy -- despite every
base_url in this module being built from 127.0.0.1 -- was actually
reachable from other hosts on the network, including its
unauthenticated-until-config-lands routes before the master key is
wired in.
* docs(cli): show curl install for the autoroute QA flow
Points readers at scripts/install-cli.sh's curl one-liner instead of
assuming uv/pip is already set up, and documents the LITELLM_CLI_REF
override for trying an unreleased branch or commit.
* fix(cli): surface a clean error on an empty or corrupt autoroute config
A configure run killed between secure_create's O_TRUNC and the write
completing leaves an empty config.yaml on disk. The next up read that
via yaml.safe_load (None) into the generated-config TypeAdapter
uncaught, surfacing a raw pydantic.ValidationError instead of pointing
the user back at `lite autoroute configure`.
* fix(cli): bind lite up's apiKeyHelper to the proxy it was started against
_ensure_fresh_login only checked token freshness, not which proxy the
cached token belonged to, and resolve_api_key_helper built a bare
`lite auth print-token` command with no --base-url. A user logged into
proxy A who ran `up --base-url proxy-b` (or LITELLM_PROXY_URL=proxy-b)
would silently get proxy A's real token wired into Claude Code's
apiKeyHelper; since apiKeyHelper is invoked bare, print-token's
existing origin check never engaged, so proxy B -- attacker-controlled
or not -- received every subsequent request's Authorization header
carrying proxy A's credential.
_ensure_fresh_login now requires the cached token's base_url to match
before treating it as usable, forcing a fresh login for the selected
proxy otherwise. resolve_api_key_helper now takes that base_url and
threads it through as an explicit --base-url, so print-token's
existing (but previously unreachable in the apiKeyHelper flow)
base_url_explicit check actually enforces the match at request time
too.
* fix(cli): surface clean errors instead of raw tracebacks in lite up/down
load_json_or_empty and read_backup both delegate to pydantic's
validate_json, which raises ValidationError on invalid JSON or a
non-object root -- neither up() nor down() caught it, so a corrupt
settings or backup file surfaced an unformatted Python traceback
instead of a clean CLI error. Both now convert to UpError, and down()
(previously uncaught entirely) and up()'s teardown path now handle it.
restore_claude_settings also gained a parent.mkdir guard before
rewriting CLAUDE_SETTINGS_PATH: if ~/.claude/ was removed while `lite
up` was running, the restore would crash before deleting the backup
file, permanently stranding it and breaking every future `lite down`.
* docs(cli): call out env-var auth for autoroute commands
* fix(cli): clean up leaked proxy and surface clean errors in autoroute
Three related gaps, all following an UpError getting raised somewhere
that wasn't catching it yet:
- up() left the just-launched ephemeral proxy running with no pid
record if load_json_or_empty/write_backup/secure_create raised after
the health check passed, mirroring the existing ProcessLaunchError
cleanup for the health-check-failure branch.
- _teardown() didn't catch restore_claude_settings raising UpError
(e.g. a corrupt backup at stop time), which would otherwise escape
to Click as an unhandled error in the normal-exit path, or print
"Error in atexit" in the atexit path. up.py's own _restore_once
handles the identical case the same way.
- read_pid_record let a corrupt PID file surface a raw
pydantic.ValidationError instead of a clean message, and did so in
down(), the command specifically meant for crash recovery. down()
now clears an unreadable pid record and continues cleanup instead of
aborting, since a corrupt pid file must never block the one command
meant to recover from exactly this kind of crash.
* docs(cli): warn against running lite up and lite autoroute up together
