Persistently route Claude Code through a LiteLLM proxy with a long-lived virtual key or the stored lite login, turn on gateway model discovery so /model lists the proxy's models, optionally pick the model Claude Code starts on, and record what changed so unconfigure restores only the keys the user has not touched since. lite login --config-claude writes through the same receipt and is undoable too. The two settings merges (lite up / --config-claude and lite autoroute) collapse into one credential-aware merge
Claude Code reads settings.json from CLAUDE_CONFIG_DIR when it is set,
while lite login --config-claude wrote to ~/.claude/settings.json and
lite claude checked that same file for its apiKeyHelper. With the
override set, lite could drop ANTHROPIC_AUTH_TOKEN because the helper
lives in a file Claude Code never reads, leaving it with no key at all.
claude_settings_path(environ) now picks the file the way Claude Code
does, and both commands go through it. The CLI test directory gets an
autouse conftest that isolates HOME, USERPROFILE, and CLAUDE_CONFIG_DIR
per test and fails any test that writes the developer's real
settings.json, restoring it first.
Add an embeddings case with its own closed-port primary and mock backup (the fallback map in
the gateway config gains the pair; LiteLLMParamsBody.mock_response accepts the list an embedding
mock needs). Assert from /metrics that the proxy's resident memory grows by no more than 200 MB
across each case where the process collector reports it (Linux), that the router counted a
successful fallback for every request, and that every spend row is a success.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The suite used to mint its own RS256 tokens from a stand-in issuer, which
could only ever prove the proxy agreed with the tests: the claims were
whatever the tests chose to sign. Every JWT bug worth catching lives in the
shape of what an identity provider really emits, so the suite now runs
against Keycloak (realm in idp_realm.json), provisions a group and a user per
test through its admin API, and signs in through the direct-access grant.
That changes what the tokens look like: sub is Keycloak's opaque user uuid
rather than a friendly name, groups arrives from a protocol mapper, aud is
the IdP's own audience, and the JWKS carries an encryption key beside the
signing key so the proxy has to select on kid. The expiry case now takes a
one-second token from a second client in the realm and waits for it to lapse
instead of forging a stale exp.
The proxy config the suite needs is unchanged. CI runs it against a Keycloak
deployed beside the ephemeral stack, which lives in the releaser repo.
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
get_candidate_model_ids_for_route (added in #40280 for the encrypted-content affinity
check) reconstructed the candidate pool by unioning the model_name and team indexes with
pattern_router.route. That diverged from how the router actually resolves a route: it took
a union instead of the first matching path, and pattern_router.route only matches the
literal name, so a provider-qualified pattern (matched by get_deployments_by_pattern, which
retries the {provider}/{model} form) was missed and the default deployment was ignored.
For an affinity follow-up on a wildcard or team-public route, that mismatch could strip
encrypted reasoning on a same-group cooldown, or return a 503 on a real cross-path switch.
Delegate the non-model_name case to _try_early_resolve_deployments_for_model_not_in_names,
the same resolver _common_checks_available_deployment uses, so candidate membership follows
the router's real precedence. With include_team_models left off it stays read-only and does
not raise. Behavior for concrete model groups and routing groups is unchanged.
Claude-Session: https://claude.ai/code/session_01KAumQbhzk6jdWWHFLA8Jar
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The FK cascade drops the LiteLLM_JWTKeyMapping row, but the cached
jwt_key_mapping:{claim}:{value} entry still resolved to the deleted token
hash, so every JWT call from that identity failed until
virtual_key_mapping_cache_ttl expired instead of auto-registering against a
recreated key. delete_verification_tokens now snapshots the mapping cache
keys before the delete and evicts them across replicas afterwards, the same
way /key/regenerate already does.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
The e2e directive has every test create its deployments through the management API and delete
them on teardown. Drop the static model_list from the gateway config; the test now registers the
closed-port primary and the mock backup itself, and the fallback map stays in router_settings
where proxy-level config belongs.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The encrypted_content_affinity check only read the pin from the Responses
input, which /v1/messages builds after the router has picked a deployment,
so a model group spread across OpenAI orgs sent follow-up turns to the
wrong org and got invalid_encrypted_content back. The check now also
decodes the pin from bridge-tagged thinking and redacted_thinking blocks
in the Anthropic messages. The bridge also keeps a deployment's own
include list next to reasoning.encrypted_content instead of replacing it.
A loopback Redis answers many commands inside the 1 ms socket timeout, so nothing guaranteed
the failure path ran. The test now holds the proxy's Redis in CLIENT PAUSE WRITE for its
duration, so every write the proxy sends, the spend counter increment included, outlives the
timeout, and lifts the pause in teardown. Reads stay live so the control connection can do that.
Enable the prometheus callback in the gateway config and assert from /metrics that the proxy
counted at least the breaker's five timeouts and that, during each case, it saw fresh timeouts,
a breaker transition, or an open breaker rejecting every call. The open breaker is the state a
customer's worker sits in, and cost tracking fails on every request either way.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
The retrieve writer and the CheckBatchCost poller are two different spend
writers, and the in-run failed batch only proves the first. Split the
key_attribution batch cell in two: retrieve_batch_cost_joins_retrieving_key,
which test_terminal_batch_cost_row_joins_the_retrieving_key claims, and
poller_batch_cost_joins_creating_key, which no test claims yet and so shows
up on the coverage dashboard as a P1 gap instead of hiding behind the
retrieve leg. The rationale records why one run cannot hand the poller a
completed batch on a stack that boots a fresh Postgres per build.
Guardrails that hand back tool_calls in their own shape (vendor JSON, user code output) raised a KeyError on the non-stream Responses write-back. Returned tool calls are now validated before comparison; a shape or count that does not line up leaves every tool-call item unchanged and logs a warning naming the guardrail. A tool call's name is written back only when the guardrail changed it, so a nameless custom_tool_call no longer picks up the custom_tool placeholder.
The batch list is served from LiteLLM_ManagedObjectTable whenever the managed
files hook is loaded, and the Buildkite e2e stacks bundle a fresh Postgres per
build, so a prior run's marker batch is never listed and the baton could only
ever pass vacuously. Each run now creates a batch OpenAI fails at validation
within seconds, retrieves it by its raw provider id with the same key until it
is failed, and asserts the {provider_batch_id}_batch_cost row that retrieve
writes joins the key's token hash and alias
`lite login --config-claude` writes an apiKeyHelper into ~/.claude/settings.json,
and `lite claude` then also exported ANTHROPIC_AUTH_TOKEN, so Claude Code opened
with its "Both ANTHROPIC_AUTH_TOKEN and apiKeyHelper set" banner and, because the
env token wins, never ran the helper that was meant to refresh the key.
When the launch key is the stored login key and settings.json carries exactly
the helper lite wrote for this base URL, `lite claude` now leaves the token out
of the env (dropping an inherited one) so Claude Code asks the helper. An
explicit --api-key or LITELLM_PROXY_API_KEY, a helper for another proxy, a
hand-written helper, or no settings file keep the previous behavior.
Team admins can create and edit team models through /model/new, PUT /model/update
and PATCH /model/{id}/update. The proxy forwards aws_session_tags to STS under its
own identity, so a team admin could pick tags that unlock aws:PrincipalTag gated
resources. Only proxy admins may now set or change aws_session_tags there; an
unchanged tag set still passes so team admins can edit other fields
Perplexity's Responses payload fails ResponsesAPIResponse validation on
truncation "" and is kept as an unvalidated model, so its usage stays a
plain dict and _stamp_responses_usage_cost raised AttributeError on every
streamed completion once reasoning made the cost non-zero. Validate the
dict into ResponseAPIUsage before stamping, keeping a provider-reported
cost when it carries one.
Resolves LIT-7391
The converse reasoning gate only matched openai.gpt-5, so gpt-6-astra fell through to
Anthropic's thinking block and Bedrock rejected the call with 400 Unknown parameter:
'thinking'. Match any openai.gpt-<digit> model at the three gate sites instead.
Nova 2 lite and pro accept forced tool_choice on Converse (verified live on
us.amazon.nova-2-lite-v1:0), so the nine Nova 2 registry keys now advertise
supports_tool_choice. The invoke dispatcher also forwards json_mode to Nova like it
already does for Anthropic and TwelveLabs.
The Responses path returns a mock for any mock_response string, so the InternalServerError
sentinel never failed there. Point the primary deployment's api_base at a closed port instead,
which fails every endpoint the same way, then parametrize the test over /chat/completions and
/v1/responses and register both on the coverage cell.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
* fix(langfuse): give session-header calls their own trace id
A client that sends only a session header (x-litellm-session-id, a vendor
x-<name>-session-id such as Claude Code's X-Claude-Code-Session-Id, a bare
x-session-id, or the Codex session/thread/conversation family) has that value
stamped into both trace_id and session_id by the proxy. Langfuse upserts a
trace by id, so every turn of a session collapsed into one growing trace and
the Sessions view showed "Total traces: 1"
Detect that aliasing in the Langfuse callback from the request headers the
callback already receives, and use litellm_call_id as the trace id for those
calls. session_id still carries the header value, so the turns stay grouped
under one session. An explicit x-litellm-trace-id, langfuse_trace_id, or
langfuse_existing_trace_id keeps its trace id, including when the caller sets
it to the same value as the session id
Co-authored-by: jesus <jesus@berri.ai>
* test(langfuse): cover direct-SDK callers without proxy request headers
* fix(langfuse): preserve session trace provenance
---------
Co-authored-by: jesus <jesus@berri.ai>
The completed marker batch is now retrieved by its raw provider id with this
run's key, so the proxy prices it inline and the {provider_batch_id}_batch_cost
row must join that key's token and alias. The CheckBatchCost poller only bills
batches it created in the same database, which a stack booted fresh per run
never holds for a completed marker, so the old unified-id assertion had no row
to find. Every run also replays one callback log through
POST /v1/rust_control_plane/logs, the third spend writer, and asserts its row
joins the key like the eight request paths
A team-scoped auto-router is stored under an internal
model_name_{team_id}_{uuid} with the caller-facing name in
model_info.team_public_model_name, and the four pre-routing strategy
registries key on that internal name. A team key asks for the public name,
so the strategy lookup missed, the team early-resolve exit handed back the
marker deployment itself, and every call 400'd with "Unmapped LLM provider".
The strategy lookup now resolves the requested name through the same
team-first, then global, then admin-across-teams deployment resolution the
deployment path uses, and looks the registries up under the model_name of
whatever that resolves to. Both exits of _common_checks_available_deployment
drop strategy markers through one helper, so a marker-only resolution is
rejected as uncallable on every path. The request team id has one reader.
Resolves LIT-7363
Claude-Session: https://claude.ai/code/session_01NU97S7d2FUDDvTk59k53Wp
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
It is a functional e2e test, not a load test, so give it its own workflow instead of a job
inside the load anomaly run. It keeps the Saturday 12:00 UTC cadence and manual dispatch, and
boots the timeout-config proxy with Postgres and Valkey services exactly as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
Add tests/e2e/router/test_redis_timeout_e2e.py against a proxy booted from
tests/e2e/gateway/redis_timeout_ci_config.yml: a real Redis with socket_timeout 0.001, so every
command times out and the circuit breaker opens, plus a primary deployment that always fails and
falls back to a healthy one, so every request carries retry breadcrumbs into cost tracking. The
test drives twenty chat requests through the proxy and asserts each answers within ten seconds,
the last third is no slower than the first, /health/liveliness stays fast, and every request
still reaches the spend log.
Gate it behind the redis_timeout marker and E2E_REDIS_TIMEOUT, exclude it from the per-PR
e2e-changed selector, register the reliability.circuit_breaker.redis_timeout.stays_responsive
cell, and run it as its own job in the weekly load anomaly workflow with a Postgres and Valkey
service. Against a v1.100.0 proxy the run wedges the worker: requests time out and liveliness
stops answering (LIT-6780).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
* fix(proxy): accept non-string callback vars in default_team_settings
A YAML boolean such as turn_off_message_logging: true in a
default_team_settings block failed TeamCallbackMetadata's str-only
callback_vars validation and errored the request before any callback
ran. Stringify the value the same way AddTeamCallback does.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop docstring from default_team_settings bool regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move default_team_settings bool regression test to mapped pre_call_utils suite
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
LiteLLM_JWTKeyMapping_token_fkey was created ON DELETE RESTRICT, so deleting
a virtual key that a JWT mapping pointed at failed with a foreign key
violation on every deletion path (/key/delete, Admin UI, alias delete,
team and user cascades). Declaring onDelete: Cascade on the relation lets
the database clean the mapping up uniformly, so the next JWT call from that
identity re-registers against the newly created key.
Rebase of #33703 onto current staging.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
When a provider returns choices as null, an object, a string or a number, the
converter said the response had no 'choices' even though the key was present in
the raw keys it listed. A shared message now keeps the old wording for a missing
key and names the offending type otherwise.
The cached-stream regression test also pins the chunk count so a leaked extra
chunk fails it.
