A login the keychain accepted whose token file could not be replaced left the
superseded secret on disk, and the next load preferred the file unconditionally,
so it served the old credential and erased the new one from the keychain on the
way past. The keychain entry now carries the timestamp of the sign-in that minted
it, and the two stores are compared on that instead.
The discovery document is accepted only when its issuer is the --base-url the user
typed and every endpoint and the resource share that origin (RFC 8414 section 3.3),
so a tampered or redirected document can no longer point the code, verifier, or
refresh token at another host. After a failed refresh the re-read token record is
used only when it continues the same credential (same proxy, token endpoint, and
resource) and has not expired, so a concurrent login against a different proxy can
never hand this one its key
When the resolved deployment is provider openai and the model is GPT-5.6 or
newer, the cache control hook now writes prompt_cache_breakpoint on the
targeted content block and sets prompt_cache_options to explicit mode unless
the caller already passed one. The /v1/messages bridges carry the marker
through (the Responses bridge moves a marked system prompt into a developer
message, since top-level instructions cannot hold one). Breakpoint counting
and the stand-down check recognise both marker kinds, and client breakpoints
already present in messages are no longer subtracted from the cap twice.
Fixes#37509
A pre-flight that times out leaves its write parked inside the keychain, holding
it against every later call, so the next read blocks on the main thread with no
timeout of its own. Anything that resolves the credential more than once in a
process hits it: an SDK Client built a second time never returns.
The vault now remembers the silence and reports the keychain unreachable for the
rest of the process rather than queueing behind the parked call.
The two tests that assert on reasoning cost read reasoning_tokens off the
response and required it to be nonzero, without ever asking the model to
reason. Both now send reasoning_effort, so the assertion rests on a
parameter the test sets rather than on the model's default behavior.
The cache-breakdown test sends it on its prime call too: OpenAI's prefix
cache keys on the reasoning setting as well as the tokens, so priming at
a different effort never produces a read.
The proxy's OAuth authorization server (dynamic registration, PKCE S256,
loopback redirects, single-use codes, refresh rotation) gains a proxy-API
audience: /authorize?resource=<proxy origin> renders a consent page with
team selection and /token mints the same per-user credential lite login
mints, so a native CLI can sign a user in through the system browser and
call /v1/* with user and team attribution. Adds GET /.well-known/litellm-cli-auth
as the versioned discovery contract for non-Python clients, POST /revoke
(RFC 7009) for logout, and lite login --pkce, lite logout, and
lite auth print-token on the CLI side. Proxy-API grants only ever redirect
to a loopback address and the server never picks a team on the user's behalf.
Fixes#37332
Both /v1/messages bridges (Responses API adapter for openai/* and the
chat-completions adapter) now derive prompt_cache_key from the first 64
characters of metadata.user_id, next to the existing user mapping. The
chat bridge only sets it when the resolved provider advertises
prompt_cache_key in its supported params, so providers that reject
unknown params are unaffected. A prompt_cache_key sent explicitly by the
client always wins over the derived value.
Fixes#37508
`lite up` treats a token record whose key the keychain would not hand over as no
login at all, and that clause had no test: every existing freshness test passed a
record carrying a real key, so deleting the clause left the whole suite green
The base install smoke check now also asserts keyring is absent, which is what
makes the lazy import in cli_keyring meaningful. keyring ships in the cli extra
only, so a plain `pip install litellm` must not be able to reach it
When a full disk refuses the replacement file and a read-only token file
refuses the rewrite in place, the only way left to get the secret off disk
is to remove the file carrying it. That file was also the note saying the
keychain went unchecked, so its absence made the next logout read a
keychain that was never confirmed as one already known to be clean.
Removing it is what frees the room the replacement was refused for, so the
note is written again on the way out and the logout after this one still
warns.
The forwarded-keys record moves off the shared metadata dict onto the
per-request kwargs, where _update_kwargs_with_deployment consumes it, so
sibling requests that share a metadata dict (abatch_completion) can no
longer clear it mid-routing. Per-tier litellm_params from the hook
response are never treated as forwarded marker params, and a deployment
only beats a forwarded value when it sets its own, not when it carries a
LiteLLM_Params default such as merge_reasoning_content_in_choices=False.
The final streaming usage frame only carries usage.cost when the proxy runs
with litellm_settings.include_cost_in_streaming_usage: true, and that flag is
readable only off the module-level litellm setting. There is no header, key,
or management route that turns it on per request, so a test cannot ask the
shared e2e proxy for it, and the proxy's config does not live in this repo.
The registry row stays as an uncovered gap with the reason recorded, rather
than being deleted, so the behavior is still on the list of things we want
covered once the gateway config is reachable.
The StreamOptions model, ChatBody.stream_options, Usage.cost, and
AnthropicMessagesResponse.id existed only for that test, so they go with it.
- accept the Live SDK's models/<id> and LiteLLM's vertex_ai/<id> when rewriting the setup model
- keep a dict service account intact instead of stringifying it
- treat same-target deployments holding different credentials as ambiguous
- guard both websocket states before every close so a second close cannot raise
- build the sendable close codes from the public CloseCode enum
Five e2e tests over routes a customer drives through the gateway, each one
pinning a fix that currently has no live coverage.
The dedicated /openai_passthrough prefix used to be swallowed by the
provider-scoped /{provider}/v1/files and /{provider}/v1/batches routes, which
bound "openai_passthrough" as a provider name and failed inside the gateway
before ever reaching OpenAI. Two tests now upload a file and list batches
through that prefix and assert OpenAI's own objects come back.
Streamed /openai_passthrough/v1/responses and /openai_passthrough/v1/embeddings
are relayed to OpenAI but still have to be costed, since the customer budgets
against this traffic. Both used to land a row the gateway could not use: the
streamed responses call logged a zero-cost row under a random id, and
embeddings wrote no row at all. Each test now reconciles the logged spend and
token counts against the response the caller was actually served.
GET /v1/files narrowed its data to the caller's own rows but left first_id and
last_id addressing the shared provider account's page, handing any caller raw
provider file ids belonging to other tenants. The new test asserts both cursors
address rows in the page the caller can see.
ResourceManager.defer now accepts any callable rather than one returning None,
so a delete that answers with a response model can be deferred as-is.
An `auto_router/<alias>` marker entry's litellm_params (for example
`aws_region_name: eu-west-3`) were forwarded onto every routed call with
setdefault and then won the `{**litellm_params, **kwargs}` merge against
the selected tier's own values, so a Bedrock tier pinned to us-east-1 was
called in eu-west-3 and failed with 400.
The hook now records which keys it actually forwarded on the request's
metadata bucket, and `_update_kwargs_with_deployment` drops every
forwarded key the selected deployment defines itself, so marker params
only fill gaps a tier leaves open. Request-supplied values still win over
both. The stamp is stripped from logged metadata like its siblings.
Fixes#37613
`lite whoami` led with "Authenticated" whenever a token file was on disk, even when the
keychain holding the credential would not give it up. The notice about that sat below the
account lines, so the session read as a working one and sent the user looking for the
problem anywhere but the keychain
Taking the secret out of ~/.litellm/token.json stages a replacement and moves it into
place, which needs room for a second file and a directory that will accept a new entry.
A full disk refuses the first and a read-only ~/.litellm the second, and logout gave up
there: it removed the file when it could, dropping the record that the keychain had never
been confirmed clear, so the logout after it reported a clean keychain it never checked
Shortening the file already in place needs neither, so the logout scrub and the legacy
migration now fall back to overwriting it where it lies. On a read-only ~/.litellm the
logout the user asked for now happens, instead of coming back with instructions to delete
the file by hand
On models without supports_mid_conversation_system, a system entry between
an assistant tool_use turn and the user tool_result turn became a user turn
in that position and the provider rejected the request ("tool_use ids were
found without tool_result blocks immediately after"). That run of entries
now goes right after the tool_result turn, where consecutive user turns
merge upstream. The converted turn also carries only role and content, as
the hoist did, so an entry with extra keys no longer 400s with "Extra
inputs are not permitted".
The e2e cache priming re-sends the identical first turn until its own cache
entry reads back before the reminder turn goes out, since Vertex can take a
few seconds to serve a freshly written entry.
Take the resolver and its timeout as parameters of the bounded helper and
bind the vertex one once at module level, so the timeout tests drive an
injected fake instead of patching a shared singleton.
A ~/.litellm that has gone read-only, or one left root-owned by a sudo login,
refuses both the scrubbed rewrite and the removal. The removal was unguarded,
so 'lite logout' ended in a PermissionError traceback with the credential still
readable in the file. It now comes back as an outcome the command reports,
naming the file and what to do about it, and a file that holds no secret is
still not worth alarming anyone over.
A logout run from an install without the keyring package treated a token file
holding its own secret as proof that no keychain entry could exist. That only
holds for the login which wrote the file. A login before it may have had the
package and put its credential in the keychain, where it outlives both the
uninstall and the file that replaced it, so logout reported a clean sweep over
a live credential. Every keychain that cannot be reached is now treated the
same way, and the message says the keychain went unchecked rather than
asserting what is in it.
A logout that could not reach the keychain deleted the token file whenever it
still held its own secret, and the next logout read that missing file as proof
the keychain was clean. It answered the warning the first run had just issued
with "Logged out successfully" while the entry an earlier login left behind was
still live. The file is the only record that something may still be in there,
which is what `_nothing_left_behind` already says it relies on, so keep it and
take only the secret out.
A keychain that did answer is a different case. `SecretStranded` means the entry
is confirmed there and would not delete, and that needs no note in the file,
while keeping one lets every later command read the credential straight back out
of the keychain, which makes "Logged out locally" untrue. That one drops the
file, as it did before.
The secret still goes first either way: a copy that cannot be replaced with a
secret-free one is removed rather than kept.
/audio/transcriptions answers a model-less request with one of two 400s
depending on whether any wildcard deployment is registered at the time, and
every suite shares one proxy, so run order decided which message came back.
The assertion pinned only the no-wildcard wording, so it went red whenever
the model-access-group suite had registered its wildcards first. It now
accepts either message and still holds the error to naming the model
Verified against a live proxy in both states: with a wildcard registered
(the message CI was seeing) and with none (the message the assertion
expected), the suite passes 3/3 either way
A /v1/realtime connection to a Vertex AI Live model accepted the WebSocket
upgrade and then went silent: a stalled Google OAuth token fetch blocked the
handler before any session event, and the eventual failure closed the socket
with a bare 1011 and no error event, so callers saw an open socket, no frames,
and no reason.
Bound the pre-session token fetch with
REALTIME_CREDENTIAL_RESOLUTION_TIMEOUT_SECONDS (20s default) and, on any
realtime failure, send an OpenAI-style error event before closing with a reason
that names the failure. Close reasons are truncated by bytes, not characters,
since an over-long reason makes the close frame itself fail.
The /vertex_ai/live WebSocket passthrough only ever looked at
default_vertex_config and the DEFAULT_VERTEXAI_* env vars, so a proxy whose
Vertex credentials live in the DB as a model entry with use_in_pass_through
had nothing to authenticate with. The upgrade still succeeded and the socket
then closed with a bare 1000 on the first client frame, which gave the client
no way to tell a misconfiguration from a normal end of session.
Credentials now also resolve from the router deployments flagged
use_in_pass_through, preferring the one matching the requested model, and a
failure to mint an access token closes 1011 with a reason naming both ways to
configure it. Upstream closes other than a plain 1000 are relayed to the client
with their code and reason, so Google's own errors reach the caller. The setup
frame's model is rewritten to the full projects/.../publishers/google/models
resource path, which is what Vertex expects and what lets a bare model id or a
gateway alias work over this route.
Seven live e2e tests covering cost-tracking regressions that currently ship
unnoticed: cache-write tokens billed at the cache-creation rate (#34046),
per-component cost_breakdown on the spend row (#31686), cache reads billed at
the cache-read discount on streamed calls (#34812), cache tokens surviving the
anthropic-messages to Responses bridge (#34957), priority-tier rates applied to
input, output and reasoning (#35923, #35925), the per-component response cost
headers summing to the total (#36965), and cost injected into the final usage
frame of an /openai passthrough stream (#36503).
Every test registers its own deployment with a distinct custom rate per
component, so a component billed at the wrong rate cannot pass. The shared
helpers in cost_rows.py encode the one thing the two surfaces disagree on: the
spend row's input_cost is gross of cache while the response's cost-input header
is net of it.
Staging the token file can succeed and the replacement still fail afterwards,
and that is the one save path where the keychain has already taken the new
secret. It was reported as a save that kept nothing, which sends the user
looking for a credential that is sitting in their keychain, and it claimed the
previous login was untouched when the one keychain slot had just been written
over.
Give that path its own outcome and its own notice. The new secret stays where
it is: the entry it replaced went the moment it landed, so no rollback brings
that back, and removing the new one too would turn a login this machine may
still be able to use into no login at all.
The remaining `CredentialNotSaved` paths all leave both stores untouched, so
the reassurance they carry is now true wherever it is printed.
Three ways the credential commands could mislead or hang.
`lite logout` on a machine that never logged in warned that a credential may
be stranded in a keychain it could not check, and told the user to install
keyring to go clear it. There was nothing there. A missing token file is now
read as the evidence it is, because logout keeps a secret-free file behind
whenever the keychain is left unconfirmed, so a later run can tell a machine
with a credential it cannot reach apart from one that never had a login. That
holds on the LITELLM_CLI_DISABLE_KEYRING path too.
`KeyringDiscardsWrites` was handled on the read and erase paths, which cannot
produce it: the null backend returns None from `get_password` rather than
raising, so only a write ever detects it. It now lives on `SecretWrite` alone
and the unreachable arms are gone.
`keyring.set_password` blocks forever under a HOME with no usable login
keychain, which is what containers, CI images, `sudo -H`, and service accounts
run with, and reads answer normally there so nothing cheaper tells them apart.
`lite login` never touched a keychain before this, so a sign-in that simply
never returns would be a new way for it to fail. Writes are pre-flighted with
a throwaway value on a bounded wait, and a keychain that stays silent falls
back to the token file. The real credential is never the thing handed to a
call that might land long after we stopped waiting.
Saving also stages the token file before the keychain is given anything, since
the file is the half a read-only or full directory refuses. A save that cannot
land now leaves both stores as it found them, which matters most when the
login it failed to replace still works.
The UI unit test job narrows a pull request to `vitest related <changed
files>`. `related` maps a file to the tests that import it, so a file no
test imports maps to nothing, and `--passWithNoTests` turns that empty
selection into a green job. package.json, package-lock.json, the Vitest,
Tailwind and TypeScript configs and tests/setupTests.ts are all in that
category even though each of them can change the behaviour of every test
in the suite, so a dashboard dependency bump merged having run no unit
tests at all and only got real coverage later, from the full run on the
push to litellm_internal_staging.
Keep `related` for the common case where a pull request only touches
files under src/, and fall back to the full suite as soon as one changed
file sits outside it. The decision lives in
.github/scripts/select_ui_test_scope.sh so it can be tested on its own,
next to the existing classify_changes.sh gate.
* fix(ptu): hand the prune a plain delete filter the query builder can serialise
The bounded sweep built its predicate as a read-only mapping view, which the query
builder refuses to serialise, so the nightly job raised as soon as a config-declared
deployment was priced. The charges were already written by then, which is why the run
looked like it had produced its rows.
The in-memory table these tests run against accepts any mapping, so only a live run
caught it. A predicate builder now returns a plain dict and is asserted as one, and the
catch-up pass has a test covering a config-declared reservation.
* refactor(ptu): build the prune predicate in one shot
Both filter shapes are known upfront, so the bounded one is constructed
directly rather than by mutating a value already declared Final.
The catch-up test took two independent clock reads, which disagree across
UTC midnight; it now derives both the reservation start and the expected
last charged day from a single read, matching the three sibling tests.
Arize Phoenix claimed it could run without a prompt_id while its compiler
requires one, so the no-prompt_id fallback could select it and fail instead
of reaching the vector-store hook. It now declines like the other managers.
UI-injected empty vector_store_ids/tags/guardrails on a DB model tripped the
dynamic-param check, and the prompt-management fallback then handed the request
to the first registered prompt manager (e.g. a saved dotprompt), whose sync
path raised "prompt_id is required" as a 500 on every /chat/completions call.
Empty dynamic params no longer count as a trigger, the fallback skips managers
whose should_run_prompt_management declines a None prompt_id, and the sync base
path returns the request unchanged for a None prompt_id like the async path.
A keyring backend can accept a write and keep nothing. That is exactly what
`keyring --disable` and PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring
select, and it raises nothing to distinguish itself, so `lite login` was
handing the credential to a black hole, scrubbing its own copy from
token.json, and printing a success message over a login that no longer
worked. Reading the value back is the only way to tell that backend apart
from a keychain that really stored the secret.
The same rule closes the rest of the gaps. A credential the token file will
not record is taken back out of the keychain instead of being left live on a
machine with no record of it, and is reported rather than raised. The
migration stages its scrubbed file before the keychain is handed anything,
so a directory that will not accept the rewrite stops the move rather than
leaving the secret in two places. Logout no longer reads a key in the file
as proof that the keychain is clear, which was never sound across two
separate runs, and only draws that conclusion when the `keyring` package is
missing outright, where nothing could have reached a keychain at all.
Removing the file when it could not be rewritten covered a full disk, but not a
~/.litellm that permits neither the rewrite nor the delete, which is what a
`sudo lite login` leaves behind. There the secret was copied into the keychain and
kept in cleartext on disk, so migration widened exposure instead of narrowing it
Migration now only keeps the vault copy if the file's copy is gone. When it is not,
the write is rolled back and the user is left exactly as they were, logged in with
one copy of the credential
Migration moved the secret into the keychain and then suppressed any OSError from
rewriting token.json, so a file that could not be rewritten kept the credential in
cleartext while every command reported success. That file is now removed instead:
signing in again costs one command, a stranded live credential costs the credential
`lite logout` also reported a clean logout whenever the keyring package was missing,
on the reasoning that an install without it could never have stored anything. The
entry belongs to the OS, so a keychain-backed login survives a logout run from a venv
without the cli extra. erase() now reports which keychain state applies, and logout
warns with the advice that fixes each one, staying quiet for file-backed logins whose
token file still carries its own secret
Also pins the migration path's tightening of a world-readable legacy token.json, and
moves the logout tests off patch() onto the injected vault
Adds e2e coverage for batches terminal state and cost write-back, failure
paths, per-backend file content downloads, and two-gateway routing (LIT-5730).
lite ships with every install of litellm, but the keyring package it needs
for keychain storage only ships with the cli extra. Such a user on a Mac was
told 'No OS keychain available' about a machine that plainly has one, with
nothing pointing at the missing package.
The vault now reports which of the three unusable states it is in, so login
can point at the install, name the kill switch, or report a genuinely absent
keychain.
