The forwarded-keys record moves off the shared metadata dict onto the
per-request kwargs, where _update_kwargs_with_deployment consumes it, so
sibling requests that share a metadata dict (abatch_completion) can no
longer clear it mid-routing. Per-tier litellm_params from the hook
response are never treated as forwarded marker params, and a deployment
only beats a forwarded value when it sets its own, not when it carries a
LiteLLM_Params default such as merge_reasoning_content_in_choices=False.
The final streaming usage frame only carries usage.cost when the proxy runs
with litellm_settings.include_cost_in_streaming_usage: true, and that flag is
readable only off the module-level litellm setting. There is no header, key,
or management route that turns it on per request, so a test cannot ask the
shared e2e proxy for it, and the proxy's config does not live in this repo.
The registry row stays as an uncovered gap with the reason recorded, rather
than being deleted, so the behavior is still on the list of things we want
covered once the gateway config is reachable.
The StreamOptions model, ChatBody.stream_options, Usage.cost, and
AnthropicMessagesResponse.id existed only for that test, so they go with it.
- accept the Live SDK's models/<id> and LiteLLM's vertex_ai/<id> when rewriting the setup model
- keep a dict service account intact instead of stringifying it
- treat same-target deployments holding different credentials as ambiguous
- guard both websocket states before every close so a second close cannot raise
- build the sendable close codes from the public CloseCode enum
Five e2e tests over routes a customer drives through the gateway, each one
pinning a fix that currently has no live coverage.
The dedicated /openai_passthrough prefix used to be swallowed by the
provider-scoped /{provider}/v1/files and /{provider}/v1/batches routes, which
bound "openai_passthrough" as a provider name and failed inside the gateway
before ever reaching OpenAI. Two tests now upload a file and list batches
through that prefix and assert OpenAI's own objects come back.
Streamed /openai_passthrough/v1/responses and /openai_passthrough/v1/embeddings
are relayed to OpenAI but still have to be costed, since the customer budgets
against this traffic. Both used to land a row the gateway could not use: the
streamed responses call logged a zero-cost row under a random id, and
embeddings wrote no row at all. Each test now reconciles the logged spend and
token counts against the response the caller was actually served.
GET /v1/files narrowed its data to the caller's own rows but left first_id and
last_id addressing the shared provider account's page, handing any caller raw
provider file ids belonging to other tenants. The new test asserts both cursors
address rows in the page the caller can see.
ResourceManager.defer now accepts any callable rather than one returning None,
so a delete that answers with a response model can be deferred as-is.
An `auto_router/<alias>` marker entry's litellm_params (for example
`aws_region_name: eu-west-3`) were forwarded onto every routed call with
setdefault and then won the `{**litellm_params, **kwargs}` merge against
the selected tier's own values, so a Bedrock tier pinned to us-east-1 was
called in eu-west-3 and failed with 400.
The hook now records which keys it actually forwarded on the request's
metadata bucket, and `_update_kwargs_with_deployment` drops every
forwarded key the selected deployment defines itself, so marker params
only fill gaps a tier leaves open. Request-supplied values still win over
both. The stamp is stripped from logged metadata like its siblings.
Fixes#37613
`lite whoami` led with "Authenticated" whenever a token file was on disk, even when the
keychain holding the credential would not give it up. The notice about that sat below the
account lines, so the session read as a working one and sent the user looking for the
problem anywhere but the keychain
Taking the secret out of ~/.litellm/token.json stages a replacement and moves it into
place, which needs room for a second file and a directory that will accept a new entry.
A full disk refuses the first and a read-only ~/.litellm the second, and logout gave up
there: it removed the file when it could, dropping the record that the keychain had never
been confirmed clear, so the logout after it reported a clean keychain it never checked
Shortening the file already in place needs neither, so the logout scrub and the legacy
migration now fall back to overwriting it where it lies. On a read-only ~/.litellm the
logout the user asked for now happens, instead of coming back with instructions to delete
the file by hand
On models without supports_mid_conversation_system, a system entry between
an assistant tool_use turn and the user tool_result turn became a user turn
in that position and the provider rejected the request ("tool_use ids were
found without tool_result blocks immediately after"). That run of entries
now goes right after the tool_result turn, where consecutive user turns
merge upstream. The converted turn also carries only role and content, as
the hoist did, so an entry with extra keys no longer 400s with "Extra
inputs are not permitted".
The e2e cache priming re-sends the identical first turn until its own cache
entry reads back before the reminder turn goes out, since Vertex can take a
few seconds to serve a freshly written entry.
Take the resolver and its timeout as parameters of the bounded helper and
bind the vertex one once at module level, so the timeout tests drive an
injected fake instead of patching a shared singleton.
A ~/.litellm that has gone read-only, or one left root-owned by a sudo login,
refuses both the scrubbed rewrite and the removal. The removal was unguarded,
so 'lite logout' ended in a PermissionError traceback with the credential still
readable in the file. It now comes back as an outcome the command reports,
naming the file and what to do about it, and a file that holds no secret is
still not worth alarming anyone over.
A logout run from an install without the keyring package treated a token file
holding its own secret as proof that no keychain entry could exist. That only
holds for the login which wrote the file. A login before it may have had the
package and put its credential in the keychain, where it outlives both the
uninstall and the file that replaced it, so logout reported a clean sweep over
a live credential. Every keychain that cannot be reached is now treated the
same way, and the message says the keychain went unchecked rather than
asserting what is in it.
A logout that could not reach the keychain deleted the token file whenever it
still held its own secret, and the next logout read that missing file as proof
the keychain was clean. It answered the warning the first run had just issued
with "Logged out successfully" while the entry an earlier login left behind was
still live. The file is the only record that something may still be in there,
which is what `_nothing_left_behind` already says it relies on, so keep it and
take only the secret out.
A keychain that did answer is a different case. `SecretStranded` means the entry
is confirmed there and would not delete, and that needs no note in the file,
while keeping one lets every later command read the credential straight back out
of the keychain, which makes "Logged out locally" untrue. That one drops the
file, as it did before.
The secret still goes first either way: a copy that cannot be replaced with a
secret-free one is removed rather than kept.
/audio/transcriptions answers a model-less request with one of two 400s
depending on whether any wildcard deployment is registered at the time, and
every suite shares one proxy, so run order decided which message came back.
The assertion pinned only the no-wildcard wording, so it went red whenever
the model-access-group suite had registered its wildcards first. It now
accepts either message and still holds the error to naming the model
Verified against a live proxy in both states: with a wildcard registered
(the message CI was seeing) and with none (the message the assertion
expected), the suite passes 3/3 either way
A /v1/realtime connection to a Vertex AI Live model accepted the WebSocket
upgrade and then went silent: a stalled Google OAuth token fetch blocked the
handler before any session event, and the eventual failure closed the socket
with a bare 1011 and no error event, so callers saw an open socket, no frames,
and no reason.
Bound the pre-session token fetch with
REALTIME_CREDENTIAL_RESOLUTION_TIMEOUT_SECONDS (20s default) and, on any
realtime failure, send an OpenAI-style error event before closing with a reason
that names the failure. Close reasons are truncated by bytes, not characters,
since an over-long reason makes the close frame itself fail.
The /vertex_ai/live WebSocket passthrough only ever looked at
default_vertex_config and the DEFAULT_VERTEXAI_* env vars, so a proxy whose
Vertex credentials live in the DB as a model entry with use_in_pass_through
had nothing to authenticate with. The upgrade still succeeded and the socket
then closed with a bare 1000 on the first client frame, which gave the client
no way to tell a misconfiguration from a normal end of session.
Credentials now also resolve from the router deployments flagged
use_in_pass_through, preferring the one matching the requested model, and a
failure to mint an access token closes 1011 with a reason naming both ways to
configure it. Upstream closes other than a plain 1000 are relayed to the client
with their code and reason, so Google's own errors reach the caller. The setup
frame's model is rewritten to the full projects/.../publishers/google/models
resource path, which is what Vertex expects and what lets a bare model id or a
gateway alias work over this route.
Seven live e2e tests covering cost-tracking regressions that currently ship
unnoticed: cache-write tokens billed at the cache-creation rate (#34046),
per-component cost_breakdown on the spend row (#31686), cache reads billed at
the cache-read discount on streamed calls (#34812), cache tokens surviving the
anthropic-messages to Responses bridge (#34957), priority-tier rates applied to
input, output and reasoning (#35923, #35925), the per-component response cost
headers summing to the total (#36965), and cost injected into the final usage
frame of an /openai passthrough stream (#36503).
Every test registers its own deployment with a distinct custom rate per
component, so a component billed at the wrong rate cannot pass. The shared
helpers in cost_rows.py encode the one thing the two surfaces disagree on: the
spend row's input_cost is gross of cache while the response's cost-input header
is net of it.
Staging the token file can succeed and the replacement still fail afterwards,
and that is the one save path where the keychain has already taken the new
secret. It was reported as a save that kept nothing, which sends the user
looking for a credential that is sitting in their keychain, and it claimed the
previous login was untouched when the one keychain slot had just been written
over.
Give that path its own outcome and its own notice. The new secret stays where
it is: the entry it replaced went the moment it landed, so no rollback brings
that back, and removing the new one too would turn a login this machine may
still be able to use into no login at all.
The remaining `CredentialNotSaved` paths all leave both stores untouched, so
the reassurance they carry is now true wherever it is printed.
Three ways the credential commands could mislead or hang.
`lite logout` on a machine that never logged in warned that a credential may
be stranded in a keychain it could not check, and told the user to install
keyring to go clear it. There was nothing there. A missing token file is now
read as the evidence it is, because logout keeps a secret-free file behind
whenever the keychain is left unconfirmed, so a later run can tell a machine
with a credential it cannot reach apart from one that never had a login. That
holds on the LITELLM_CLI_DISABLE_KEYRING path too.
`KeyringDiscardsWrites` was handled on the read and erase paths, which cannot
produce it: the null backend returns None from `get_password` rather than
raising, so only a write ever detects it. It now lives on `SecretWrite` alone
and the unreachable arms are gone.
`keyring.set_password` blocks forever under a HOME with no usable login
keychain, which is what containers, CI images, `sudo -H`, and service accounts
run with, and reads answer normally there so nothing cheaper tells them apart.
`lite login` never touched a keychain before this, so a sign-in that simply
never returns would be a new way for it to fail. Writes are pre-flighted with
a throwaway value on a bounded wait, and a keychain that stays silent falls
back to the token file. The real credential is never the thing handed to a
call that might land long after we stopped waiting.
Saving also stages the token file before the keychain is given anything, since
the file is the half a read-only or full directory refuses. A save that cannot
land now leaves both stores as it found them, which matters most when the
login it failed to replace still works.
`make lint` hands every path in the diff against the base branch to `ruff
format --check`, including the ones the branch deleted, so any branch that
moves or removes a file under `litellm/` fails the gate with "No such file or
directory" instead of a formatting complaint.
test-linting.yml already filters those out with `--diff-filter=ACMR`, so the
Makefile was the half that drifted. Match it.
PR #37579 read `text-X hover:text-X` on a shadcn Button as dead weight and
removed the hover half. On the ghost and outline variants it was not dead: both
carry their own `hover:text-foreground`, and the duplicate in the className was
the thing displacing it through tailwind-merge. Dropping it handed the hover
back to the variant, so the Remove button in a team's logging settings, the
chat storage banner's dismiss control, and the collapsed enterprise-usage rail
all lose their colour the moment you point at them.
Each of the three now carries a distinct hover value, following the alpha-step
idiom the rest of that migration used, which restores the colour and keeps
`local/no-noop-hover-variant` satisfied.
Every other hover utility that PR dropped sits on a plain element or a variant
with no competing `hover:text-`, so those stay as they are.
The UI unit test job narrows a pull request to `vitest related <changed
files>`. `related` maps a file to the tests that import it, so a file no
test imports maps to nothing, and `--passWithNoTests` turns that empty
selection into a green job. package.json, package-lock.json, the Vitest,
Tailwind and TypeScript configs and tests/setupTests.ts are all in that
category even though each of them can change the behaviour of every test
in the suite, so a dashboard dependency bump merged having run no unit
tests at all and only got real coverage later, from the full run on the
push to litellm_internal_staging.
Keep `related` for the common case where a pull request only touches
files under src/, and fall back to the full suite as soon as one changed
file sits outside it. The decision lives in
.github/scripts/select_ui_test_scope.sh so it can be tested on its own,
next to the existing classify_changes.sh gate.
* fix(ui): restore hover feedback and dark-mode variants lost in the token migration
PR #37576 mapped hardcoded Tailwind palette classes onto semantic tokens. Two-tone
hover pairs collapsed onto a single token, so 116 hover utilities across 49 files
became identical to their base class and produced no visible feedback, and in seven
files a dark: variant was dropped while its hardcoded light partner survived, leaving
those elements stuck light in dark mode.
Hover states now follow the alpha-step idiom the shadcn primitives already use
(hover:bg-primary/80, hover:bg-success/20): a duplicated hover:text-X or hover:bg-X
becomes /80, hover:border-border becomes hover:border-ring, and a duplicate is
dropped where another hover utility on the element already carries the change. One
transition-colors that no longer animated anything is removed.
For the dark-mode gaps, indigo maps onto info and amber onto warning. There is no
purple token in globals.css, so the purple sites keep their palette classes and get
their dark: partner back.
* fix(ui): add an eslint rule that fails a hover: utility identical to its base
The token migration collapsed two-tone hover pairs by hand, so nothing catches
the next one. `local/no-noop-hover-variant` reads every string literal and
template chunk and errors when a `hover:X` sits alongside a bare `X`, which is
exactly the shape that renders no hover feedback. It ships at error with no
suppression baseline, so the eleven sites that already carried a dead hover
before the migration are fixed here too.
The rule reads one class string at a time, so a base class supplied by a
different ternary branch than its hover partner is left alone: a selected row
whose resting colour already matches its hover colour is deliberate, not a bug.
* fix(ptu): hand the prune a plain delete filter the query builder can serialise
The bounded sweep built its predicate as a read-only mapping view, which the query
builder refuses to serialise, so the nightly job raised as soon as a config-declared
deployment was priced. The charges were already written by then, which is why the run
looked like it had produced its rows.
The in-memory table these tests run against accepts any mapping, so only a live run
caught it. A predicate builder now returns a plain dict and is asserted as one, and the
catch-up pass has a test covering a config-declared reservation.
* refactor(ptu): build the prune predicate in one shot
Both filter shapes are known upfront, so the bounded one is constructed
directly rather than by mutating a value already declared Final.
The catch-up test took two independent clock reads, which disagree across
UTC midnight; it now derives both the reservation start and the expected
last charged day from a single read, matching the three sibling tests.
* refactor(ui): map hardcoded Tailwind palette classes onto semantic tokens
The dashboard painted itself with literal palette utilities (text-gray-500,
bg-blue-50, border-red-200) that resolve to one fixed color regardless of
theme, so the shadcn token layer and its .dark block could never take effect.
A codemod (scripts/codemod-color-tokens.mjs) rewrites 3,232 of those across
254 files onto the existing token scale: neutrals become foreground /
muted-foreground / muted / border / card, and red, green, amber and blue
collapse into destructive, success, warning and info, with the pale -50 to
-300 tints expressed as opacity modifiers on the same token. Hover and focus
variants map to accent so they lift rather than recess. 210 now-redundant
dark: variants are dropped since the tokens carry both modes.
The .dark palette is retuned to a neutral gray ramp with the sidebar recessed
below the content canvas, replacing the blue-tinted shadcn default where the
sidebar read as a full-height card floating on a near-black page.
Nothing sets the .dark class yet, so light mode is unchanged and dark mode
stays inert until a theme toggle lands.
* chore(ui): drop the one-shot color-token codemod script
Arize Phoenix claimed it could run without a prompt_id while its compiler
requires one, so the no-prompt_id fallback could select it and fail instead
of reaching the vector-store hook. It now declines like the other managers.
UI-injected empty vector_store_ids/tags/guardrails on a DB model tripped the
dynamic-param check, and the prompt-management fallback then handed the request
to the first registered prompt manager (e.g. a saved dotprompt), whose sync
path raised "prompt_id is required" as a 500 on every /chat/completions call.
Empty dynamic params no longer count as a trigger, the fallback skips managers
whose should_run_prompt_management declines a None prompt_id, and the sync base
path returns the request unchanged for a None prompt_id like the async path.
A keyring backend can accept a write and keep nothing. That is exactly what
`keyring --disable` and PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring
select, and it raises nothing to distinguish itself, so `lite login` was
handing the credential to a black hole, scrubbing its own copy from
token.json, and printing a success message over a login that no longer
worked. Reading the value back is the only way to tell that backend apart
from a keychain that really stored the secret.
The same rule closes the rest of the gaps. A credential the token file will
not record is taken back out of the keychain instead of being left live on a
machine with no record of it, and is reported rather than raised. The
migration stages its scrubbed file before the keychain is handed anything,
so a directory that will not accept the rewrite stops the move rather than
leaving the secret in two places. Logout no longer reads a key in the file
as proof that the keychain is clear, which was never sound across two
separate runs, and only draws that conclusion when the `keyring` package is
missing outright, where nothing could have reached a keychain at all.
Nothing in the dashboard renders antd any more, so the package and the
scaffolding around it can go. This removes `antd` and
`@ant-design/cssinjs` from package.json, deletes the global StyleProvider
the root layout wrapped every page in, drops the `antd` cascade layer and
the z-index override that lifted Base UI popups over an antd Modal, and
retires the lint rules that policed antd imports and antd class selectors
in tests.
Fifteen test files still carried `vi.mock("antd", ...)` factories for
components that stopped importing antd during the migration. They were
inert, and they resolve the real module, so they would have broken the
moment the package left node_modules.
The compatibility shims keep their behaviour and lose the antd name:
`antdRules`/`antdRequired` become `validatorRules`/`requiredRule`,
`isAntdUrl` becomes `isValidUrl`, and `ABOVE_ANTD_MODAL` becomes
`NESTED_DIALOG_LAYER`. Comments that explain why a contract looks the way
it does still name antd, because that history is the reason.
Removing the file when it could not be rewritten covered a full disk, but not a
~/.litellm that permits neither the rewrite nor the delete, which is what a
`sudo lite login` leaves behind. There the secret was copied into the keychain and
kept in cleartext on disk, so migration widened exposure instead of narrowing it
Migration now only keeps the vault copy if the file's copy is gone. When it is not,
the write is rolled back and the user is left exactly as they were, logged in with
one copy of the credential
Migration moved the secret into the keychain and then suppressed any OSError from
rewriting token.json, so a file that could not be rewritten kept the credential in
cleartext while every command reported success. That file is now removed instead:
signing in again costs one command, a stranded live credential costs the credential
`lite logout` also reported a clean logout whenever the keyring package was missing,
on the reasoning that an install without it could never have stored anything. The
entry belongs to the OS, so a keychain-backed login survives a logout run from a venv
without the cli extra. erase() now reports which keychain state applies, and logout
warns with the advice that fixes each one, staying quiet for file-backed logins whose
token file still carries its own secret
Also pins the migration path's tightening of a world-readable legacy token.json, and
moves the logout tests off patch() onto the injected vault
Adds e2e coverage for batches terminal state and cost write-back, failure
paths, per-backend file content downloads, and two-gateway routing (LIT-5730).
lite ships with every install of litellm, but the keyring package it needs
for keychain storage only ships with the cli extra. Such a user on a Mac was
told 'No OS keychain available' about a machine that plainly has one, with
nothing pointing at the missing package.
The vault now reports which of the three unusable states it is in, so login
can point at the install, name the kill switch, or report a genuinely absent
keychain.
Drop the inline notes on keychain erasure and disk-vs-vault precedence in favour
of docstrings on the two functions that own those rules, and remove a stale
section header and a field note that the code already says plainly.
* feat(ptu): accrue flat cost for PTU deployments declared in config.yaml
The flat-cost rollup reads deployments from LiteLLM_ProxyModelTable, and config.yaml
models never reach that table by design, so a PTU deployment declared there accrued no
flat cost at all while still billing its traffic per token. The provider bills the
reservation whichever file declared it.
The rollup now also reads the deployments the router holds that no database row owns,
identified by db_model, skipping the per-request credential clones that carry
original_model_id and reuse their source's PTU config under a fresh id. Registering such
a deployment zeroes its pricing, since reserved capacity already pays for the traffic it
serves, and leaving a rate unset falls back to the public cost map, which makes the double
charge the default rather than an opt-in.
The rules both halves apply now live in one module. The rollup's test for what it will
charge and the router's test for what to zero have to agree, or a deployment one accepts
and the other declines serves its traffic for free. That module also owns the fields the
write endpoints already zero, so the two paths cannot drift: tiered_pricing is emptied
rather than zeroed because its tiers outrank the rates beside them, the search context
table is written zeroed because an absent one means the provider default, and any further
rate the deployment itself declares is zeroed alongside the standing set.
The prune is bounded to the deployments a run scanned, but only for a run that priced a
config-declared deployment. Deciding a row is garbage on staleness alone stays correct
while every run derives its charges from the same table, so a database-only run sweeps
exactly as it did before; once one host's charges come from a file the others cannot read,
a row it never considered is not evidence of anything.
Behaviour change worth calling out: a zeroed deployment sorts ahead of an unpriced sibling
in QualityRouter's cost tiebreak, where an unset rate previously sorted last. Reserved
capacity really is the cheaper choice, but the ordering moves.
* refactor(ptu): drop a Final rebind and two redundant isinstance guards
The basedpyright budget rejected reassigning a Final in the datetime coercion and
two isinstance calls the router entry's own type already guarantees. Filtering the
built records rather than the raw entries removes both guards and leaves
_router_deployment as the single validator.
