The Responses-API to /chat/completions bridge yields ModelResponseStream
chunks that carry choices followed by a trailing event object that has no
choices key. stream_chunk_builder assumed every chunk was subscriptable at
"choices", so assembling those chunks raised KeyError('choices') and was
re-wrapped as a 500 APIError building the streaming usage.
Guard each choices access with .get("choices") so choices-less chunks are
skipped instead of crashing. Behavior is unchanged for chunks that do carry
choices, since .get("choices") is truthy only for a non-empty choices list.
Adds a regression test that assembles content across chunks followed by a
trailing chunk with no choices key.
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
The LLM classifier's conversation context cut each prior turn head-only, so a turn
opening with an incident report and closing with the actual request reached the
classifier as the incident report alone. Keeping head and tail costs the same
budget and is what the truncation literature measures as best for classifying
long text.
On mapped pass-through routes, of which /vertex_ai is one,
user_api_key_auth accepts the caller key from a header literally named
litellm_user_api_key and applies it last, so it overrides every other source.
The credential-less filter neither dropped it nor resolved the caller key from
it, so a virtual key there reached Google past a real x-goog-api-key, and a
bring-your-own Authorization could be stripped when auth actually came from that
header. Drop it by name and resolve it at highest precedence.
* refactor(ui): install the shadcn field primitive
`components/shared/form/field.tsx` was the upstream base-vega `field` source
living outside `components/ui/`. It exported the same ten symbols as upstream,
so `npx shadcn add` could never update it and it had already drifted: its
`FieldLabel` was missing the hover and focus-visible ring utilities upstream
now ships for labels that wrap a nested field.
Install the primitive and point the 77 importers at it. The copy is deleted
rather than kept as a wrapper because it added nothing beyond `forwardRef`,
which React 19 makes unnecessary since `ref` arrives as an ordinary prop.
`field.test.tsx` moves next to the primitive with no edits to its contents,
and its nineteen tests, ref assertions included, pass against the generated
file. That is the evidence the swap is behaviour-preserving.
Two nested-field call sites pick up the upstream hover and focus-visible
styling that the stale copy had been missing.
(cherry picked from commit 947f7fa674c83bfc57f43ad8bfc89c894da947a2)
* test(ui): cover the nested-field interaction cues FieldLabel had lost
The stale copy of `field` was missing the hover, focus-visible and disabled
selectors upstream applies to a label that wraps a nested field, so installing
the primitive restored them with nothing asserting they stay.
Assert the class contract rather than the rendered effect. jsdom evaluates
neither `:has()` nor `:focus-visible`, and Tailwind is not compiled under
vitest, so a test that clicked or tabbed would pass on an element with no
styling at all. Checking the utilities are present is the assertion that
actually fails when they go missing, which is the way they were lost before.
Verified by stripping the four selectors from the primitive: both tests fail,
and both pass once it is restored.
(cherry picked from commit 5a5dbf64270d9d1285dbc4a7af76bb3d927778a8)
The recursive_detector code-quality gate fails on litellm_internal_staging
because _flatten_form_field and _flatten_form_data_field in
llm_request_utils.py are recursive but absent from IGNORE_FUNCTIONS. Both are
bounded structural recursion over an already-parsed JSON-shaped request body
(a finite tree, no cycles possible), matching the existing ignored walkers, so
add them to the ignore list with a justification comment.
Follow-up to #38130. The function has no callers in the repo or the docs and is
not exported from `litellm/__init__.py`, and `token_counter` already does the same
job better, so keeping a second entry point only preserves a trap.
That trap is real: Greptile flagged on #38130 that `token_counter` picks the claude
tokenizer only for bare ids. `claude-sonnet-4-5` resolves to huggingface_tokenizer,
while `claude-3-opus-20240229` and `anthropic/claude-sonnet-4-5` fall back to the
OpenAI one, 24 tokens against 27 on the same string. Deleting the wrapper removes
the surface rather than papering over it; the selection gap in `token_counter`
itself is worth its own fix.
BREAKING CHANGE: `from litellm.utils import prompt_token_calculator` no longer
resolves. Use `litellm.token_counter(model=..., text=...)`.
The dashboard used `cva@1.0.0-beta.4` with the object-argument API behind
`@/lib/cva.config`, while shadcn emits `class-variance-authority` with the
positional API. Every `shadcn add` of a cva-based primitive therefore needed a
hand fix-up before it compiled, which meant `components/ui/` could never match
a fresh CLI run and `shadcn add <name> --diff` reported the whole file as
changed instead of showing real upstream drift.
Swap the dependency, and regenerate `badge`, `button`, `button-group`,
`input-group` and `tabs` straight from the base-vega registry so they are now
byte-identical to the CLI output plus prettier.
Two primitives could not be regenerated because they are local code rather
than registry items, so they move out of `components/ui/`: `sidebar` (203
lines against upstream's 730, and only `leftnav` consumes it) and `meter`
(no registry entry at all, it wraps Base UI's Meter).
The customisations that were baked into the regenerated files move to
wrappers, following the rule that `components/ui/` holds CLI output and
anything on top of it lives outside:
- badge carried info, success and warning variants that duplicated the
existing `StatusBadge` tone map, so its five call sites now use
`StatusBadge`, which gains an optional `className`
- input-group's addon focuses `[data-slot=input-group-control]` rather than
upstream's `input`, which matters because the chat composer puts a textarea
there. That handler now sits at the one call site that needs it
`cx` keeps its previous twMerge behaviour. It came from the old
`defineConfig({hooks: {onComplete: twMerge}})`, and CVA's own `cx` is plain
clsx, so pointing it at `cn` avoids silently dropping conflict resolution in
the six files that use it.
`Sidebar.test.tsx` covers the failure mode this migration can hide: passing
the object form to the positional API is accepted by clsx and renders the
literal class string "base variants defaultVariants", so the component loses
every style while the type checker and the existing suite stay green.
The resolver placed both operator-configured key headers at the top of its
precedence, but user_api_key_auth only overrides with litellm_key_header_name;
a pass_through_endpoints litellm_user_api_key is checked last. So a request that
authenticated via Authorization while also sending a pass-through header could
have the wrong value chosen, leaving the authenticated Authorization key
forwarded. Order the resolver exactly like get_api_key: override first, built-in
headers next, pass-through header last.
user_api_key_auth also accepts the caller key from a pass_through_endpoints
entry's headers.litellm_user_api_key, not just litellm_key_header_name. Drop
every operator-configured caller-key header by name and treat them as
top-precedence caller-key sources, so a virtual key sent through one is never
forwarded to Google.
LoggingWorker._ensure_queue nulled self._queue on a loop change, discarding every
pending LoggingTask (each an un-awaited spend-logging coroutine) with no counter and
only a debug log. SDK callers using asyncio.run() per request and mixed sync/async
processes rebind the queue's loop and silently lose spend rows and observability events.
Drain the stale queue and move the pending tasks onto a fresh queue bound to the new
loop, warn with the carried-over count, and keep flush()/join() honest since the queue
is no longer thrown away. Adds a regression test that fills the queue before the loop
change and asserts every task survives and still executes.
The LIT-4761 streaming-classification tests passed only the bring-your-own
Google OAuth token in Authorization and mocked get_litellm_virtual_key, a shape
that cannot authenticate in production. The credential-less filter now resolves
the caller key by auth precedence, so a lone Authorization value reads as the
key and is stripped. Send the virtual key in x-litellm-api-key, matching a real
request, so Authorization is preserved and the classification assertions run.
Adds the `gemini_family` bundled template to the auto-router tab, a
heuristic-classifier preset alongside the existing Anthropic and OpenAI
family presets.
Tiers ascend in cost across the Gemini lineup:
SIMPLE gemini-2.5-flash-lite $0.10 / $0.40
MEDIUM gemini-3.1-flash-lite $0.25 / $1.50
COMPLEX gemini-3.7-flash $0.75 / $3.75
REASONING gemini-3.1-pro-preview $2.00 / $12.00
Uses concrete model ids rather than Google's `gemini-*-latest` aliases.
Those aliases hot-swap to the newest release of their variation (stable,
preview or experimental) with only a two-week notice, while their rows in
model_prices_and_context_window.json are pinned at 2.5-generation rates,
so a swap onto a 3.x model would bill at the stale price and silently
undercount auto-router spend. A pin test asserts no tier resolves to a
`-latest` alias and that all four rungs are distinct.
A downstream disconnect mid-relay was recording the chunk whose write never
landed, so replay would hand back a byte the record run never delivered. Append
each chunk after its yield returns, and label the truncation from the generator
close, so the recording holds exactly what the proxy received.
The caching-local, proxy-extras and enterprise-package shards each budget
pytest 20m but cap the whole job at 55m. Setup can consume up to 35m, and
the runner adds 5m of overhead, so the job deadline can preempt pytest
inside its own advertised budget and the shard dies without a test report.
check_workflow_startup_safety enforces that invariant and is currently
failing on litellm_internal_staging, which reds the code-quality job for
every open PR. Raising the three caps to 60m satisfies 20 + 35 + 5.
The filter's own Bearer-only stripping missed the other schemes
user_api_key_auth accepts, so a virtual key echoed as `Authorization: Basic
<key>` alongside a higher-precedence auth header did not match the caller key
and was forwarded to Google. Reuse the auth module's _get_bearer_token so the
comparison strips exactly what authentication does (Bearer / bearer / Basic /
AWS4-HMAC-SHA256), falling back to the raw value for a bare token.
Reject shebangs even when preceded by a UTF-8 BOM or leading whitespace,
and stop misclassifying UTF-8 text that happens to start with the
ASCII-printable magics BZh (bzip2) or dex\\n (Android DEX) as archives
or executables by applying the same UTF-8 carve-out already used for MZ
The credential-less filter derived the caller key only from x-litellm-api-key,
Authorization, and the custom header, but the route authenticates through
Depends(user_api_key_auth), which also accepts the key from x-goog-api-key. A
virtual key sent only in x-goog-api-key therefore authenticated yet was kept as
a preserved upstream header and forwarded to Google. Resolve the caller key by
the same precedence get_api_key uses and value-strip exactly that, so a key in
x-goog-api-key is stripped while a real Google key alongside a higher-precedence
virtual key is preserved.
Enumerating credential-bearing kwargs in RETRY_BREADCRUMB_EXCLUDED_KWARGS is always one
new kwarg behind: it missed top-level extra_headers and provider token fields, which
log_retry still copied into router.previous_models verbatim. Scrub the breadcrumb with
mask_credentials_in_payload instead, so credential-named values are masked at any depth
(extra_headers.authorization, api_key, aws_secret_access_key, vertex_credentials,
azure_ad_token, and future kwargs), and leave the exclusion set to the request payload and
router walk state only.
This hardens the in-memory breadcrumb; it is not a fix for a reproduced SpendLogs leak. The
SpendLogs metadata allowlist and the universal previous_models stripping already keep this
breadcrumb off every persisted surface.
Parametrize the regression test over provider_specific_header, extra_headers, and api_key,
asserting the raw credential value never survives into previous_models for any shape while
the container key still reaches the breadcrumb
The record/replay harness stored a streamed provider response as one
buffered body, so a replayed stream arrived coalesced and the
/v1/messages streaming test could not be edge-wired. Keep each SSE
transfer chunk in the bundle in the order the provider sent it (a new
streamed response shape at BUNDLE_FORMAT_VERSION 4) so replay reproduces
the provider's split points, the recorded usage chunk keeps its
position, and a mid-stream upstream error replays as the same
mid-stream error rather than a clean body.
Resolves LIT-5742
Flatten dict-backed multipart bodies so a scalar list becomes one field with a
tuple value, which httpx emits as a repeated part per element, instead of
collapsing to the last element under dict.update. Nested objects still flatten
to key[subkey] like the OpenAI SDK, and the file-tuple video path is untouched.
The hand-rolled drop set missed Ocp-Apim-Subscription-Key, so a caller
Azure APIM secret in that header was forwarded to Google on the
credential-less branch. Derive the name-drop set from the canonical
SpecialHeaders.litellm_credential_header_names(), minus Authorization and
x-goog-api-key which double as real Google credentials and are value-stripped
instead. New credential headers added there are now dropped automatically.
Uploaded files reaching the RAG ingest path were trusted by client
filename and content-type, so archives and executable scripts were
ingested and malicious content was never screened. Enforce controls at
the upload boundary before the file leaves the proxy:
- classify content by magic bytes and a strict UTF-8 decode, never by
the client filename or content-type
- allowlist PDF and UTF-8 text; reject archives and executables/scripts
- cap upload size (512MB) via a bounded read
- run every accepted upload through a dependency-injected malware
scanner, failing closed on scan error; the default scanner flags the
EICAR test file so the hook is validated end to end
- give accepted uploads a server-generated filename so the client name
never reaches storage
- set Content-Disposition attachment and X-Content-Type-Options nosniff
on vector-store file downloads
The router hop _ageneric_api_call_with_fallbacks canonicalises the passthrough
call type onto litellm_metadata, and the cost callback reads spend attribution
from that bucket while only backfilling user_api_key* keys from metadata. The
helper was building on metadata, so agent_id and user_api_end_user_max_budget
were silently dropped before the callback ever saw them. Build and pass the
attribution under litellm_metadata so every field survives.
log_retry copied every kwarg into the previous_models breadcrumb, so a client's
forwarded Authorization (provider_specific_header) and the deployment api_key /
headers rode along in an in-memory structure whose comment says it reaches spend
logs and logging callbacks. Those values have no diagnostic use in a breadcrumb.
Add provider_specific_header, headers, and api_key to RETRY_BREADCRUMB_EXCLUDED_KWARGS
so the credential is never placed there in the first place. This is defense in depth:
no persisted leak exists today, since the SpendLogs metadata allowlist and every
logging integration already drop previous_models before serialization. Removing the
credential at the source means a future logging path cannot expose it either
user_api_key_auth also authenticates a caller from the operator-configured
general_settings.litellm_key_header_name, reading that header straight off
the request, so a virtual key sent there survived the credential-less Vertex
forwarding filter and reached Google alongside a real bring-your-own
credential. Value-strip every header whose value matches the caller's key
from any accepted source, including that custom header.
The claude branch called the anthropic SDK's `Anthropic().count_tokens`, which the
SDK removed, so every claude call raised AttributeError. Counting now goes through
litellm's own token_counter, which handles anthropic models offline and drops the
SDK dependency entirely.
Hiding that was a swallowed error: `except Exception: Exception("Anthropic import
failed please run `pip install anthropic`")` built the exception without raising
it, so an environment missing the SDK fell through to the unguarded
`from anthropic import ...` on the next line and got a bare ModuleNotFoundError
instead of the install hint.
That was the codebase's last PLW0133, so the rule graduates from the ratcheted
budget into ruff.toml where it hard-fails, and editors get the diagnostic inline.
The file-less multipart branch added a third mutually-exclusive request-shape
branch, so response can no longer be Final. Suppress the type-discipline gate
the way the codebase does for other multi-branch locals.
On the credential-less Vertex passthrough branch, drop every header that
can only carry LiteLLM caller auth (x-litellm-api-key, api-key, x-api-key)
by name, since Google never consumes them, and strip the virtual key by
value from Authorization / x-goog-api-key, which may instead hold a genuine
bring-your-own Google credential. This closes the residual leak where a
distinct caller secret in api-key or x-api-key still reached upstream.
