The LLM classifier's conversation context cut each prior turn head-only, so a turn
opening with an incident report and closing with the actual request reached the
classifier as the incident report alone. Keeping head and tail costs the same
budget and is what the truncation literature measures as best for classifying
long text.
On mapped pass-through routes, of which /vertex_ai is one,
user_api_key_auth accepts the caller key from a header literally named
litellm_user_api_key and applies it last, so it overrides every other source.
The credential-less filter neither dropped it nor resolved the caller key from
it, so a virtual key there reached Google past a real x-goog-api-key, and a
bring-your-own Authorization could be stripped when auth actually came from that
header. Drop it by name and resolve it at highest precedence.
The recursive_detector code-quality gate fails on litellm_internal_staging
because _flatten_form_field and _flatten_form_data_field in
llm_request_utils.py are recursive but absent from IGNORE_FUNCTIONS. Both are
bounded structural recursion over an already-parsed JSON-shaped request body
(a finite tree, no cycles possible), matching the existing ignored walkers, so
add them to the ignore list with a justification comment.
`AsyncHTTPHandler.__aexit__` and `BaseLLM.__exit__` were declared taking only
`self`, but Python always calls them with (exc_type, exc_val, exc_tb). Any
`async with AsyncHTTPHandler() as client:` therefore raised TypeError while
unwinding, which both skipped the client close and replaced whatever error the
body raised.
The three arguments default to None, matching `httpx.AsyncClient.__aexit__`, so
a bare `await handler.__aexit__()` keeps working and no existing caller changes.
Graduate PLE0302 into ruff.toml so a signature that the protocol cannot call
fails the lint step instead of waiting for a caller to hit it. PYI036 already
had these two annotated wrong, but its budget of 3 absorbed them.
The existing `__aexit__` test called the method bare, matching the broken
signature rather than the protocol, so it passed against the bug. It now drives
the handler through `async with`.
Follow-up to #38130. The function has no callers in the repo or the docs and is
not exported from `litellm/__init__.py`, and `token_counter` already does the same
job better, so keeping a second entry point only preserves a trap.
That trap is real: Greptile flagged on #38130 that `token_counter` picks the claude
tokenizer only for bare ids. `claude-sonnet-4-5` resolves to huggingface_tokenizer,
while `claude-3-opus-20240229` and `anthropic/claude-sonnet-4-5` fall back to the
OpenAI one, 24 tokens against 27 on the same string. Deleting the wrapper removes
the surface rather than papering over it; the selection gap in `token_counter`
itself is worth its own fix.
BREAKING CHANGE: `from litellm.utils import prompt_token_calculator` no longer
resolves. Use `litellm.token_counter(model=..., text=...)`.
The resolver placed both operator-configured key headers at the top of its
precedence, but user_api_key_auth only overrides with litellm_key_header_name;
a pass_through_endpoints litellm_user_api_key is checked last. So a request that
authenticated via Authorization while also sending a pass-through header could
have the wrong value chosen, leaving the authenticated Authorization key
forwarded. Order the resolver exactly like get_api_key: override first, built-in
headers next, pass-through header last.
user_api_key_auth also accepts the caller key from a pass_through_endpoints
entry's headers.litellm_user_api_key, not just litellm_key_header_name. Drop
every operator-configured caller-key header by name and treat them as
top-precedence caller-key sources, so a virtual key sent through one is never
forwarded to Google.
LoggingWorker._ensure_queue nulled self._queue on a loop change, discarding every
pending LoggingTask (each an un-awaited spend-logging coroutine) with no counter and
only a debug log. SDK callers using asyncio.run() per request and mixed sync/async
processes rebind the queue's loop and silently lose spend rows and observability events.
Drain the stale queue and move the pending tasks onto a fresh queue bound to the new
loop, warn with the carried-over count, and keep flush()/join() honest since the queue
is no longer thrown away. Adds a regression test that fills the queue before the loop
change and asserts every task survives and still executes.
The LIT-4761 streaming-classification tests passed only the bring-your-own
Google OAuth token in Authorization and mocked get_litellm_virtual_key, a shape
that cannot authenticate in production. The credential-less filter now resolves
the caller key by auth precedence, so a lone Authorization value reads as the
key and is stripped. Send the virtual key in x-litellm-api-key, matching a real
request, so Authorization is preserved and the classification assertions run.
A downstream disconnect mid-relay was recording the chunk whose write never
landed, so replay would hand back a byte the record run never delivered. Append
each chunk after its yield returns, and label the truncation from the generator
close, so the recording holds exactly what the proxy received.
The filter's own Bearer-only stripping missed the other schemes
user_api_key_auth accepts, so a virtual key echoed as `Authorization: Basic
<key>` alongside a higher-precedence auth header did not match the caller key
and was forwarded to Google. Reuse the auth module's _get_bearer_token so the
comparison strips exactly what authentication does (Bearer / bearer / Basic /
AWS4-HMAC-SHA256), falling back to the raw value for a bare token.
The credential-less filter derived the caller key only from x-litellm-api-key,
Authorization, and the custom header, but the route authenticates through
Depends(user_api_key_auth), which also accepts the key from x-goog-api-key. A
virtual key sent only in x-goog-api-key therefore authenticated yet was kept as
a preserved upstream header and forwarded to Google. Resolve the caller key by
the same precedence get_api_key uses and value-strip exactly that, so a key in
x-goog-api-key is stripped while a real Google key alongside a higher-precedence
virtual key is preserved.
Enumerating credential-bearing kwargs in RETRY_BREADCRUMB_EXCLUDED_KWARGS is always one
new kwarg behind: it missed top-level extra_headers and provider token fields, which
log_retry still copied into router.previous_models verbatim. Scrub the breadcrumb with
mask_credentials_in_payload instead, so credential-named values are masked at any depth
(extra_headers.authorization, api_key, aws_secret_access_key, vertex_credentials,
azure_ad_token, and future kwargs), and leave the exclusion set to the request payload and
router walk state only.
This hardens the in-memory breadcrumb; it is not a fix for a reproduced SpendLogs leak. The
SpendLogs metadata allowlist and the universal previous_models stripping already keep this
breadcrumb off every persisted surface.
Parametrize the regression test over provider_specific_header, extra_headers, and api_key,
asserting the raw credential value never survives into previous_models for any shape while
the container key still reaches the breadcrumb
The record/replay harness stored a streamed provider response as one
buffered body, so a replayed stream arrived coalesced and the
/v1/messages streaming test could not be edge-wired. Keep each SSE
transfer chunk in the bundle in the order the provider sent it (a new
streamed response shape at BUNDLE_FORMAT_VERSION 4) so replay reproduces
the provider's split points, the recorded usage chunk keeps its
position, and a mid-stream upstream error replays as the same
mid-stream error rather than a clean body.
Resolves LIT-5742
Flatten dict-backed multipart bodies so a scalar list becomes one field with a
tuple value, which httpx emits as a repeated part per element, instead of
collapsing to the last element under dict.update. Nested objects still flatten
to key[subkey] like the OpenAI SDK, and the file-tuple video path is untouched.
The hand-rolled drop set missed Ocp-Apim-Subscription-Key, so a caller
Azure APIM secret in that header was forwarded to Google on the
credential-less branch. Derive the name-drop set from the canonical
SpecialHeaders.litellm_credential_header_names(), minus Authorization and
x-goog-api-key which double as real Google credentials and are value-stripped
instead. New credential headers added there are now dropped automatically.
Uploaded files reaching the RAG ingest path were trusted by client
filename and content-type, so archives and executable scripts were
ingested and malicious content was never screened. Enforce controls at
the upload boundary before the file leaves the proxy:
- classify content by magic bytes and a strict UTF-8 decode, never by
the client filename or content-type
- allowlist PDF and UTF-8 text; reject archives and executables/scripts
- cap upload size (512MB) via a bounded read
- run every accepted upload through a dependency-injected malware
scanner, failing closed on scan error; the default scanner flags the
EICAR test file so the hook is validated end to end
- give accepted uploads a server-generated filename so the client name
never reaches storage
- set Content-Disposition attachment and X-Content-Type-Options nosniff
on vector-store file downloads
The router hop _ageneric_api_call_with_fallbacks canonicalises the passthrough
call type onto litellm_metadata, and the cost callback reads spend attribution
from that bucket while only backfilling user_api_key* keys from metadata. The
helper was building on metadata, so agent_id and user_api_end_user_max_budget
were silently dropped before the callback ever saw them. Build and pass the
attribution under litellm_metadata so every field survives.
log_retry copied every kwarg into the previous_models breadcrumb, so a client's
forwarded Authorization (provider_specific_header) and the deployment api_key /
headers rode along in an in-memory structure whose comment says it reaches spend
logs and logging callbacks. Those values have no diagnostic use in a breadcrumb.
Add provider_specific_header, headers, and api_key to RETRY_BREADCRUMB_EXCLUDED_KWARGS
so the credential is never placed there in the first place. This is defense in depth:
no persisted leak exists today, since the SpendLogs metadata allowlist and every
logging integration already drop previous_models before serialization. Removing the
credential at the source means a future logging path cannot expose it either
user_api_key_auth also authenticates a caller from the operator-configured
general_settings.litellm_key_header_name, reading that header straight off
the request, so a virtual key sent there survived the credential-less Vertex
forwarding filter and reached Google alongside a real bring-your-own
credential. Value-strip every header whose value matches the caller's key
from any accepted source, including that custom header.
The claude branch called the anthropic SDK's `Anthropic().count_tokens`, which the
SDK removed, so every claude call raised AttributeError. Counting now goes through
litellm's own token_counter, which handles anthropic models offline and drops the
SDK dependency entirely.
Hiding that was a swallowed error: `except Exception: Exception("Anthropic import
failed please run `pip install anthropic`")` built the exception without raising
it, so an environment missing the SDK fell through to the unguarded
`from anthropic import ...` on the next line and got a bare ModuleNotFoundError
instead of the install hint.
That was the codebase's last PLW0133, so the rule graduates from the ratcheted
budget into ruff.toml where it hard-fails, and editors get the diagnostic inline.
On the credential-less Vertex passthrough branch, drop every header that
can only carry LiteLLM caller auth (x-litellm-api-key, api-key, x-api-key)
by name, since Google never consumes them, and strip the virtual key by
value from Authorization / x-goog-api-key, which may instead hold a genuine
bring-your-own Google credential. This closes the residual leak where a
distinct caller secret in api-key or x-api-key still reached upstream.
Adds a regression asserting the value-based strip also drops the caller's
virtual key when it is duplicated into the api-key and x-api-key headers,
while a genuine bring-your-own Google credential still forwards.
Add an endpoint-level regression test asserting can_user_make_model_call
receives the litellm_params after health_check_params are merged in, so the
merge-before-auth ordering cannot silently regress and let a request smuggle
a field past authorization.
* fix(ui): boot the UI image as an arbitrary uid by anchoring nginx writes under /tmp
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): type the arbitrary-uid image test fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
AzureVideoConfig subclasses OpenAIVideoConfig and so inherits the new
use_multipart_form_data() -> True. Azure's /openai/v1/videos surface is
OpenAI-SDK-compatible, so the JSON->multipart flip is intentional; assert it
through the real handler so the inherited behavior can't silently regress.
The openai/azure/compat image-edit funnel merged non_default_params and
extra_body straight into the multipart body, so a nested value (e.g.
extra_body={"metadata": {...}}) reached the httpx encoder and 500'd with
"Invalid type for value. Expected primitive type". Route the funnel through
a shared flattener that serializes nested values as OpenAI-SDK bracket fields
(key[subkey], lists as key[], bools lowercased, None/empty dropped), matching
the wire format of the rest of this fix.
The credential-less Vertex passthrough dropped the caller's LiteLLM
virtual key only from Authorization by exact match. A caller who sent
the same key in x-goog-api-key (which doubles as a real Google
credential) had it accepted as a credential and forwarded upstream.
Drop the virtual key by value across every forwarded header, normalizing
any Bearer prefix, so no header name carries it to Google.
