_build_image_content_item hands a caller-supplied url to
BedrockImageProcessor.process_image_async, which buffers the whole body through
async_safe_get before returning. The 4 MB check runs on bytes that are already
resident, so it rejects an oversized image without preventing the allocation --
and _build_input_content_items gathers these concurrently, so one request with
several urls multiplies it. An arbitrarily large or indefinitely chunked response
is enough to exhaust proxy memory.
async_safe_get takes an optional max_bytes and, when given one, streams the body
and aborts past the cap with PayloadTooLargeError. Omitted, it keeps the previous
buffering, so every existing caller -- including the model-call image paths in
factory.py -- is byte-for-byte unchanged. Only the guardrail passes it.
The rebuilt response drops content-encoding and content-length: aiter_bytes
yields decoded bytes, so carrying those over would describe the body wrongly.
PayloadTooLargeError subclasses ValueError, like SSRFError, so callers already
treating a bad remote response as a rejected fetch need no new except arm. The
guardrail names it explicitly anyway, so an operator reading the log sees "too
large" rather than "could not be read".
The two existing remote-url tests now stub `stream` rather than `get`, which is
the transport the capped path uses. The new test asserts on how many bytes were
pulled -- without that, an unstubbed transport would raise for the wrong reason
and the test would pass against the unfixed code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apply_guardrail reads inputs["images"], but two optimizations return before that
point and both decide what to scan from `texts` alone:
_apply_incremental_request_scan (only_scan_new_messages) sends just the new text
segments and returns. Benign text plus a policy-violating image, on a session
whose text is already cached, is never scanned -- and the proxy still reports the
guardrail as having run. Nothing caches images either, so "already seen" cannot
be established for them in the first place.
_select_messages_for_apply_guardrail (experimental_use_latest_role_message_only)
marks a latest user message with no text as skip_scan. An image-only message is
exactly that shape, so the whole request was dropped from the scan.
An image now forces the full image-aware path in both cases. The optimization is
lost for image-carrying requests -- an incremental turn with an image rescans its
text rather than skipping it -- which is the safe direction: correctness over the
optimization, failing closed rather than silently open.
Falling through skip_scan leaves filtered_messages None; the existing
`*(filtered_messages or ())` unpack and _merge_masked_texts's empty-input guard
already handle that, so the image-only scan needs no other change.
Both tests fail on the code without this change, for the reason named in each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AWS caps images at 4 MB each and 20 per request
(https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-mmfilter.html).
Nothing checked either; BEDROCK_APPLY_GUARDRAIL_CHUNK_BUDGET_CHARS is the text-unit
quota and says nothing about images. An oversized image is now reported through
on_unscannable_image with the measured size instead of surfacing as an opaque AWS
error.
_bin_pack_bedrock_content measured only text, so an image counted as 0 and
`used + 0 <= budget` always held: 45 images packed into a single batch of 45. That
measurement was complete when a content item could only be text; putting images in
the payload is what invalidated it. Packing now carries a second dimension for the
image count. Image bytes are deliberately not charged against `budget`, which is a
different quota.
_apply_guardrail_content_with_chunking splits up front when the image count is over
the limit. Chunking is otherwise reactive, and the substrings
_is_input_too_large_error matches ("text unit", "too long", ...) are all text-shaped,
so an image-count rejection may never reach that fallback. Bisection could not
rescue it either: _split_bedrock_content halves a lone item by its "text", which is
empty for an image, so it gives up and re-raises the original error. Recursion
terminates because every batch _bin_pack_bedrock_content returns is already within
the image limit.
Nested images needed no work here: _extract_tool_result already collects them out of
tool_result blocks, so with apply_guardrail reading inputs["images"] an image inside
a tool result is scanned.
Four tests. The oversized case builds a real 5 MB data URI rather than patching the
decoder, so the size check runs against what the decoder actually produces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The payload fix alone never runs on a real request. ProxyLogging._execute_guardrail_hook
(proxy/utils.py:1216) routes any guardrail that defines `apply_guardrail` through
unified_guardrail unless it sets `use_native_lifecycle_hooks`:
has_apply_guardrail = "apply_guardrail" in type(callback).__dict__ and not getattr(
callback, "use_native_lifecycle_hooks", False)
target = unified_guardrail if has_apply_guardrail else callback
BedrockGuardrail defines apply_guardrail and does not set that flag, so a
/v1/chat/completions request reaches apply_guardrail, which read inputs["texts"] only.
Its own docstring said "images unchanged". Measured against a live guardrail with the
IMAGE modality enabled, same messages, same account:
guardrail.async_pre_call_hook png imageUnits=1 gif blocked 400
ProxyLogging.pre_call_hook png imageUnits=0 gif passed through
The second row is what a proxy user gets, and it matches the imageUnits: 0 the reporter
measured in #35332.
Nothing new is needed on the extraction side. The endpoint translations already populate
inputs["images"]: OpenAIChatCompletionsHandler from `image_url` parts
(openai/chat/guardrail_translation/handler.py:253), AnthropicMessagesHandler from
`image`/`source` blocks. Five guardrails already consume that field (vigil_guard,
custom_code, deepkeep, straiker, generic_guardrail_api), so the contract exists and
Bedrock was the one dropping it.
apply_guardrail now appends the images as one extra user message and lets the normal
message path build the payload, so the unified and native routes share the decoding,
the png/jpeg check and on_unscannable_image instead of drifting apart. Requests that
carry an image but no text are no longer skipped.
_normalize_image_input handles the two shapes that field carries. The OpenAI
translation appends the caller's image_url verbatim, already a data: URI or an https
URL. The Anthropic one returns source["data"] only, dropping media_type, so the entry
is bare base64 that the decoder would reject as unreadable and, under
on_unscannable_image=block, turn a legitimate /v1/messages call into a 400. The format
is sniffed back from the base64 prefix.
Images are only read for input_type == "request"; the OUTPUT source scans
model-generated text.
Three tests, all failing before this change with "image never reached the payload:
['text']".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
initialize_bedrock enumerates its kwargs explicitly, so the new setting parsed
and rendered but never reached the guardrail. An operator who opted into "allow"
kept getting 400s on unscannable images with nothing to explain why. Same shape
as the chunk_budget_chars regression, so the test follows that one and asserts
through initialize_guardrail rather than the constructor.
Also trims the docstrings added by this branch down to the rationale that is not
already obvious from the code, and drops a stale line describing a return
convention these helpers no longer use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A part tagged `type: "image_url"` that also carries a `text` key was scanned as
text and returned before the image branch ran, while the provider transformations
branch on `type` and send it to the model as an image. That let a caller defeat
an IMAGE-modality guardrail, and defeat on_unscannable_image, by pairing the
image with a benign decoy string.
Classify by the declared type first, so an image_url part always takes the image
path regardless of what other fields it carries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Bedrock guardrail built its ApplyGuardrail payload from text content
items only. An OpenAI-format image part has no "text" key, so it was
forwarded to the model but silently dropped from the guardrail payload:
the proxy reported the guardrail as having run while the guardrail never
saw the image (#35332 measured contentPolicyImageUnits: 0 on a request
that demonstrably carried one). BedrockContentItem had no image field
either, and masking dropped image parts out of the request entirely.
- Send image content items in the ApplyGuardrail INPUT payload, decoded
through the existing BedrockImageProcessor.
- Add the image types ApplyGuardrail accepts (png/jpeg only).
- Keep non-text parts when masking rewrites message content.
An image ApplyGuardrail cannot scan (anything but png/jpeg, or a remote
url we refuse to fetch) still reaches the model, so skipping it silently
would let a caller defeat an IMAGE-modality guardrail just by picking a
format the API rejects. on_unscannable_image controls that and defaults
to "block"; set it to "allow" to keep serving such requests unscanned.
Remote urls are only fetched while litellm.user_url_validation is on.
With validation disabled async_safe_get degrades to an unrestricted,
redirect-following GET and the url comes straight from the caller, so
fetching there would turn the guardrail into an SSRF primitive.
Fixes#35332
Supersedes #35338
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A request carrying a session/trace header fans the header value into
litellm metadata as both trace_id and session_id. LangSmith then rejected
the whole ingest batch: a root run's trace_id must equal the run id
embedded in dotted_order (400), and a run-body session_id must reference
an existing tracer session (404/422). Override caller trace_id on runs
that post as roots and drop session_id only when it mirrors trace_id,
so deliberate child-run and valid tracer-session fields still pass through.
A quoted routine call qualified by a schema and sitting inside a CREATE INDEX
expression, ON "Foo" (public."f"(col)), walked its qualifier read-through back
across the opening paren to the ON that introduces the indexed table, so the call
was misread as a relation and dropped from the call set, leaving a rewrite in that
routine unscanned. A word now only introduces the name when nothing but whitespace
and qualifier dots lies between them, so a paren in that gap keeps ON (and any
relation-introducing keyword) from reaching across it and the call stays a call.
The Responses-API to /chat/completions bridge yields ModelResponseStream
chunks that carry choices followed by a trailing event object that has no
choices key. stream_chunk_builder assumed every chunk was subscriptable at
"choices", so assembling those chunks raised KeyError('choices') and was
re-wrapped as a 500 APIError building the streaming usage.
Guard each choices access with .get("choices") so choices-less chunks are
skipped instead of crashing. Behavior is unchanged for chunks that do carry
choices, since .get("choices") is truthy only for a non-empty choices list.
Adds a regression test that assembles content across chunks followed by a
trailing chunk with no choices key.
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
/global/activity/cache_hits now returns an error_breakdown: failed spend
logs bucketed per call_type by error code and error class, read from
metadata->error_information. Clicking a red failed-requests segment on
the cache activity chart opens a per-code bar chart; hovering a bar
lists the error classes behind that code.
The video edit endpoint parsed the multipart body but dropped the uploaded
source video, only normalizing it to an id. When a raw file is uploaded it now
flows through videos.main -> the http handler -> the provider transform, which
emits multipart/form-data with the source video as a file part, matching the
official OpenAI SDK's videos.edit wire format. Edit-by-id still egresses JSON.
The LLM classifier's conversation context cut each prior turn head-only, so a turn
opening with an incident report and closing with the actual request reached the
classifier as the incident report alone. Keeping head and tail costs the same
budget and is what the truncation literature measures as best for classifying
long text.
On mapped pass-through routes, of which /vertex_ai is one,
user_api_key_auth accepts the caller key from a header literally named
litellm_user_api_key and applies it last, so it overrides every other source.
The credential-less filter neither dropped it nor resolved the caller key from
it, so a virtual key there reached Google past a real x-goog-api-key, and a
bring-your-own Authorization could be stripped when auth actually came from that
header. Drop it by name and resolve it at highest precedence.
The recursive_detector code-quality gate fails on litellm_internal_staging
because _flatten_form_field and _flatten_form_data_field in
llm_request_utils.py are recursive but absent from IGNORE_FUNCTIONS. Both are
bounded structural recursion over an already-parsed JSON-shaped request body
(a finite tree, no cycles possible), matching the existing ignored walkers, so
add them to the ignore list with a justification comment.
Follow-up to #38130. The function has no callers in the repo or the docs and is
not exported from `litellm/__init__.py`, and `token_counter` already does the same
job better, so keeping a second entry point only preserves a trap.
That trap is real: Greptile flagged on #38130 that `token_counter` picks the claude
tokenizer only for bare ids. `claude-sonnet-4-5` resolves to huggingface_tokenizer,
while `claude-3-opus-20240229` and `anthropic/claude-sonnet-4-5` fall back to the
OpenAI one, 24 tokens against 27 on the same string. Deleting the wrapper removes
the surface rather than papering over it; the selection gap in `token_counter`
itself is worth its own fix.
BREAKING CHANGE: `from litellm.utils import prompt_token_calculator` no longer
resolves. Use `litellm.token_counter(model=..., text=...)`.
A quoted routine call with a SQL comment between its name and parenthesis,
`"backfill" /* reason */ ()`, is a real call that rewrites rows at boot, but the
call-site check read the raw SQL and stopped at the comment, so the routine was
read as uncalled and its rewrite slipped through. Read the call test from the
masked text instead, where every comment is already blanked to spaces, so a
comment between the name and its parenthesis is skipped exactly as whitespace is,
line, block and nested block comments alike, while a like-named non-call
identifier still opens no call and stays masked
The resolver placed both operator-configured key headers at the top of its
precedence, but user_api_key_auth only overrides with litellm_key_header_name;
a pass_through_endpoints litellm_user_api_key is checked last. So a request that
authenticated via Authorization while also sending a pass-through header could
have the wrong value chosen, leaving the authenticated Authorization key
forwarded. Order the resolver exactly like get_api_key: override first, built-in
headers next, pass-through header last.
A migration that defines an uncalled row-rewriting routine and elsewhere
references a quoted column, table, index, or constraint sharing the routine's
name was wrongly flagged: the guard restored every double-quoted identifier
before the name search, so a like-named identifier read as a call. Restore only
quoted names that open a call, followed by "(", so a routine invoked through a
quoted identifier is still caught while a like-named non-call identifier stays
masked and no rewrite-free migration is rejected
user_api_key_auth also accepts the caller key from a pass_through_endpoints
entry's headers.litellm_user_api_key, not just litellm_key_header_name. Drop
every operator-configured caller-key header by name and treat them as
top-precedence caller-key sources, so a virtual key sent through one is never
forwarded to Google.
LoggingWorker._ensure_queue nulled self._queue on a loop change, discarding every
pending LoggingTask (each an un-awaited spend-logging coroutine) with no counter and
only a debug log. SDK callers using asyncio.run() per request and mixed sync/async
processes rebind the queue's loop and silently lose spend rows and observability events.
Drain the stale queue and move the pending tasks onto a fresh queue bound to the new
loop, warn with the carried-over count, and keep flush()/join() honest since the queue
is no longer thrown away. Adds a regression test that fills the queue before the loop
change and asserts every task survives and still executes.
The LIT-4761 streaming-classification tests passed only the bring-your-own
Google OAuth token in Authorization and mocked get_litellm_virtual_key, a shape
that cannot authenticate in production. The credential-less filter now resolves
the caller key by auth precedence, so a lone Authorization value reads as the
key and is stripped. Send the virtual key in x-litellm-api-key, matching a real
request, so Authorization is preserved and the classification assertions run.
A migration that defines a row-rewriting routine and calls it as
"backfill"() at the top level slipped past the checker, since masking
blanks double-quoted identifiers before the routine-call search runs, so
the call could not be found by name and the body read as uncalled. mask()
now returns those identifier spans and outside_definition puts them back,
so a call written through a quoted identifier reads as the call it is and
the routine's body gets scanned the same as a bare call
the x-litellm-model upload path returns ids wrapped with
encode_file_id_with_model (litellm:<raw>;model,<m> base64'd). chat
completions + /v1/responses forwarded the wrapped id straight to the
provider, breaking openai (file not found / >64 chars), gemini
(unknown mime), etc. wire up the existing get_original_file_id +
is_model_embedded_id helpers in update_messages_with_model_file_ids
and update_responses_input_with_model_file_ids — falls through after
the managed-files path so existing flows are unchanged. 3 new
regression tests + dem proof len 71 -> 26.
A downstream disconnect mid-relay was recording the chunk whose write never
landed, so replay would hand back a byte the record run never delivered. Append
each chunk after its yield returns, and label the truncation from the generator
close, so the recording holds exactly what the proxy received.
The filter's own Bearer-only stripping missed the other schemes
user_api_key_auth accepts, so a virtual key echoed as `Authorization: Basic
<key>` alongside a higher-precedence auth header did not match the caller key
and was forwarded to Google. Reuse the auth module's _get_bearer_token so the
comparison strips exactly what authentication does (Bearer / bearer / Basic /
AWS4-HMAC-SHA256), falling back to the raw value for a bare token.
The DML scan reconstructed a DO/EXECUTE'd literal with undouble, which collapses
each doubled quote to one character and shrinks the text. Every offset after a
collapsed pair then shifted, so a row-rewrite scanned out of the literal reported
an earlier file line and could miss a data-migration-ok marker placed on its real
line. Reconstruct with defuse_escapes instead, turning each doubled quote into a
quote and a space so the pair keeps its two characters and every offset holds,
while a nested -- or /* still stays inside its string.
The credential-less filter derived the caller key only from x-litellm-api-key,
Authorization, and the custom header, but the route authenticates through
Depends(user_api_key_auth), which also accepts the key from x-goog-api-key. A
virtual key sent only in x-goog-api-key therefore authenticated yet was kept as
a preserved upstream header and forwarded to Google. Resolve the caller key by
the same precedence get_api_key uses and value-strip exactly that, so a key in
x-goog-api-key is stripped while a real Google key alongside a higher-precedence
virtual key is preserved.
Enumerating credential-bearing kwargs in RETRY_BREADCRUMB_EXCLUDED_KWARGS is always one
new kwarg behind: it missed top-level extra_headers and provider token fields, which
log_retry still copied into router.previous_models verbatim. Scrub the breadcrumb with
mask_credentials_in_payload instead, so credential-named values are masked at any depth
(extra_headers.authorization, api_key, aws_secret_access_key, vertex_credentials,
azure_ad_token, and future kwargs), and leave the exclusion set to the request payload and
router walk state only.
This hardens the in-memory breadcrumb; it is not a fix for a reproduced SpendLogs leak. The
SpendLogs metadata allowlist and the universal previous_models stripping already keep this
breadcrumb off every persisted surface.
Parametrize the regression test over provider_specific_header, extra_headers, and api_key,
asserting the raw credential value never survives into previous_models for any shape while
the container key still reaches the breadcrumb
The call-detection restore splices into a fixed-position list, so its
.ljust(end - start) holds that length invariant and a test guards it. The
DML-scan recursion instead hands the undoubled literal to a fresh scan_region
as its own region, whose length feeds nothing, so the pad only appends
trailing spaces that shift no keyword and change no reported line. Drop it and
the docstring clause that claimed it kept the offsets landing
Extract a shared _valid_max_results predicate that rejects bools (an int
subclass) and non-positive values, and reuse it from both the connection-mode
request count and the response-side cap so both paths honor the same contract.
