_build_image_content_item hands a caller-supplied url to
BedrockImageProcessor.process_image_async, which buffers the whole body through
async_safe_get before returning. The 4 MB check runs on bytes that are already
resident, so it rejects an oversized image without preventing the allocation --
and _build_input_content_items gathers these concurrently, so one request with
several urls multiplies it. An arbitrarily large or indefinitely chunked response
is enough to exhaust proxy memory.
async_safe_get takes an optional max_bytes and, when given one, streams the body
and aborts past the cap with PayloadTooLargeError. Omitted, it keeps the previous
buffering, so every existing caller -- including the model-call image paths in
factory.py -- is byte-for-byte unchanged. Only the guardrail passes it.
The rebuilt response drops content-encoding and content-length: aiter_bytes
yields decoded bytes, so carrying those over would describe the body wrongly.
PayloadTooLargeError subclasses ValueError, like SSRFError, so callers already
treating a bad remote response as a rejected fetch need no new except arm. The
guardrail names it explicitly anyway, so an operator reading the log sees "too
large" rather than "could not be read".
The two existing remote-url tests now stub `stream` rather than `get`, which is
the transport the capped path uses. The new test asserts on how many bytes were
pulled -- without that, an unstubbed transport would raise for the wrong reason
and the test would pass against the unfixed code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
apply_guardrail reads inputs["images"], but two optimizations return before that
point and both decide what to scan from `texts` alone:
_apply_incremental_request_scan (only_scan_new_messages) sends just the new text
segments and returns. Benign text plus a policy-violating image, on a session
whose text is already cached, is never scanned -- and the proxy still reports the
guardrail as having run. Nothing caches images either, so "already seen" cannot
be established for them in the first place.
_select_messages_for_apply_guardrail (experimental_use_latest_role_message_only)
marks a latest user message with no text as skip_scan. An image-only message is
exactly that shape, so the whole request was dropped from the scan.
An image now forces the full image-aware path in both cases. The optimization is
lost for image-carrying requests -- an incremental turn with an image rescans its
text rather than skipping it -- which is the safe direction: correctness over the
optimization, failing closed rather than silently open.
Falling through skip_scan leaves filtered_messages None; the existing
`*(filtered_messages or ())` unpack and _merge_masked_texts's empty-input guard
already handle that, so the image-only scan needs no other change.
Both tests fail on the code without this change, for the reason named in each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
AWS caps images at 4 MB each and 20 per request
(https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-mmfilter.html).
Nothing checked either; BEDROCK_APPLY_GUARDRAIL_CHUNK_BUDGET_CHARS is the text-unit
quota and says nothing about images. An oversized image is now reported through
on_unscannable_image with the measured size instead of surfacing as an opaque AWS
error.
_bin_pack_bedrock_content measured only text, so an image counted as 0 and
`used + 0 <= budget` always held: 45 images packed into a single batch of 45. That
measurement was complete when a content item could only be text; putting images in
the payload is what invalidated it. Packing now carries a second dimension for the
image count. Image bytes are deliberately not charged against `budget`, which is a
different quota.
_apply_guardrail_content_with_chunking splits up front when the image count is over
the limit. Chunking is otherwise reactive, and the substrings
_is_input_too_large_error matches ("text unit", "too long", ...) are all text-shaped,
so an image-count rejection may never reach that fallback. Bisection could not
rescue it either: _split_bedrock_content halves a lone item by its "text", which is
empty for an image, so it gives up and re-raises the original error. Recursion
terminates because every batch _bin_pack_bedrock_content returns is already within
the image limit.
Nested images needed no work here: _extract_tool_result already collects them out of
tool_result blocks, so with apply_guardrail reading inputs["images"] an image inside
a tool result is scanned.
Four tests. The oversized case builds a real 5 MB data URI rather than patching the
decoder, so the size check runs against what the decoder actually produces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_image_sources returned source["data"] only. An Anthropic image block has three
shapes (types/llms/anthropic.py:259) and only the base64 one carries "data", so
{"type": "url", "url": ...} yielded nothing and the image never reached any
guardrail at all.
This is not Bedrock-specific. Five guardrails consume
GenericGuardrailAPIInputs["images"] (vigil_guard, custom_code, deepkeep, straiker,
generic_guardrail_api) and every one of them was blind to url sources on
/v1/messages.
base64 now returns a data URI rather than the bare payload. A consumer otherwise
has no way to recover media_type, and an API like Bedrock's ApplyGuardrail needs
the format to build its request.
The file shape stays unresolvable here: the bytes live behind the Files API and
this extractor has no client to fetch them. Documented rather than silently
dropped, so a consumer treating a missing entry as "no image to scan" is a known
gap and not a surprise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The payload fix alone never runs on a real request. ProxyLogging._execute_guardrail_hook
(proxy/utils.py:1216) routes any guardrail that defines `apply_guardrail` through
unified_guardrail unless it sets `use_native_lifecycle_hooks`:
has_apply_guardrail = "apply_guardrail" in type(callback).__dict__ and not getattr(
callback, "use_native_lifecycle_hooks", False)
target = unified_guardrail if has_apply_guardrail else callback
BedrockGuardrail defines apply_guardrail and does not set that flag, so a
/v1/chat/completions request reaches apply_guardrail, which read inputs["texts"] only.
Its own docstring said "images unchanged". Measured against a live guardrail with the
IMAGE modality enabled, same messages, same account:
guardrail.async_pre_call_hook png imageUnits=1 gif blocked 400
ProxyLogging.pre_call_hook png imageUnits=0 gif passed through
The second row is what a proxy user gets, and it matches the imageUnits: 0 the reporter
measured in #35332.
Nothing new is needed on the extraction side. The endpoint translations already populate
inputs["images"]: OpenAIChatCompletionsHandler from `image_url` parts
(openai/chat/guardrail_translation/handler.py:253), AnthropicMessagesHandler from
`image`/`source` blocks. Five guardrails already consume that field (vigil_guard,
custom_code, deepkeep, straiker, generic_guardrail_api), so the contract exists and
Bedrock was the one dropping it.
apply_guardrail now appends the images as one extra user message and lets the normal
message path build the payload, so the unified and native routes share the decoding,
the png/jpeg check and on_unscannable_image instead of drifting apart. Requests that
carry an image but no text are no longer skipped.
_normalize_image_input handles the two shapes that field carries. The OpenAI
translation appends the caller's image_url verbatim, already a data: URI or an https
URL. The Anthropic one returns source["data"] only, dropping media_type, so the entry
is bare base64 that the decoder would reject as unreadable and, under
on_unscannable_image=block, turn a legitimate /v1/messages call into a 400. The format
is sniffed back from the base64 prefix.
Images are only read for input_type == "request"; the OUTPUT source scans
model-generated text.
Three tests, all failing before this change with "image never reached the payload:
['text']".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
initialize_bedrock enumerates its kwargs explicitly, so the new setting parsed
and rendered but never reached the guardrail. An operator who opted into "allow"
kept getting 400s on unscannable images with nothing to explain why. Same shape
as the chunk_budget_chars regression, so the test follows that one and asserts
through initialize_guardrail rather than the constructor.
Also trims the docstrings added by this branch down to the rationale that is not
already obvious from the code, and drops a stale line describing a return
convention these helpers no longer use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A part tagged `type: "image_url"` that also carries a `text` key was scanned as
text and returned before the image branch ran, while the provider transformations
branch on `type` and send it to the model as an image. That let a caller defeat
an IMAGE-modality guardrail, and defeat on_unscannable_image, by pairing the
image with a benign decoy string.
Classify by the declared type first, so an image_url part always takes the image
path regardless of what other fields it carries.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Bedrock guardrail built its ApplyGuardrail payload from text content
items only. An OpenAI-format image part has no "text" key, so it was
forwarded to the model but silently dropped from the guardrail payload:
the proxy reported the guardrail as having run while the guardrail never
saw the image (#35332 measured contentPolicyImageUnits: 0 on a request
that demonstrably carried one). BedrockContentItem had no image field
either, and masking dropped image parts out of the request entirely.
- Send image content items in the ApplyGuardrail INPUT payload, decoded
through the existing BedrockImageProcessor.
- Add the image types ApplyGuardrail accepts (png/jpeg only).
- Keep non-text parts when masking rewrites message content.
An image ApplyGuardrail cannot scan (anything but png/jpeg, or a remote
url we refuse to fetch) still reaches the model, so skipping it silently
would let a caller defeat an IMAGE-modality guardrail just by picking a
format the API rejects. on_unscannable_image controls that and defaults
to "block"; set it to "allow" to keep serving such requests unscanned.
Remote urls are only fetched while litellm.user_url_validation is on.
With validation disabled async_safe_get degrades to an unrestricted,
redirect-following GET and the url comes straight from the caller, so
fetching there would turn the guardrail into an SSRF primitive.
Fixes#35332
Supersedes #35338
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A request carrying a session/trace header fans the header value into
litellm metadata as both trace_id and session_id. LangSmith then rejected
the whole ingest batch: a root run's trace_id must equal the run id
embedded in dotted_order (400), and a run-body session_id must reference
an existing tracer session (404/422). Override caller trace_id on runs
that post as roots and drop session_id only when it mirrors trace_id,
so deliberate child-run and valid tracer-session fields still pass through.
The drilldown now self-dismisses when refetched activity has no failures
for its call_type, instead of holding a selection the chart no longer
shows. groupErrorBuckets is rewritten as pure filter/map/sort over the
already-grouped SQL rows.
A quoted routine call qualified by a schema and sitting inside a CREATE INDEX
expression, ON "Foo" (public."f"(col)), walked its qualifier read-through back
across the opening paren to the ON that introduces the indexed table, so the call
was misread as a relation and dropped from the call set, leaving a rewrite in that
routine unscanned. A word now only introduces the name when nothing but whitespace
and qualifier dots lies between them, so a paren in that gap keeps ON (and any
relation-introducing keyword) from reaching across it and the call stays a call.
The Responses-API to /chat/completions bridge yields ModelResponseStream
chunks that carry choices followed by a trailing event object that has no
choices key. stream_chunk_builder assumed every chunk was subscriptable at
"choices", so assembling those chunks raised KeyError('choices') and was
re-wrapped as a 500 APIError building the streaming usage.
Guard each choices access with .get("choices") so choices-less chunks are
skipped instead of crashing. Behavior is unchanged for chunks that do carry
choices, since .get("choices") is truthy only for a non-empty choices list.
Adds a regression test that assembles content across chunks followed by a
trailing chunk with no choices key.
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
/global/activity/cache_hits now returns an error_breakdown: failed spend
logs bucketed per call_type by error code and error class, read from
metadata->error_information. Clicking a red failed-requests segment on
the cache activity chart opens a per-code bar chart; hovering a bar
lists the error classes behind that code.
The video edit endpoint parsed the multipart body but dropped the uploaded
source video, only normalizing it to an id. When a raw file is uploaded it now
flows through videos.main -> the http handler -> the provider transform, which
emits multipart/form-data with the source video as a file part, matching the
official OpenAI SDK's videos.edit wire format. Edit-by-id still egresses JSON.
The LLM classifier's conversation context cut each prior turn head-only, so a turn
opening with an incident report and closing with the actual request reached the
classifier as the incident report alone. Keeping head and tail costs the same
budget and is what the truncation literature measures as best for classifying
long text.
On mapped pass-through routes, of which /vertex_ai is one,
user_api_key_auth accepts the caller key from a header literally named
litellm_user_api_key and applies it last, so it overrides every other source.
The credential-less filter neither dropped it nor resolved the caller key from
it, so a virtual key there reached Google past a real x-goog-api-key, and a
bring-your-own Authorization could be stripped when auth actually came from that
header. Drop it by name and resolve it at highest precedence.
* refactor(ui): install the shadcn field primitive
`components/shared/form/field.tsx` was the upstream base-vega `field` source
living outside `components/ui/`. It exported the same ten symbols as upstream,
so `npx shadcn add` could never update it and it had already drifted: its
`FieldLabel` was missing the hover and focus-visible ring utilities upstream
now ships for labels that wrap a nested field.
Install the primitive and point the 77 importers at it. The copy is deleted
rather than kept as a wrapper because it added nothing beyond `forwardRef`,
which React 19 makes unnecessary since `ref` arrives as an ordinary prop.
`field.test.tsx` moves next to the primitive with no edits to its contents,
and its nineteen tests, ref assertions included, pass against the generated
file. That is the evidence the swap is behaviour-preserving.
Two nested-field call sites pick up the upstream hover and focus-visible
styling that the stale copy had been missing.
(cherry picked from commit 947f7fa674c83bfc57f43ad8bfc89c894da947a2)
* test(ui): cover the nested-field interaction cues FieldLabel had lost
The stale copy of `field` was missing the hover, focus-visible and disabled
selectors upstream applies to a label that wraps a nested field, so installing
the primitive restored them with nothing asserting they stay.
Assert the class contract rather than the rendered effect. jsdom evaluates
neither `:has()` nor `:focus-visible`, and Tailwind is not compiled under
vitest, so a test that clicked or tabbed would pass on an element with no
styling at all. Checking the utilities are present is the assertion that
actually fails when they go missing, which is the way they were lost before.
Verified by stripping the four selectors from the primitive: both tests fail,
and both pass once it is restored.
(cherry picked from commit 5a5dbf64270d9d1285dbc4a7af76bb3d927778a8)
The recursive_detector code-quality gate fails on litellm_internal_staging
because _flatten_form_field and _flatten_form_data_field in
llm_request_utils.py are recursive but absent from IGNORE_FUNCTIONS. Both are
bounded structural recursion over an already-parsed JSON-shaped request body
(a finite tree, no cycles possible), matching the existing ignored walkers, so
add them to the ignore list with a justification comment.
Follow-up to #38130. The function has no callers in the repo or the docs and is
not exported from `litellm/__init__.py`, and `token_counter` already does the same
job better, so keeping a second entry point only preserves a trap.
That trap is real: Greptile flagged on #38130 that `token_counter` picks the claude
tokenizer only for bare ids. `claude-sonnet-4-5` resolves to huggingface_tokenizer,
while `claude-3-opus-20240229` and `anthropic/claude-sonnet-4-5` fall back to the
OpenAI one, 24 tokens against 27 on the same string. Deleting the wrapper removes
the surface rather than papering over it; the selection gap in `token_counter`
itself is worth its own fix.
BREAKING CHANGE: `from litellm.utils import prompt_token_calculator` no longer
resolves. Use `litellm.token_counter(model=..., text=...)`.
The dashboard used `cva@1.0.0-beta.4` with the object-argument API behind
`@/lib/cva.config`, while shadcn emits `class-variance-authority` with the
positional API. Every `shadcn add` of a cva-based primitive therefore needed a
hand fix-up before it compiled, which meant `components/ui/` could never match
a fresh CLI run and `shadcn add <name> --diff` reported the whole file as
changed instead of showing real upstream drift.
Swap the dependency, and regenerate `badge`, `button`, `button-group`,
`input-group` and `tabs` straight from the base-vega registry so they are now
byte-identical to the CLI output plus prettier.
Two primitives could not be regenerated because they are local code rather
than registry items, so they move out of `components/ui/`: `sidebar` (203
lines against upstream's 730, and only `leftnav` consumes it) and `meter`
(no registry entry at all, it wraps Base UI's Meter).
The customisations that were baked into the regenerated files move to
wrappers, following the rule that `components/ui/` holds CLI output and
anything on top of it lives outside:
- badge carried info, success and warning variants that duplicated the
existing `StatusBadge` tone map, so its five call sites now use
`StatusBadge`, which gains an optional `className`
- input-group's addon focuses `[data-slot=input-group-control]` rather than
upstream's `input`, which matters because the chat composer puts a textarea
there. That handler now sits at the one call site that needs it
`cx` keeps its previous twMerge behaviour. It came from the old
`defineConfig({hooks: {onComplete: twMerge}})`, and CVA's own `cx` is plain
clsx, so pointing it at `cn` avoids silently dropping conflict resolution in
the six files that use it.
`Sidebar.test.tsx` covers the failure mode this migration can hide: passing
the object form to the positional API is accepted by clsx and renders the
literal class string "base variants defaultVariants", so the component loses
every style while the type checker and the existing suite stay green.
A quoted routine call with a SQL comment between its name and parenthesis,
`"backfill" /* reason */ ()`, is a real call that rewrites rows at boot, but the
call-site check read the raw SQL and stopped at the comment, so the routine was
read as uncalled and its rewrite slipped through. Read the call test from the
masked text instead, where every comment is already blanked to spaces, so a
comment between the name and its parenthesis is skipped exactly as whitespace is,
line, block and nested block comments alike, while a like-named non-call
identifier still opens no call and stays masked
The resolver placed both operator-configured key headers at the top of its
precedence, but user_api_key_auth only overrides with litellm_key_header_name;
a pass_through_endpoints litellm_user_api_key is checked last. So a request that
authenticated via Authorization while also sending a pass-through header could
have the wrong value chosen, leaving the authenticated Authorization key
forwarded. Order the resolver exactly like get_api_key: override first, built-in
headers next, pass-through header last.
A migration that defines an uncalled row-rewriting routine and elsewhere
references a quoted column, table, index, or constraint sharing the routine's
name was wrongly flagged: the guard restored every double-quoted identifier
before the name search, so a like-named identifier read as a call. Restore only
quoted names that open a call, followed by "(", so a routine invoked through a
quoted identifier is still caught while a like-named non-call identifier stays
masked and no rewrite-free migration is rejected
user_api_key_auth also accepts the caller key from a pass_through_endpoints
entry's headers.litellm_user_api_key, not just litellm_key_header_name. Drop
every operator-configured caller-key header by name and treat them as
top-precedence caller-key sources, so a virtual key sent through one is never
forwarded to Google.
LoggingWorker._ensure_queue nulled self._queue on a loop change, discarding every
pending LoggingTask (each an un-awaited spend-logging coroutine) with no counter and
only a debug log. SDK callers using asyncio.run() per request and mixed sync/async
processes rebind the queue's loop and silently lose spend rows and observability events.
Drain the stale queue and move the pending tasks onto a fresh queue bound to the new
loop, warn with the carried-over count, and keep flush()/join() honest since the queue
is no longer thrown away. Adds a regression test that fills the queue before the loop
change and asserts every task survives and still executes.
The LIT-4761 streaming-classification tests passed only the bring-your-own
Google OAuth token in Authorization and mocked get_litellm_virtual_key, a shape
that cannot authenticate in production. The credential-less filter now resolves
the caller key by auth precedence, so a lone Authorization value reads as the
key and is stripped. Send the virtual key in x-litellm-api-key, matching a real
request, so Authorization is preserved and the classification assertions run.
A migration that defines a row-rewriting routine and calls it as
"backfill"() at the top level slipped past the checker, since masking
blanks double-quoted identifiers before the routine-call search runs, so
the call could not be found by name and the body read as uncalled. mask()
now returns those identifier spans and outside_definition puts them back,
so a call written through a quoted identifier reads as the call it is and
the routine's body gets scanned the same as a bare call
the x-litellm-model upload path returns ids wrapped with
encode_file_id_with_model (litellm:<raw>;model,<m> base64'd). chat
completions + /v1/responses forwarded the wrapped id straight to the
provider, breaking openai (file not found / >64 chars), gemini
(unknown mime), etc. wire up the existing get_original_file_id +
is_model_embedded_id helpers in update_messages_with_model_file_ids
and update_responses_input_with_model_file_ids — falls through after
the managed-files path so existing flows are unchanged. 3 new
regression tests + dem proof len 71 -> 26.
Adds the `gemini_family` bundled template to the auto-router tab, a
heuristic-classifier preset alongside the existing Anthropic and OpenAI
family presets.
Tiers ascend in cost across the Gemini lineup:
SIMPLE gemini-2.5-flash-lite $0.10 / $0.40
MEDIUM gemini-3.1-flash-lite $0.25 / $1.50
COMPLEX gemini-3.7-flash $0.75 / $3.75
REASONING gemini-3.1-pro-preview $2.00 / $12.00
Uses concrete model ids rather than Google's `gemini-*-latest` aliases.
Those aliases hot-swap to the newest release of their variation (stable,
preview or experimental) with only a two-week notice, while their rows in
model_prices_and_context_window.json are pinned at 2.5-generation rates,
so a swap onto a 3.x model would bill at the stale price and silently
undercount auto-router spend. A pin test asserts no tier resolves to a
`-latest` alias and that all four rungs are distinct.
A downstream disconnect mid-relay was recording the chunk whose write never
landed, so replay would hand back a byte the record run never delivered. Append
each chunk after its yield returns, and label the truncation from the generator
close, so the recording holds exactly what the proxy received.
The caching-local, proxy-extras and enterprise-package shards each budget
pytest 20m but cap the whole job at 55m. Setup can consume up to 35m, and
the runner adds 5m of overhead, so the job deadline can preempt pytest
inside its own advertised budget and the shard dies without a test report.
check_workflow_startup_safety enforces that invariant and is currently
failing on litellm_internal_staging, which reds the code-quality job for
every open PR. Raising the three caps to 60m satisfies 20 + 35 + 5.
