Adds live coverage for three P0 logging-registry cells against a real proxy
and the real sinks: logging.s3.success.writes_object,
logging.s3.failure.writes_object and logging.datadog.failure.exports_metric.
The failure cases drive a genuine upstream rejection; each test registers a
deployment whose provider api_key is invalid, so OpenAI itself returns the
401 and litellm's failure path is what has to deliver. Delivery is then read
back out of the sink, never inferred from the proxy's own response: the s3
tests fetch the object with the AWS SDK and assert the stored payload's
status, model, cost and prompt, while the datadog test reuses the existing
logs-search reader and asserts the event's failure status alongside the
provider's error class, code and name. Every object a test writes is deleted
on teardown.
Both failure tests correlate on the x-litellm-call-id of the attempt they
accepted rather than on the prompt. A rejection at the gateway is logged as a
failure too, carrying the same prompt, so a virtual key that briefly 401s
before the auth cache catches up would otherwise contribute a second record
and turn the exactly-one assertion red on correct behavior.
A logging integration is a process-wide callback rather than a per-request
option, so callback_config.py lets a test declare the destination it needs:
it reads the registered callbacks back from /get/config/callbacks, registers
the missing one through /config/update, and unregisters exactly what it
registered afterwards. A proxy that already ships the integration is left
untouched. Every write is a read-modify-write of the live callback list, so
enabling or disabling a destination cannot clobber a registration made
concurrently by another test on the same proxy.
A read-modify-write is still not atomic and cannot be made so here, because
the config API offers only a whole-list write and a server-side
read-remove-write, with no per-entry update to compare-and-set against. Each
write therefore re-reads the list and fails, naming the entries, if anything
registered beforehand that belongs to someone else has gone, which turns a
silent change to a shared proxy's logging configuration into a diagnosable
failure. Entries that appear only after a write are a later registration
rather than damage and are left alone.
litellm_provider_cache_creation_input_tokens_metric only read the
Anthropic-style top-level usage.cache_creation_input_tokens and had no
prompt_tokens_details fallback, unlike its cache-read twin. OpenAI models
that bill prompt cache writes report them only in
prompt_tokens_details.cache_write_tokens, so the counter never fired for
them. Resolve provider cache read/write tokens through a shared helper
that falls back to prompt_tokens_details.cache_write_tokens (canonical)
then cache_creation_tokens when the explicit top-level field is absent,
and give litellm_input_cache_creation_tokens_metric the same fallback for
raw usage dicts that only carry cache_write_tokens
Same cause as the otel handler test: this file builds its request as a
SimpleNamespace carrying only `state`, and the validation handler now reads
`request.url.path` to pick an error contract, so the fake needs a url
While here, cover what the two existing tests do not. They only exercise the
proxy-wide 422, and the control plane's 400 problem document was reachable only
through the route test, which registers its own copy of the handler in a local
app rather than the real one. Two cases now pin the real handler directly: a
`/management/v1` path returns problem+json with a `detail` string, and paths that
merely resemble the prefix (`/management`, `/v1/management/foo`) keep the 422
shape their callers parse
Both failures are from this branch, not pre-existing
The component allowlist test asserts the gateway and backend route sets union to
the whole app, so any route on neither is a 404 on both pods. Allowlist the
`/management/v1/` prefix on the backend, next to the other control plane
entries, so every resource that moves under it later is covered without a
per-resource edit
The otel handler test builds its request as a SimpleNamespace carrying only
`state`. The validation handler now reads `request.url.path` to decide whether
the caller is on a surface with its own error contract, so the fake needs a url;
a real Request always has one, which is why the handler does not guard for it
The control plane branch returns early, and nothing covered that it still closes
the dangling SERVER span first, so those requests would have leaked a span
apiece. Added a case that pins it; removing the close call fails it
The RFC 9457 `type` was `https://docs.litellm.ai/errors/<slug>`, copied from the
standard's own error example. That path is a 404 and there is no docs section
behind it, so every error body shipped a broken link
RFC 9457 only requires `type` to identify the problem type; it encourages, but
does not require, that dereferencing it yield documentation. An https URI makes a
promise we are not keeping, so use `urn:litellm:error:<slug>` instead, which
carries the same machine-readable identity with nothing to resolve. Switching to
an https base later is a contract change for anyone matching on `type`, so that
should wait for pages that actually exist
A test pins the identifier against regressing to an https docs URL, since the
existing assertion built the expected value from the same constant and would have
stayed green whatever it held
`/customer/aliases` shipped two days ago and has not been in a release, so its
wire contract is still free to change. This lands it on the control-plane
contract before that stops being true, since after a release the path, the param
names and the envelope would all need a permanent legacy adapter
The endpoint becomes `GET /management/v1/spend_logs/end_users`. It is a facet,
the distinct values one column takes over a filtered query on a resource, not an
entity collection; naming it after `customers` implied it listed the end-user
table when it actually reads spend logs, which is a different row set. Serving it
under the parent resource means its filters are the parent's filters, so the
dropdown offers exactly the values the logs table can show without two endpoints
having to keep agreeing on that
Contract changes: `size` becomes `page_size`, `search` becomes `q`, the window
moves from flat `start_date` / `end_date` to `filter[startTime][gte]` / `[lte]`,
and the body becomes `{data, meta, links}`. Unknown query params are now a 400
rather than being silently dropped, because an ignored filter over-returns data.
Errors are RFC 9457 problem documents on this prefix only; every other route
keeps the shape its callers already parse
`links` is what makes the rest deferrable. The dashboard hook follows the
server's `links.next` instead of computing `page + 1`, so moving this to cursor
pagination later changes the links and nothing the client does. That matters
because the inner scan is a sliding window, so offset paging can currently skip
or repeat an end user across pages; the fix is a follow-up, and the hypermedia
means it will not be a breaking one
Cursor mode, `sort`, `include`, ETag / `If-None-Match` and the generic `ListSpec`
framework are all deliberately out of scope here. They are additive or internal,
so none of them needs to beat the release
* fix(guardrails): resolve judge_model credentials via Router in llm_as_a_judge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): wire llm_router into DB-backed judge guardrail init paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert patch endpoint forwards llm_router to sync
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): resolve judge Router lazily and fix wildcard/alias dispatch
Resolve the proxy Router at judge-call time via an injected provider instead of
capturing it at construction, so a DB-backed judge guardrail created before the
Router exists no longer captures None permanently. Select the Router path with
router.get_model_list(model_name=judge_model) so wildcard routes and
model_group_alias keys resolve, not just literal deployment names. Isolate the
judge call from user-traffic routing with num_retries=0 and fallbacks=[].
Revert the llm_router threading through the DB sync/reinit/create/approve/patch
paths since the lazy provider makes it unnecessary. Replace mocked-Router tests
with real Router coverage for plain deployments, model_group_alias, and wildcard
routes, plus lazy per-call resolution.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden judge verdict parsing and guard proxy import
Strip markdown fences and surrounding prose before json.loads so fencing-prone
judge models evaluate instead of failing open, guard the proxy_server import in
_default_router_provider so an unimportable proxy falls back to the SDK, and
snapshot/restore global callback lists in the DB-path judge registry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): reject non-object judge verdicts instead of failing open as success
* fix(guardrails): route hidden model_group_alias judge models through the Router
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Cache web identity STS credentials in the shared IAM cache (restores the
pre-v1.85.0 behavior removed by #27125) and cap the Google OIDC token cache
TTL at the token's own exp claim minus a 60s margin, never caching an
already-expired token
* fix(e2e): stop the cache-settings test from persisting a degraded Redis config
TestCacheSettings.test_update_persists_cache_backend_to_get read the live cache
settings and wrote them back, intending a no-op. Its capture modelled only
type/host/port, so on a TLS cluster the write-back silently dropped `ssl` and
`redis_startup_nodes`.
That is not recoverable on its own. `/cache/settings` persists what it receives
into LiteLLM_CacheConfig, that row outranks the YAML `cache_params`, and
init_cache_settings_in_db re-applies it on a timer, so a restart does not clear
it. The proxy ends up driving a TLS-only cluster endpoint as a plaintext
standalone node and every Redis call blocks to socket timeout.
On the affected deployment that took out rate limiting entirely (the v3 limiter
is a Lua script on Redis with no DB fallback), Redis-only budget levels (tag,
per-model, team-member, per-window), spend tracking, `ResetBudgetJob` (which
self-starved at 54 skipped runs per 15 min), and `ProxyConfig.add_deployment`,
whose last statement syncs guardrails and never ran. 60 of 72 failures in one
run traced back here.
The settings blob is now round-tripped verbatim via a RootModel over an
exhaustive value union, so a subset cannot be written. Two guards make a
regression fail loudly at this test instead of silently downstream:
- refuse to write when GET reports redis_type=cluster but omits
redis_startup_nodes, which is the exact precondition for persisting a
downgrade. GET resolves the stored row overlaid with REDIS_* env and never
reads YAML, so a cluster configured only in YAML cannot round-trip here
- compare /cache/ping before and after, so a write that breaks connectivity
fails this test rather than every suite that follows
The underlying product defect is filed as LIT-4816: GET cannot express the
effective config, and a partial POST is allowed to downgrade transport. This
change only stops the suite from triggering it; the Admin UI can still do so.
basedpyright clean (0 errors) under the e2e gate.
* fix(e2e): scope the bedrock guardrail per request and send OpenAI's current token param
Two failures that had nothing to do with the guardrail or route under test.
create_bedrock_guardrail registered with default_on=True, which applies the
guardrail to every request the proxy serves. The upstream ApplyGuardrail call was
answering 403, and that came back to unrelated traffic as
`403 Bedrock guardrail request failed`, failing three a2a tests and a passthrough
headers test alongside the bedrock one. The harness already supports the
per-request `guardrails` selector, so the guardrail is now registered opted out of
default_on and selected by the test that wants it. A broken upstream guardrail
fails its own test instead of whatever else is running.
Note this only contains the blast radius; the 403 itself still needs the
bedrock:ApplyGuardrail permission (or a valid guardrail identifier) on the
deployment, so test_bedrock_pre_call_blocks_harmful_prompt can still fail on its
own until that is sorted.
The OpenAI passthrough body sent `max_tokens`, which newer models reject with
"Unsupported parameter: 'max_tokens' is not supported with this model. Use
'max_completion_tokens' instead." Passthrough forwards the body untranslated, so
drop_params does not apply and the body has to satisfy OpenAI's contract
directly. vllm_chat keeps max_tokens, which vLLM accepts.
basedpyright clean (0 errors) under the e2e gate.
* fix(e2e): drop the pinned a2a api_key that broke every message/send
#34512 pinned `api_key="os.environ/ANTHROPIC_API_KEY"` on the a2a bridge agent.
The a2a bridge forwards the agent's litellm_params straight into
litellm.acompletion() without expanding "os.environ/" indirection, so that literal
string was sent upstream as x-api-key and every message/send failed with
`AnthropicException - {"type":"authentication_error","message":"invalid x-api-key"}`.
Omitting api_key restores the normal provider resolution: litellm reads
ANTHROPIC_API_KEY from the proxy's own environment for this provider, which is what
the agent-owner flow depends on and what the suite did before #34512.
Verified against a live proxy, same agent shape each time:
api_key omitted -> message/send 200
api_key "os.environ/ANTHROPIC_API_KEY" -> message/send 500 invalid x-api-key
api_key <literal key> -> message/send 200
and the key itself is valid (direct call to api.anthropic.com returns 200), so this
was indirection that never got expanded rather than a bad credential.
This accounts for four failures (test_semver_protocol_version_registers_and_serves,
test_message_send_runs_completion_bridge, test_pinned_v0_3_serves_flat_message_shape,
test_pinned_v1_0_serves_nested_message_shape). They were previously reported as
`403 Bedrock guardrail request failed`, because a default_on Bedrock guardrail
short-circuited the request before it ever reached the bridge and hid this.
The bridge silently ignoring "os.environ/" in agent params is a product defect in
its own right, filed separately; anyone configuring an agent credential that way
through the UI hits the same wall.
basedpyright clean (0 errors) under the e2e gate.
* test(e2e): make the load suite less aggressive against a shared proxy
750 users at spawn rate 50 saturated the request path hard enough to distort the
latency-sensitive suites sharing the same proxy, and it spends real provider money
at that rate. Drop to 200 users at spawn rate 20.
The RPS floor moves with the user count rather than staying put, so the assertion
keeps its meaning instead of becoming a formality: 355 RPS over 750 users is
~0.47 RPS/user, and 90 over 200 holds that same per-user expectation with a
similar pass margin. A request-path regression still trips it.
All four knobs stay env-overridable (E2E_LOAD_USERS, E2E_LOAD_SPAWN_RATE,
E2E_LOAD_DURATION_SECONDS, E2E_LOAD_MIN_RPS) for a deliberate load run.
Note the recorded failure for this test was "no requests completed in 60s", which
was the gateway wedged on unreachable Redis rather than a throughput regression;
this change is about not perturbing its neighbours, not about that failure.
* fix(e2e): make the reasoning-tokens assertion exercise a request that reasons
test_openai_chat_reasoning_reports_reasoning_tokens asked "A train travels 60 miles
in 1.5 hours. What is its average speed in mph?" at reasoning_effort="low", then
asserted reasoning_tokens > 0. The model answers that directly without reasoning, so
0 is correct behavior and the assertion was testing the model's discretion rather
than litellm's reporting.
Verified against a live proxy on a dedicated openai/gpt-5.6 deployment, matching how
the test provisions its model:
reasoning_effort=low, one-step arithmetic -> reasoning_tokens=0
reasoning_effort=high, the prompt used here -> reasoning_tokens=114
Raised to high effort with a prompt that requires a proof plus a search, so the
field under test is actually populated and the assertion fails only if litellm stops
surfacing it.
While confirming this I also checked prompt caching, which needed no change:
cached_tokens comes back 3615 of 3618 prompt tokens on a repeated large prefix
against a dedicated deployment. An earlier reading of 0 was an artifact of probing a
fan-out alias whose requests land on different deployments, not a caching defect.
* test(e2e): skip the files-list test while LIT-4820 is open
GET /v1/files does not include a just-uploaded file. The upload returns 200 and
GET /v1/files/{id} resolves it, but the listing never contains it: the returned set
stays fixed at 27 entries whose newest created_at is roughly ten hours older than
the upload, on both the managed (/v1/files?model=) and provider-scoped
(/openai/v1/files) routes. Polled for 40s, so not an eventual-consistency window.
Filed as LIT-4820. Skipping keeps a known, ticketed product bug from holding the
suite red and masking a new regression somewhere else in the same test.
The assertion is left exactly as it was on purpose. It encodes the contract we
actually want, that a file retrievable by id is also enumerable, and anything that
lists files (a UI picker, cleanup tooling that lists then deletes and would
therefore leak provider-side files) depends on it. Relaxing it to get green would
delete the signal. The skip reason says so and links the ticket, and the ticket
records that removing this marker is part of its definition of done.
Matches the existing pattern in this file, where test_unified_file_and_batch_create
skips with a reason citing LIT-3266.
While skipped, the registry cell llm.files.openai.list.nonstream.works has no
passing covering test, so files-list coverage reports as uncovered rather than
passing, which is the honest state.
* fix(e2e): parse Sentinel node lists in the cache-settings model
The value union covered scalar lists and lists of mappings, but not lists of
lists. `redis_startup_nodes` holds host/port mappings while `sentinel_nodes` holds
positional pairs (CACHE_SETTINGS_FIELDS documents `[['localhost', 26379]]`), so on
a Sentinel deployment pydantic rejected the response:
sentinel_nodes.list[dict[str,...]].1
Input should be a valid dictionary [input_value=['localhost', 26380]]
The round-trip test reads GET /cache/settings before it writes anything, so that
rejection failed the test at the read, before any assertion ran. A Sentinel
deployment would have looked like a broken cache-settings route rather than a
model too narrow to parse a documented shape.
A list element may now be a scalar, a list or a mapping, which covers both node
shapes without special-casing either and tolerates a heterogeneous list instead of
rejecting the whole response.
Adds TestCacheSettingsModel, harness-level with no `e2e` marker so it runs without
a proxy, covering all four backend shapes (cluster mappings, sentinel pairs, plain
node, url mode with a null discrete field) plus transport() key selection.
Confirmed it fails on the previous union and passes on this one:
old union -> 1 failed, 4 passed (the sentinel case)
new union -> 5 passed
* test(e2e): remove the cache-settings round-trip test
The test could not fail for the thing it claimed to test, and could break the
deployment it ran against. Both halves of that are worth stating.
It read the live settings, wrote back identical values, and asserted the read-back
matched. If POST /cache/settings were a complete no-op that returned 200 and touched
nothing, GET would still return the values read a moment earlier and the test would
pass. It verified that GET is stable, not that the route persists anything.
Against that, /cache/settings persists what it receives into LiteLLM_CacheConfig,
that row outranks YAML cache_params, and init_cache_settings_in_db re-applies it on a
timer. A write that omits ssl or redis_startup_nodes converts a TLS cluster into a
plaintext standalone client and every later Redis call blocks to socket timeout. On
2026-07-25 that failed 60 of 72 tests in one run: rate limiting stopped enforcing,
Redis-only budgets admitted billable over-budget spend, ResetBudgetJob self-starved,
and guardrail sync never ran.
Guarding the previous shape was not sufficient. Writing the blob verbatim plus a
cluster precondition and a /cache/ping check narrowed the hazard but did not remove
it, because GET cannot express the effective config: it resolves the stored row
overlaid with REDIS_* env and never reads YAML. On a fresh deploy it cannot see
YAML's ssl to echo back, so a TLS non-cluster deployment could still have a row
written that drops it. No round-trip through this route is safe on a shared proxy.
Removed with the models and helpers it owned, and TestCacheSettingsModel with them
since it existed only to protect that parsing.
The registry row mgmt.cache_settings.update.happy_path stays, now carrying the
rationale for why it is deliberately uncovered and what a safe test would require
(an isolated proxy, or LIT-4816 fixed so a partial write cannot downgrade
transport). Coverage therefore reports this cell as a gap, which is the honest
state. Collector passes --strict; the module still collects 11 tests.
Anthropic cache_control breakpoints are positional: each one caches the
prefix ending at the part that carries it. Compresr flattened every text
part of a message into one string and wrote the compressed result back
into the first text part only, which dropped every later breakpoint and,
when a non-text part sat between text parts, moved the trailing text to
the other side of it.
The positional invariant now has one owner. guardrail_hooks/content_text.py
holds content_to_text alongside is_all_text_parts and
merge_rewritten_text_parts, so a compressed string is only ever written
back over a contiguous run of text parts, and the merged part carries the
last declared breakpoint and its TTL.
Compresr consumes that owner at both ends: _select_targets no longer
selects a row holding a non-text part, and _replace_text_in_content
returns such a row unchanged rather than merging across it. Rows whose
content is a plain string are unaffected.
Mixed rows therefore stop being compressed, which is a deliberate trade;
no single-string write-back can preserve a breakpoint across a non-text
part, so the alternative is silently caching a different prefix than the
caller configured.
* fix(auth): route JWT default-team into memberships instead of the create payload
JWT auto-provisioning (get_user_object with user_id_upsert) merged
litellm.default_internal_user_params verbatim into the Prisma user create,
including a teams key. When a default team is configured through the Admin
UI it is stored as a list of NewUserRequestTeam objects, but the user
table's teams column is String[], so the create raised a Prisma type error
and every JWT-authenticated request 401'd with the user never created.
Mirror the /user/new path: strip teams (and available_teams) out of the
create payload, then route the configured default team through
check_if_default_team_set / add_new_user_to_default_team so provisioned
users get real membership rows. Reuse the synthetic PROXY_ADMIN
UserAPIKeyAuth pattern already used by the team-upsert path to satisfy the
membership permission gate, and import the helpers lazily to avoid the
auth_checks <-> internal_user_endpoints import cycle.
* fix(auth): propagate max_budget_in_team when adding users to default teams
* fix: use pipe union instead of Optional for UP045 budget
* fix(proxy): enforce global max_budget against the resettable proxy budget row
The global proxy budget check compared litellm.max_budget against
SUM(spend) from the MonthlyGlobalSpend view, whose window is hardcoded
to a trailing 30 days. litellm.budget_duration was stored and reset on
a user row that enforcement never read, and startup budgeted the admin
user's own row (default_user_id) instead of the litellm-proxy-budget
aggregate row the spend writer increments per request. Net effect: 1d,
7d and 30d all behaved as a trailing 30 day cap that never reset on the
configured duration.
Startup now upserts the budget onto the litellm-proxy-budget row (and
zeroes lifetime accrual when first putting a row on a reset schedule),
enforcement loads global spend from that row, and ResetBudgetJob drops
the cached global spend accumulator when it resets that row so the cap
unblocks immediately after each window.
Fixes https://github.com/BerriAI/litellm/issues/31292
* refactor(proxy): address review nits on global proxy budget fix
Drop the redundant litellm_proxy_budget_name parameter from
_upsert_proxy_budget_with_reset_at_backfill; its only caller always passed
LITELLM_PROXY_BUDGET_NAME, and any other value would write the budget to a
row enforcement never reads.
Introduce GLOBAL_PROXY_SPEND_CACHE_KEY in constants.py and use it at every
site that previously built the key from litellm_proxy_admin_name (auth
loads, spend-writer increments, startup warm, reset-job invalidation), so
the reader and invalidator can no longer drift apart. The literal key value
is unchanged. Also drop the now-pointless litellm_proxy_admin_name
parameter from _warm_global_spend_cache and the proxy_server import from
the reset-job helper.
The /openai/{endpoint} passthrough forwards a raw OpenAI-format request to
api.openai.com (or OPENAI_API_BASE) with the proxy's OPENAI_API_KEY swapped in,
and still logs a costed pass_through_endpoint SpendLogs row. Nothing exercised
that path end to end.
Adds a live /openai/v1/chat/completions passthrough test that asserts a 2xx
completion and a costed row with custom_llm_provider=openai, mirroring the
gemini and anthropic passthrough cost tests, and registers
llm.chat_completions.openai.passthrough.nonstream.cost_logged.
* fix(proxy): hash caller-supplied key in key update audit log object_id
* test: bound audit-log wait to the captured task instead of gathering the loop
/images/edits is a distinct native route from /images/generations: a multipart
request with the source image sent as the 'image' part plus an edit prompt, not
a JSON body. Nothing exercised it end to end.
Adds a live test that registers an OpenAI image model, sends a small generated
PNG plus an edit prompt to /v1/images/edits, and asserts an image comes back
(b64 or url). Generalizes the multipart transport helper with a file_field
argument (default 'file') so the image part can be named 'image', adds an
image_edit client method, the images_edits endpoint to the coverage schema, and
the llm.images_edits.openai.basic.nonstream.works cell.
Both OpenAI tool-call tests failed with "Function tools with reasoning_effort
are not supported for gpt-5.6 in /v1/chat/completions. To use function tools,
use /v1/responses or set reasoning_effort to 'none'."
This is a provider constraint, not a litellm defect. gpt-5.6 applies a default
reasoning effort, so the raw OpenAI API rejects tools even when the request
sets no reasoning_effort at all; only an explicit "none" is accepted. litellm
does not force that value when tools are present, and gpt-5.6 carries
supports_none_reasoning_effort=True in the model map, so passing it through is
the supported path and keeps these tests on /chat/completions.
Verified against the live stage proxy: the old body still reproduces the 400,
while adding reasoning_effort="none" returns tool_calls=1 non-streaming and
streams tool_calls deltas.
A failed MCP tool call aimed its error.* attributes at request_root_span(),
a ContextVar written on the ASGI request task. A stateful streamable-HTTP
session runs every message on the single task the session's initialize POST
spawned, so inside the message handler that ContextVar still holds the
initialize request's SERVER span. That span ended long ago, so the SDK
dropped every write (five 'Setting attribute on ended span' warnings plus
set_status and _add_event per failed call) and the POST that actually
failed carried no error at all. The identity attributes seeded onto the
server span went the same way.
Publish the live transport span on the ASGI scope of the request being
handled and read it back in the message handler through req_ctx.request,
the Request the streamable-HTTP transport attaches to each message. That
replaces the session-scoped field with a per-message one: a JSON-RPC
response POST deliberately skips the per-session lock, since it can arrive
while the tool call awaiting it is still in flight, so a field on the
shared auth object could be overwritten mid-call and send the tool call's
telemetry to the response's request. A scope also dies with its request
rather than holding a finished span on idle session state.
Publishing re-anchors the request root for the message so guardrail spans
and identity seeding follow, and only a transport still open for writes is
anchored or stamped: a notification POST can answer before the session task
is done, and moving dropped writes from one finished span to another is no
fix. Live capture goes from seven ended-span warnings and an unmarked
transaction to zero warnings and ERROR on the POST that carried the call.
The bedrock guardrail e2e test could never pass on stage. Two reasons.
It sent a bomb-making prompt expecting "stock hate/violence filters" to block,
but the guardrail the suite points at (wk4ijrsk7ska, "husky") has no
contentPolicy at all; it denies the topic and words "bread"/"cake" plus
profanity. ApplyGuardrail returns action=NONE for the old prompt, so the
request passes and the test reports "default-on guardrail did not block".
Send a prompt the configured policy actually denies instead.
It also registered the guardrail with aws_access_key_id /
aws_secret_access_key / aws_region_name set to "os.environ/..." strings. Those
env vars are deliberately absent from the gateway (static AWS keys hijack RDS
IAM auth), and guardrail litellm_params do not expand os.environ/ indirection,
so the literal string reached boto and failed with "Invalid AWS region format:
'os.environ/AWS_REGION'". Drop all three and let the gateway sign
ApplyGuardrail with its own pod-identity role, which is how the standard stack
is meant to reach Bedrock.
Verified against the live stage proxy: registering the guardrail with only
identifier/version and sending the new prompt returns 400 "Violated guardrail
policy", satisfying both assertions.
* test(e2e): point four suites at models the providers still serve
Four llm_translation tests failed against upstream because the model they name
no longer exists. Each replacement was verified against the live stage proxy.
deepseek/deepseek-reasoner is gone; the DeepSeek API now lists only
deepseek-v4-flash and deepseek-v4-pro. Use deepseek/deepseek-v4-pro, which
still returns message.reasoning_content by default and still drops it for both
reasoning_effort="none" and thinking={"type": "disabled"} (litellm maps the
former to the latter, so the provider rejecting a bare "none" does not matter).
amazon.titan-image-generator-v2:0 returns "This model version has reached the
end of its life"; amazon.nova-canvas-v1:0 is the text-to-image model Bedrock
still offers in us-east-1.
Bedrock's Rerank API requires a full model ARN and rejects a bare model id with
"The provided model ARN for reranking is invalid", regardless of model or
region. Pass the ARN for cohere.rerank-v3-5:0, which is available in the
stack's us-east-1.
vertex_ai/gemini-embedding-2 404s as an unknown publisher model on this
project; vertex_ai/text-embedding-005 returns a vector.
* test(e2e): skip the hosted_vllm chat test when its server is unset
test_hosted_vllm_chat_returns_content read os.environ["HOSTED_VLLM_API_BASE"]
directly, so a stack without that env var failed the test with a bare KeyError
instead of reporting an environment gap. The batches suite already skips on the
same variable, and the vertex passthrough tests use pytest.skip for the same
reason, so follow that idiom here.
Drop the HOSTED_VLLM_API_KEY plumbing: the stage vLLM stand-in serves
/v1/chat/completions unauthenticated, and api_key is optional on
LiteLLMParamsBody, so passing it added nothing.
Default the backend to the model that server actually serves,
Qwen/Qwen2.5-0.5B-Instruct-GGUF:Q4_K_M, rather than a Llama id it never had.
Verified against the live stage proxy: a deployment with just that model and
api_base returns "hello".
* fix(model_map): mark deepseek v4-pro and v4-flash as reasoning-capable
Review on #34567 flagged that deepseek/deepseek-v4-pro is not marked
reasoning-capable while the e2e control case requires reasoning_content back
from it. The behavior premise is inverted, but it surfaced a real data gap: the
model map never gained supports_reasoning for the v4 models when DeepSeek
retired deepseek-reasoner, which did carry the flag.
Both models do reason. Against the live API with no reasoning params, v4-pro
returns 106 chars of reasoning_content and v4-flash returns 54, and both drop
it for thinking={"type":"disabled"}.
The stale flag had a real consequence beyond metadata: DeepSeekChatConfig
._thinking_mode_active() gates on supports_reasoning(), so with the flag unset
it returned False even when a caller passed thinking={"type": "enabled"},
skipping the multi-turn check that reasoning_content be passed back on
assistant messages. Param support itself was never gated, which is why
reasoning_effort="none" still mapped to thinking disabled.
Verified with LITELLM_LOCAL_MODEL_COST_MAP=True: supports_reasoning now
reports True for deepseek/deepseek-v4-pro and deepseek/deepseek-v4-flash.
tencent/deepseek-v4-pro is left alone; that route was not exercised here.
/vllm/batches and /vllm/files are high-volume passthrough routes with no e2e
coverage. They ride litellm's generic /vllm/{endpoint} forwarder, so the test
uploads a JSONL through /vllm/v1/files and creates a batch through
/vllm/v1/batches (BatchClient with provider=vllm), asserting the forwarded file
and batch objects come back. Lives next to TestHostedVllmBatch and is skip-marked
for the same reason: no live vLLM server (HOSTED_VLLM_API_BASE) in the e2e env.
Adds the two llm-translation registry cells.
* fix(router): don't cool down parent deployment on advisor sub-call failure
Advisor orchestration issues a sub-call to a different provider/credentials than the selected deployment. When that sub-call fails (e.g. a 401 because no advisor API key is configured), the exception propagates up and the router's deployment_callback_on_failure attributes it to the healthy parent deployment's model_info.id, cooling it down and rejecting unrelated callers to the same model group.
Tag advisor sub-call failures on the exception and skip cooldown for them in deployment_callback_on_failure. The exception is tagged rather than wrapped so its type is preserved and retry/fallback classification and the client-facing error are unchanged. Genuine executor/deployment failures are untagged and still cool down as before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): tag advisor orchestration failures via provider-neutral util
Address review on LIT-4565: move the cooldown-exemption marker into
litellm/router_utils/cooldown_handlers.py so the router imports it at
module top instead of an in-function anthropic import, and extend the
exemption to AdvisorMaxIterationsError so a max-iterations orchestration
failure no longer cools down the healthy executor deployment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): merge model-level guardrails before pre_call_hook
DB/UI-assigned guardrails (litellm_params.guardrails) only fire on
post_call paths today: _check_and_merge_model_level_guardrails is called
in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but
never before pre_call_hook in common_request_processing.py:963. PR #23774
fixed the non-streaming post_call case; pre_call was left broken.
At the pre_call site, add_litellm_data_to_request strips client-supplied
metadata.model_info (pricing spoofing guard) and route_request hasn't run
yet, so model_info.id is unavailable. Extend the helper to fall back to
llm_router.get_deployment_by_model_group_name(model_alias) when model_id
is missing — that uses the O(1) model-name index already maintained by
the router.
Closes#29652
* fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload
Addresses veria-ai Medium finding + proxy-infra CI failure on this PR.
ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool
hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format
was omitting the field, so a key with mcp_rpm_limit could exceed it via the
MCP path.
Reads from kwargs.get("mcp_rate_limit_server_name") to match how
pre_call_tool_check resolves the alias-then-server-name fallback before
invoking hooks.
* fix(proxy): union guardrails across group deployments on alias fallback
Addresses second veria-ai Medium on #29654: the alias fallback called
get_deployment_by_model_group_name(), which returns ONE deployment.
A guardrail set on a non-first deployment would silently not run on
pre_call when the model_id is missing.
Switch to get_model_list(model_name=...) and take the UNION of
litellm_params.guardrails across all matching deployments (with dedup).
Trade-off documented in the comment: pre_call cannot know which
deployment route_request will select, so the conservative choice is to
apply any guardrail set on any eligible deployment.
Updated test stubs to use get_model_list. Added 3 new tests covering
union, dedup, and the all-empty case.
* test(model_level_guardrails): align integration test with get_model_list union API
* fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint
Addresses 3rd veria-ai Medium on #29654:
add_litellm_data_to_request preserves client-supplied metadata.model_info
when the caller's key/team has allow_client_pricing_override. The pre_call
merge previously trusted that id, so a caller could spoof an unguarded
model_info.id while requesting a guarded alias and bypass guardrails.
New `trust_client_model_info: bool` param on the helper. The pre_call
call site passes False; post_call paths (existing) keep True.
Also fixes the ruff failure on the union loop: pulled the .get() into a
local + isinstance(list) check before iterating, so mypy stops complaining
about `object` not being iterable.
2 new regression tests covering spoof-and-bypass + default-trust behavior.
* fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance
Addresses two more reviewer findings on #29654:
veria-ai Medium: route_request resolves team-scoped public model names
with metadata.user_api_key_team_id. The pre_call alias fallback called
get_model_list(model_name=...) without the team_id, so team-scoped
deployments were invisible and their pre_call guardrails silently
skipped. Now reads team_id from metadata or litellm_metadata and passes
it to get_model_list.
greptile P1: the isinstance(deployment_guardrails, list) guard added for
mypy narrowing silently dropped bare-string guardrail values that the
existing post_call path used to truthy-accept. Restored by wrapping a
scalar string into a one-element list on both paths.
4 new tests: team_id passthrough (metadata + litellm_metadata), scalar
on post_call, scalar on alias-union. 36/36 tests pass.
* style: black formatting on _check_and_merge_model_level_guardrails team_id assignment
* chore: ruff format
* fix(lint): remove unused noqa PLR0915 directive
RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic
because PLR0915 is not in this repo's enabled ruff rule set
(lint.extend-select in ruff.toml), so the directive suppresses nothing
and fails the lint job.
* refactor(proxy): hoist guardrail-merge import to module top
The pre_call guardrail-merge helper was imported inside
common_processing_pre_call_logic with a # noqa: PLC0415, which the
type-discipline gate counts as an unexplained suppression (LIT003). The
inline import's cyclic-import justification does not hold: this module
already imports from litellm.proxy.utils at top level, and utils.py does
not import common_request_processing at module load. Fold the helper
into the existing top-level import and drop the inline import, clearing
the suppression instead of budgeting for it.
---------
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Validating the cursor whenever `after` was non-None turned `?after=` into a
400, which the listing has always read as "no cursor". Only a cursor the
client actually sent is looked up now, matching the sibling managed-resource
listing.
An `after` that does not resolve to a batch the caller can list now returns
400 instead of an empty page. An empty page is indistinguishable from the end
of the list, so a stale or malformed cursor silently truncated a client's batch
list. The lookup is scoped to the caller's own rows, so a Prisma cursor can no
longer be anchored to another user's batch.
`has_more` now comes from whether an extra row exists rather than from whether
the page came back full. Reporting fullness made every client fetch one extra
empty page when the batch count was an exact multiple of `limit`, and made a
page shortened by an unparseable row look like the end of the list, hiding the
older batches behind it.
Also drops the unreachable `target_model_names` oversampling branch; that
argument raises a few lines above it.
Two follow-ups from review on the upstream-reported usage contract.
An unusable cost header fell through to the endpoint's flat cost_per_request
instead of the zero the contract promises, so a target that contradicted itself
got billed an estimate it had just disowned. A target that speaks this contract
now owns the cost for the request whether or not the value it sent parsed.
The reported total also cannot be split into prompt and completion, so reading
one out of it under token_rate_limit_type input or output yielded zero and left
the TPM window uncharged; pass-through traffic then ran past a limit it is
meant to share with the general API. Usage that carries no split now charges
its total under every limit type, while usage that does carry one is untouched.
Restore the original PR's behavior on every path that did not already
resolve: no database, a lookup error, a missing managed-file row, or a
row without a storage_url all fall back to dispatching the original id,
which the managed-files deployment hook still maps. This drops the 404
and 503 fail-closed responses I had added, which were the only behaviors
that diverged from litellm_internal_staging.
The change is now strictly additive: when a managed-file row with a
storage_url exists, the unified batch branch substitutes it so providers
like Vertex receive a real gs:// path instead of the opaque token; every
other path behaves exactly as before. Verified live that non-managed,
managed-owner, multi-model load-balanced, and missing-row requests are
byte-identical to base
PassThroughGenericEndpoint.cost_per_request defaults to 0.0, so every
config-defined endpoint forwards a flat 0.0 even when the operator never
configured one, and the success handler applied it over whatever cost was
already established. That silently zeroed the cost an upstream reported for
the request. The flat value is an estimate for targets LiteLLM cannot price,
so it now yields to a target that priced the request itself; it still applies
unchanged when no cost was reported.
The cap was an env-tunable knob in constants.py. Nothing needs to tune it:
it exists so DISTINCT cannot run over an unbounded row set, and picking a
value is a correctness decision, not deployment configuration. An env var
also makes the bound unverifiable, since the same code can behave very
differently between two proxies.
It is now a plain constant next to its only caller, mirroring how
SPEND_LOGS_PAGINATION_COUNT_CAP sits beside ui_view_spend_logs, and it takes
that constant's value: both reads of LiteLLM_SpendLogs now stop at the same
depth. constants.py goes back to matching staging exactly.
The existing test only asserted the parameter equalled the constant, which
is tautological; raising the constant to a billion kept it green while
removing the bound. A second test pins the value against the logs page's
cap, so an arbitrary change to either one fails.
* fix(proxy): attribute spend to org for team-linked keys minted without org_id
Keys attached to an org-linked team but minted without an organization_id
produced spend that was never credited to the org: the spend writer reads
user_api_key_dict.org_id with no team fallback, while the org budget check
resolves the org from the team. The check therefore ran against a counter
fed by almost none of the org's traffic and never tripped.
Backfill org_id from the freshly fetched team object in
_run_centralized_common_checks, per request only, so the spend writer and
the budget check read the same org. A key with an explicitly pinned org_id
always wins, and the cached key row is never mutated, so moving a team to
a different org takes effect on the next auth once the team cache
refreshes.
* test(proxy): cover CLI session-token org backfill from team
CLI session tokens from /sso/cli/poll are minted with a real team_id but
no org_id, and their auth path decrypts the blob without the combined_view
team join that fills org for DB keys. Spend from these tokens reached the
team but never the org, so org budgets never tripped. The regression test
mints a real CLI token, runs it through the centralized checks, and
asserts the credential leaves auth with the team's org.
A pass-through target that fans a single HTTP request out to several models
internally cannot be priced from its response body, so LiteLLM had nothing to
record and every such request landed in the spend logs with zero cost and zero
tokens. The target now reports the totals for the whole request in
x-litellm-response-cost and x-litellm-total-tokens response headers, and
LiteLLM records those values as-is rather than recomputing them.
The headers are read on every upstream response, so a request that burned
tokens before failing still books its spend on the failure row instead of
being dropped for having a 4xx/5xx status. Only what the upstream actually
reported is written, so a target that sends a cost but no token count keeps
the token count LiteLLM derived on its own; a target that sends neither header
is untouched, which is the normal case for Anthropic, Vertex and friends.
Two supporting fixes fall out of this. The rate limiter only pulled token
counts off response shapes it models, so pass-through usage never charged the
TPM window and a team could exceed its shared token limit through pass-through
traffic alone; it now falls back to combined_usage_object. And the streaming
success path reset response_cost unconditionally before the assembled response
recomputed it, which discarded any cost a pass-through handler had already
established (the pass-through branch right below it has always intended to
preserve exactly that).
With end_date omitted the floor was end-anchored to now including its
time-of-day, so an explicit start_date exactly 30 days back parsed as
midnight, compared below the floor, and was invisibly clamped to a
mid-day instant: up to a day of spend disappeared while the response
start_date still printed the full calendar date. Anchoring the floor to
today's UTC midnight makes every comparison in the window derivation
date-pure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
