* fix(bedrock): map Anthropic batch row params the way real time does
* fix(bedrock): let a batch row's allowed_openai_params reach the mapper
* test(bedrock): assert the batch thinking value matches the real-time mapping
* fix(bedrock): keep json_mode out of Anthropic batch rows and pin route-prefixed deployments
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): match provider-prefixed fallback keys for bare model groups
* fix(router): infer the fallback key's provider the way routing does for bare model groups
A bare model group served by a wildcard deployment (claude-sonnet-4-6 routed to anthropic/*) now finds a fallback keyed <provider>/<group>. The provider is inferred through one shared helper, inferred_provider, which the pattern router already used inline, so the fallback lookup and routing agree on the prefix. The lookup only infers a provider when some fallback key ends in /<group>, so alias-style groups never hit the resolver
* fix(router): resolve context window and content policy fallback keys through the shared lookup
---------
Co-authored-by: Jason Dougherty <jasondoc3@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(cost-map): add gemini priority and flex prices to nano-banana-pro-preview and video token price to 3.1 flash live
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): add video token price to bare gemini-3.1-flash-live-preview
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): reproduce encrypted_content_affinity 503 when origin has no boundary peer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): keep encrypted content affinity turn one out of the response cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): degrade encrypted_content_affinity when the origin has no encryption-boundary peer
A routed-group candidate that is currently unavailable and shares its
(api_base, api_key) with no healthy deployment used to raise a proxy-level
503/429 from _unavailable_origin_error, even though healthy siblings in the
same model group could still serve the turn. Strip the encrypted reasoning
and dispatch to the healthy pool instead, matching the existing cross-group
behavior, and log a warning naming the origin model_id and routed group
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): bound the degraded-affinity log marker and assert the strip on the wire
Address review findings: restore num_retries alongside
optional_pre_call_checks in the integration test teardown, record scenario
request bodies on the scripted upstream so the tests can assert no encrypted
reasoning reaches the sibling, and truncate the client-supplied model_id in
the degraded-dispatch warning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tests): only record JSON bodies on the scripted upstream
Multipart uploads to scripted POST routes have no JSON body, so gate the
observation recording on the request content-type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): make encrypted_content_affinity runtime-toggleable so /config/update can turn it off
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: run the claude_code harness unit-test trees in the lint job
* ci: gate the harness step on Python files plus its installer and workflow
* ci: fire the harness step on dependency manifest changes too
* test: cover the harness gate's installer, manifest, and workflow triggers
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(spend): capture-rate check of LiteLLM spend against the OpenAI bill
* fix(spend): claim the alert lock after the check, NaN gauge on no rate, 180-day range cap, live settings, OpenAI adapter under llms
* fix(spend): chart the capture-rate gauge in the all-metrics dashboard and clear it when the check is removed
* fix(prometheus): record the capture-rate gauge when api_provider is an excluded label
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): cap an agent key's tools at what the invoking user and team may call
The invoking user's and team's x-litellm-user-id / x-litellm-team-id, echoed back by the
agent, already narrowed which MCP servers the agent key could reach, but not which tools on
those servers. An agent granted every tool on a server kept them all when acting for a user
who may only call a subset. The caller's team and user tool grants now intersect the agent's
tool list on each server, mirroring the servers axis, so the headers only ever narrow.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(mcp): pick the caller principal explicitly instead of getattr in the tool grant stub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): return immutable tool sequences from the agent caller tool ceiling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(e2e): carve harness tests out of the no-unit-tests hard rule
The Hard Rules bullet in tests/e2e/AGENTS.md banned unit tests of any
kind, the harness's own included, while the same file's claude_code/
and load/ entries, CONTRIBUTING.md, and the harness conftest all
describe markerless harness tests that run without a proxy. Reword the
rule to keep the product-feature and no-mocks bans, name the harness
trees as the one exception with the standard they are judged by, and
put the harness sentence back in the marker paragraph so the two docs
agree
* docs(e2e): name env vars set through monkeypatch as inputs, not patches
The rule banned monkeypatching anywhere under tests/e2e while its harness
carve-out named root-level test_*.py files that set env vars through
pytest's monkeypatch fixture. Say the ban is about patching code and that
an env var set that way is an input, so the examples and the rule agree
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ui): explain unbackfilled key lifetime spend and ship a backfill script
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): cover linked budgets, deleted keys and double-hashed logs in total_spend backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): count duplicate archived tokens once in total_spend backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): only rebuild resetting archived rows in total_spend backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): split spend log rebuild into opt-in backfill_key_total_spend_from_spend_logs.sql
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(db_scripts): scope backfill verify query to non-resetting keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(db_scripts): lift every key to at least current spend in backfill_key_total_spend.sql
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(db_scripts): align spend log backfill header with lifted floor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): point resetting keys at the spend log backfill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): drop script name from lifetime spend tooltip
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): translate /v1/responses batch rows through the Responses-to-Chat bridge
Vertex batch uploads treated every non-embeddings JSONL row as a chat
completions body, so a /v1/responses row lost its input and reached GCS
as a blank text part. Route detection now recognizes /v1/responses rows
and bridges them to chat through the same Responses-to-Chat bridge the
real-time path uses. That bridge call moves out of the Bedrock files
transformation into a shared helper both providers call, forwarding the
record's fields as sent, like real time, instead of validating them
against the SDK TypedDicts whose required keys clients omit.
* chore(batches): type the Vertex responses test helper and drop the quoted input cast
* fix(batches): translate developer messages to system on Vertex and Bedrock batch rows like real time
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests
* ci: move provider-independent MCP tests into tests/unit and run mcp-integration from litellm-tests
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* build: point the local proxy unit targets at the nested tests/unit/proxy tree
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* build: point the local proxy unit targets at the nested tests/unit/proxy tree
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(proxy): let callbacks filter the model listing routes per caller
* fix(proxy): offer every listed name to the listing callback, agent groups and deployment lookups included
* fix(proxy): hide aliases of a team model by its public name and offer /model/info lookups the listed name
* fix(proxy): map a malformed model listing filter return to the proxy error contract and document legacy team names
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* docs(github): require UI before/after screenshots and intentional UX change note in PR template
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(github): move intentional change note into TLDR rules and dedupe screenshots
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(github): refresh user flow screenshots with new commits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
npm keeps an unpublished version's timestamp in the packument's time map but drops it from versions, so the resolver could hand npm install a version it refuses with ETARGET. Only versions still present in versions are candidates now.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy_cli): add --validate_config dry-run flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(tests): format --validate_config CliRunner calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy_cli): run --validate_config before the ollama auto-start
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy_cli): restore file and add ollama validate_config regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* feat(terraform): add display_name to litellm_model resource and model data sources
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): persist display_name on update and read /model/info data envelope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(terraform): drop PATCH /model/{model_id}/update from endpoint audit allowlist
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): surface external display_name removal as drift on refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(terraform): rerun after uv download timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* ci: fail the unit shard when circleci tests split errors
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
#42443 pins a floating LiteAdmin button to the bottom-right corner, where it covers the logs page's next-page control. Flip the per-user Hide LiteAdmin switch during global setup so every spec reusing the admin storage state loads with the button hidden.
* feat(compat-matrix): resolve and install the Claude Code CLI per run
* chore(compat-matrix): drop the installer header comment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(playground): only auto-scroll the chat while pinned to the bottom
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(playground): keep scroll pin through programmatic scrolls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(playground): stop forcing the chat to scroll to the bottom on every update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(playground): drop scroll pinning integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(playground): scroll only the chat pane, not the page, while streaming
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(playground): format ChatUI with prettier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(playground): stop following streamed tokens, add jump to bottom button
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(playground): jump to bottom lands on the last message, not the spacer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): keep batch output_file_id null until Vertex reports outputInfo
Vertex only sets outputInfo.gcsOutputDirectory once a batch job has written
output. Falling back to outputConfig's outputUriPrefix named the per-model
directory shared by every batch of the deployment, an object that never
exists, so the proxy minted a managed file for it under the first key and
every other key's file calls on that id were 403s
* fix(vertex_ai): treat a null gcsOutputDirectory as no output file yet
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Prisma types a raw array parameter from the first batch a connection sees. After a flush in
which every member cost was a whole number (a free model), the connection's cached statement
expected int8[] and every later fractional batch on it failed with "improper binary format in
array element", so member spend silently stopped landing while team spend kept rising.
The rows now travel as one JSON document unpacked by jsonb_to_recordset with the column types
declared in SQL, so Postgres types the numbers and the batch shape no longer matters.
* feat(proxy): add uncapped server-side team usage export route
GET /team/daily/activity/export answers the same scoping as
/team/daily/activity/aggregated with one unbounded rollup query, so keys
past USAGE_TOP_API_KEYS_LIMIT are included. Supports daily,
daily_with_keys, daily_with_users and daily_with_models export types as
CSV (default) or JSON. The PTU flat-cost sentinel stays in the plain
daily rollup and is excluded from the keyed and per-model exports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): export team usage server-side when the key list was truncated
When the aggregated spend response reports api_key truncation, EntityUsage
passes a serverExport into the export modal that downloads CSV or JSON
from GET /team/daily/activity/export instead of building the file from
the truncated on-screen data. apiClient gains a responseType option so
the download can arrive as a Blob, and truncation no longer blocks the
export button
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover team usage export types, sentinel handling and scope
Unit tests pin the uncapped key rollup past USAGE_TOP_API_KEYS_LIMIT,
PTU sentinel inclusion in the daily rollup and exclusion elsewhere, the
per-user fold, and the CSV column layout. Integration tests exercise the
route against a live proxy, including member scope denial
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): tidy team usage export route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): use membership test for export type branch (PLR1714)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format exportBlockedReason test with prettier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): satisfy type-discipline gate in team usage export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pass export rows as a sequence to the response model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy-behavior): cover team usage export in the daily activity scope matrix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): carry PTU flat cost and escape formulas in team usage export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep the truncation export block on surfaces without a server export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): drop redundant comments in team export call and modal test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit cells for team usage export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tighten team usage export audit cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type the export params tuple and fold user keys in one pass
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): bring entity usage export helpers under eslint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): prettier-format UsagePageView after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): move tests.rs files inline or under tests/
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): move tests.rs files inline or under tests/
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(rust): inline path-included test files into their owning src files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(rust): drop stray proptest regression file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(rust): cover lowercase, empty and non-authorization headers in bearer detection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): route unmapped openai family model ids to converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): rename the e2e openai family backend constant
global.openai.gpt-6-sol has a cost-map row now, so the constant no longer
names an unmapped model
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(logging): pass provider response headers to callbacks on every endpoint
Custom callbacks only received kwargs["response_headers"] for chat
completions. Responses, image generation and edit, speech, and
transcription calls either never recorded the provider's headers or
recorded them in one place and not the other.
Every handler now records the provider's httpx headers on the response's
hidden params as "headers" (raw) and "additional_headers" (processed,
with LiteLLM's own entries winning on a clash), and the logging object
derives model_call_details["response_headers"] from those hidden params
before cost calculation on the non-stream and both streaming success
paths, keeping a handler-set value authoritative. Binary speech responses
expose their hidden params to the standard logging payload, and the sync
OpenAI transcription request always fetches the raw response.
* test(images): point the legacy image and speech fakes at the raw response surface
Image generation now goes through the SDK's raw response so the provider headers can be read, and the speech binary response now carries hidden params. The unit fakes in the image generation, xinference, proxy provider, image edit, Vertex speech, and otel suites still pinned the old call surface and the old "no hidden params" assertion, so they read an uncalled mock or a fake response without headers.
* test(images): drop the rewritten mock comments and the generated edit PNGs
* test(images): move the llm-span test's image fake to the raw response surface
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(vertex): native batch JSONL passthrough with cost tracking
Add a per-request `passthrough=true` multipart field on `POST /v1/files`
(and the same kwarg on `litellm.create_file`) that uploads a native
Vertex AI batch JSONL to the deployment's GCS bucket unchanged, so rows
using `googleSearch` and other Gemini-only features run as written and
the output, `groundingMetadata` included, comes back untouched.
Passthrough is sticky through the GCS object path
(`litellm-vertex-files/passthrough/...`), so batch create and output
retrieval inherit it without new state. Native output rows are costed
from their `usageMetadata` with the deployment's model and model_info,
in the polling and retrieve paths and for the existing global
`disable_vertex_batch_output_transformation` flag, which billed $0
before.
The proxy requires the target to resolve to vertex_ai deployments only,
refuses `passthrough` with a non-batch purpose, a non-default
`target_storage`, or pre-call guardrails, and validates native rows on
`request` instead of the OpenAI batch keys.
* refactor(vertex): keep native batch row pricing inside the Vertex adapter
Moves native Vertex batch row detection, response parsing, and per-row
pricing from litellm/batches/batch_utils.py into
litellm/llms/vertex_ai/batches/transformation.py, so batch_utils only
aggregates the rows it gets back. Adds tests/test_litellm/files to the
misc unit shard so the new test directory is claimed by a shard.
* fix(files): say what a passthrough batch upload takes when a row is not native
The missing-key 400 listed bare key names, so an OpenAI-shaped row under
passthrough=true read "Each line must be a JSON object with keys request".
The batch line shape now carries its own hint, and the passthrough one says
a passthrough upload takes native Vertex batch rows with a request key
* fix(batches): bill native Vertex embedding batch rows on the native cost path
A native Vertex output row whose response holds an embedding was validated as a
generateContent response, so the documented tokenCount-only shape counted as a failed
row. Price embedding rows from their own usage (promptTokenCount, else tokenCount) with
the helper the transformed embeddings path already used, and drop the prompt-details
helper nothing calls anymore.
* fix(batches): keep modality batch rates on native Vertex embedding rows
An embedding row that carries usageMetadata was billed from promptTokenCount alone, so
its promptTokensDetails no longer reached the audio, image, and video batch rates the
way it did before the native cost path. Run every row with usageMetadata through the
Gemini usage parser and keep the flat tokenCount fallback for embedding rows without it.
* fix(batches): price native Vertex batch rows by modelVersion under a wildcard deployment
A `vertex_ai/*` deployment hands the batch cost path `*` as the deployment model, which
no cost map resolves, so every native (passthrough or flag-on) row was billed at $0. A
wildcard deployment model now defers to the row's own `modelVersion`, the way the
transformed path already prices by the row's `model`.
Also moves the native passthrough tests under tests/test_litellm, the tree codecov
reads, and covers the raw upload chunking, the embedding output translation, the
unpriceable-row path, and the flag-on dispatch.
* fix(batches): keep explicit deployment prices for native Vertex rows without a modelVersion
Under a wildcard deployment a native batch row that carries no modelVersion (an embedding
row, or a generateContent row Vertex returned without one) was billed at $0 even when the
deployment's model_info sets explicit batch prices, because the cost calculator was never
called. The row now falls back to the wildcard name, which the cost calculator prices from
the explicit model_info, and only a row with neither a modelVersion nor a deployment model
is billed at $0 with the warning
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(prometheus): add model_group label to deployment request and rate limit metrics
litellm_deployment_total_requests, litellm_deployment_success_responses,
litellm_deployment_failure_responses, litellm_deployment_tpm_limit and
litellm_deployment_rpm_limit had no way to identify which model_group a
pooled deployment belongs to, only requested_model, litellm_model_name and
model_id, none of which name the alias a model_name resolves through when
it fans out to more than one deployment.
model_group was already resolved onto enum_values for every request in
async_log_success_event, so this is a label-list addition for the metrics
built directly from that enum_values (the two request counters). The
failure counter builds its own UserAPIKeyLabelValues locally and had a
model_group variable already in scope that it never passed through, and
the tpm/rpm limit gauges are set from a helper that took no model_group
parameter at all even though its only caller already had it on
enum_values. Both now thread the value through.
* test(prometheus): expect model_group in deployment success/total request labels
test_set_llm_deployment_success_metrics asserts the exact label set passed
to litellm_deployment_success_responses.labels() and
litellm_deployment_total_requests.labels(), which now includes model_group
since it was added to those metrics' label list.
* fix(prometheus): bound model_group on deployment failure metrics
On a pre-routing reject (no deployment selected), model_group is
caller-supplied via litellm_params.metadata and was passed through
unbounded, letting an unrecognized value mint unlimited label series on
litellm_deployment_failure_responses / litellm_deployment_total_requests.
Bound it with the same _bounded_requested_model_label used for
requested_model on this path. When a deployment is actually selected,
model_group is router-resolved and passed through as-is.
Also documents the model_group parameter on
_set_deployment_tpm_rpm_limit_metrics and the bounding behavior on
set_llm_deployment_failure_metrics.
---------
Co-authored-by: ahamedshaik16 <24526479+ahamedshaik16@users.noreply.github.com>
* test(rust): encode anthropic response serialization shape as rstest cases
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(rust): shape Anthropic Messages requests natively
The Rust Messages route only relayed the body. It now runs the request shaping
the Python handler does for the direct Anthropic provider: history sanitizers
(empty blocks, tool ids, replayed web search results, provider_specific_fields,
encrypted reasoning, advisor blocks), reasoning_effort and adaptive/legacy
thinking translation against the model's capability flags, the sampling and
speed gates under drop_params, the metadata allowlist, additional_drop_params,
reasoning auto summary, OAuth and ANTHROPIC_AUTH_TOKEN credentials,
provider_specific_header merging and anthropic-beta injection. Capability flags
and LiteLLM settings reach Rust through route_host.shaping(). A request the
route rejects before the call now maps to BadRequestError instead of
APIConnectionError
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(rust): port Anthropic Messages shaping tests and pin comment contracts as cases
Every Python unit test that exercises the ported shaping for the direct
Anthropic provider now has a named rstest counterpart, and every comment that
stated a behavior contract is deleted in favor of a case that pins it. Measured
with cargo-mutants over the touched files, all viable mutants are caught
Porting the tests surfaced parity gaps, fixed here to match Python: every
casing of a forwarded anthropic-beta header is merged, replayed web search
results are rewritten from their own block (an empty result keeps its slot and
a server_tool_use with a non-string query stays), an empty output_config.effort
falls back to medium, speed and reasoning effort errors quote values the way
Python does, additional_drop_params apply after metadata validation and the
auto summary and never touch model or messages, and a non-string
metadata.user_id is rejected before the call
* fix(rust): resolve Messages credentials through the secret source and scope headers by resolved provider
The native Messages route read ANTHROPIC_API_KEY, ANTHROPIC_AUTH_TOKEN and the
base URL straight from the process environment, so a key or base held in a
configured secret manager was never found. Each provider config now declares
its secret names and the route resolves them through the same SecretSource the
OCR route uses, with the Python bridge passing in litellm's configured manager
provider_specific_header entries were scoped by the explicit
custom_llm_provider only, falling back to anthropic, so an azure_ai/ model
lost its azure_ai scoped headers. Scoping now happens in the route after the
provider is resolved from the model, as Python's handler does
The Azure config now adds the same anthropic-beta feature headers Python's
Azure route adds, and the metadata allowlist, reasoning auto summary and
history sanitizers move from the core route into the llms crate, mirroring
their home in Python's messages handler
* test(rust): escape the dot in the metadata.user_id match pattern
* refactor(rust-bridge): project Messages capabilities without mutable dicts
The capability flags and effort tiers were built as dict comprehensions,
which the type-discipline gate counts as mutable construction, and the
asdict call carried a mutable-ok suppression that suppressed nothing. The
flags are now passed one by one and the effort tiers are a frozen dataclass,
which asdict projects to the same map the native side reads
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(guardrails): run the cache-hit redis outage test on the shared owned_redis helper
test_redis_outage_keeps_serving_in_memory_hits (#42780) spawned `redis-server`
straight from PATH. The CircleCI integration machine has no redis-server
binary, so the test died with FileNotFoundError before reaching the proxy.
Use integration._support.redis_process.owned_redis, which runs the local
binary when there is one and otherwise the job's redis-cache container, and
take the outage with its stop()/start() pair the way test_redis_recovery
already does. The assertions are unchanged.
* test(guardrails): describe the owned_redis stop as an outage, not a kill
