Commit graph

47455 commits

Author SHA1 Message Date
yuneng-jiang
6dfcc46c9e
Merge pull request #40026 from BerriAI/litellm_fix_migrated_pages_sidebar_test
test(ui): make navigation smoke resilient to router refactors
2026-09-07 11:17:32 -07:00
yucheng-berri
2700ffe8c9
fix: capture provider request id in failure logging payloads (#40045)
* fix: capture provider request id in failure logging payloads

* fix: include common provider request id headers

* test: remove accidental formatter churn

* test: type provider request id cases

* test: cover azure and google request id headers

* refactor: share provider request id headers from constants

* chore: remove redundant request header comment

* test: cover provider header lookup failures
2026-09-07 11:12:34 -07:00
Mateo Wang
9275cf42ed
Merge pull request #40015 from BerriAI/litellm_fix_check_run_name_collisions
fix(ci): stop the auto-close duplicates job colliding with the required test check
2026-09-07 10:55:34 -07:00
Mateo Wang
8e7f748e8a
Merge pull request #39870 from BerriAI/litellm_lit_6917_make_check_test_tree_ruff
fix(make check): lint the test tree on tests-only changes like CI does
2026-09-07 10:55:28 -07:00
Mateo Wang
c8ef043087
Merge pull request #39981 from BerriAI/litellm_lit_7079_bridge_preserve_provider_metadata
fix: keep provider id and metadata on Responses API bridged chat completions
2026-09-07 10:55:25 -07:00
Mateo Wang
3dac0ba79b
Merge pull request #39850 from BerriAI/litellm_fix_realtime_reasoning_double_bill
fix(cost): bill realtime reasoning tokens nested in text_tokens once
2026-09-07 10:55:20 -07:00
Mateo Wang
642a0f68ae
Merge pull request #39530 from BerriAI/litellm_fix_gateway_rustls_provider
fix(ai-gateway): dial upstream WebSockets over an explicit rustls provider
2026-09-07 10:55:15 -07:00
Mateo Wang
9700f666d0
Merge pull request #39839 from BerriAI/litellm_async_remote_image_fetch
fix(async): move remote image fetches off the event loop for Snowflake, Bedrock invoke Claude, Mantle and Gemini
2026-09-07 10:55:04 -07:00
Mateo Wang
a9556c7bad
Merge pull request #39965 from BerriAI/litellm_fix_oci_cohere_stream_tool_turn_dup
fix(oci): stream Cohere tool-calling answers once
2026-09-07 10:54:59 -07:00
Mateo Wang
a676f5eb6d
Merge pull request #39800 from BerriAI/litellm_claude_md_qa_screenshots
docs(claude): have runs embed their own QA screenshots on visual changes
2026-09-07 10:54:49 -07:00
yuneng-jiang
5930549bf8
Merge pull request #40140 from BerriAI/litellm_fix_autorouter_prisma_filter
fix(router): serialize heuristic tuning quota filters for Prisma
2026-09-07 10:52:38 -07:00
yujonglee
8da735410c
test(ocr): trace callback lifecycle parity (#40063) 2026-09-07 10:44:22 -07:00
yujonglee
217cb12623
refactor(rust): remove per-request enablement arguments (#39928)
* refactor(rust): remove per-request enablement arguments

* fix(rust): remove ignored transcription enablement

* refactor(rust): remove OCR-specific bridge controls
2026-09-07 10:43:45 -07:00
Yuneng Jiang
f40c3842dc
fix(router): serialize heuristic tuning quota filters for Prisma 2026-09-07 10:08:19 -07:00
yuneng-jiang
a2b7868a5b
test(ui): pin wire contracts for key, model and MCP server forms (#40019)
* test(ui): pin wire contracts for key, model and MCP server forms

Add vitest cases that pin what the key edit, key create, model edit and
MCP server edit forms put on the wire: an edited field reaches the
request with its new value, a cleared field reaches it as an explicit
null, and the dirty-only body is pinned as an expected failure until
each form moves to pickDirty. Model edit also pins the cost-map-derived
model_info fields as an expected failure.

KeyEditView hands a cleared max_budget to KeyInfoView as an empty
string and handleKeyUpdate maps it to null, so the null is pinned at
the /key/update boundary in key_info_view.test.tsx and the KeyEditView
case is an expected failure. buildEditServerPayload passes a cleared
description through as an empty string, so that case is an expected
failure too.

* test(ui): split masked model_info pins and retarget the create tracker

The model_info expected-failure case held three assertions, and it.fails
stops at the first one, so a later revamp that fixed max_input_tokens
while leaving mode leaking would still report an expected failure. Split
it into one case per pinned field group so each flips on its own.

The key create tracker asserted a body of only key_alias, which a create
can never send: key_type, user_id, duration and metadata are always
mounted. Retarget it at the real over-send, which is the Optional
Settings section adding fifteen undefined-valued keys when the user opens
it without filling anything in.
2026-09-07 09:53:18 -07:00
mubashir1osmani
a1e7293fa9
fix(mcp): apply key and team guardrails to MCP tool calls (#39629)
* fix(mcp): apply key and team guardrails to MCP tool calls

Guardrails attached to a virtual key or team were only enforced on LLM
routes. The synthetic request built for MCP tool call guardrail hooks
carried no guardrails in its metadata, so a guardrail with default_on
false never ran on tools/call even when the key explicitly listed it.
Resolve key, team, and project guardrails onto the synthetic request
with the same helper the chat path uses.

* fix(mcp): pass project metadata through without a mutable default

* fix(mcp): mark the request dict parameter mutable-ok with a reason

* test(mcp): explain the premium_user patch and tighten the helper docstring
2026-09-07 16:45:16 +00:00
Mateo Wang
1c7b13bdbf
Merge pull request #38755 from BerriAI/litellm_mistral_voxtral_tts_speech
feat(mistral): add text-to-speech support for /v1/audio/speech
2026-09-07 09:38:57 -07:00
Mateo Wang
50f54b9c7e
Merge pull request #40014 from BerriAI/litellm_lit_7036_retry_policy_400s
fix(router): skip the refusing deployment when retrying a non-transient error
2026-09-07 09:38:43 -07:00
yujonglee
728d0953af
ci: simplify Rust checks and remove wheel PR comments (#39975)
* ci: limit Rust workflows to Rust directory changes

* ci: run Rust checks when their workflow changes

* ci: report Rust wheels only for successful Rust changes

* ci: keep Rust wheel reports in the workflow summary

* ci: group Rust lint and validation jobs

* ci: keep Rust job names distinct from required lint and test checks

* ci: drop the unused Python setup from the Rust lint job

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-07 09:00:58 -07:00
devin-ai-integration[bot]
168a0055a2
chore(lint): stop ratcheting *-budget.json on PR branches (#39937)
Some checks failed
Unit Tests / Vertex AI (push) Has been cancelled
Unit Tests / proxy-auth (push) Has been cancelled
Unit Tests / proxy-endpoints (push) Has been cancelled
Unit Tests / proxy-extras (push) Has been cancelled
Unit Tests / proxy-infra (push) Has been cancelled
Unit Tests / proxy-server (push) Has been cancelled
Unit Tests / responses-caching-types (push) Has been cancelled
Postgres Tests / proxy-security (push) Has been cancelled
Postgres Tests / schema-migration (push) Has been cancelled
Postgres Tests / proxy-behavior (push) Has been cancelled
Unit Tests: Documentation Validation / documentation (push) Has been cancelled
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Has been cancelled
Unit Tests / caching-local (push) Has been cancelled
Unit Tests / core-utils (push) Has been cancelled
Unit Tests / enterprise-package (push) Has been cancelled
Unit Tests / enterprise-routing (push) Has been cancelled
Unit Tests / integrations (push) Has been cancelled
Unit Tests / All Other Providers (push) Has been cancelled
Unit Tests: Proxy DB Operations / auth-checks (push) Has been cancelled
Unit Tests: Proxy DB Operations / budgets (push) Has been cancelled
Unit Tests: Proxy DB Operations / custom-logging (push) Has been cancelled
Unit Tests: Proxy DB Operations / db-and-spend (push) Has been cancelled
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Has been cancelled
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Has been cancelled
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Has been cancelled
Unit Tests: Proxy DB Operations / key-generation (push) Has been cancelled
Unit Tests: Proxy DB Operations / logging-misc (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-runtime (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-server-core (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-utils (push) Has been cancelled
2026-09-06 12:44:19 -07:00
mateo-berri
5825cc7593 chore: merge litellm_internal_staging into litellm_mistral_voxtral_tts_speech
Some checks failed
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
LiteLLM Rust / release wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
2026-09-06 03:24:55 -07:00
mateo-berri
4edf6f4dc5 fix(ci): keep an expression matrix directive out of the comparison
`include:` or `exclude:` written as `${{ ... }}` read back as a string, and
the sweep treated that as the directive being absent, so it expanded every
combination GitHub would have dropped. A job whose `name:` holds no matrix
value then looked like it repeated one name across combinations that never
run. An absent directive still means no rows; anything that is not a list
of rows now joins the names left out of the comparison
2026-09-06 02:59:16 -07:00
mateo-berri
5a27e11263 fix(ci): compare names one workflow run settles the same way
A `name:` whose only leftover expressions read a `github.` property other
than `github.job` is filled in identically for every job of the run that
publishes it, so two jobs of one workflow carrying it land on the same
check run. Those names now compare against the other jobs of their own
file instead of sitting in the blind-spot bucket. They stay out of the
comparison across files, where two workflows can run on different events
2026-09-06 02:32:50 -07:00
mateo-berri
aa2c41f489 fix(ci): stop the name-collision check failing legal workflows
An expression at `jobs.<id>.strategy` is legal on GitHub, but the model
required a mapping there, so a workflow using one made the whole file
unreadable and turned code-quality red. That job's names are now a blind
spot like any other name the sweep cannot work out offline.

A matrix whose `name:` holds no matrix value publishes that one name once
per combination, which leaves a required context just as ambiguous as two
jobs sharing a name, so it now reports instead of deduping.

A file that does not parse as one YAML document is reported the way the
module already promised, rather than escaping as a traceback.
2026-09-06 01:59:03 -07:00
Mateo Wang
02522a5441
Merge pull request #39983 from BerriAI/litellm_lit_7081_azure_ai_gpt_6_astra_pricing
feat(cost-map): add azure_ai/gpt-6-astra Foundry pricing
2026-09-06 01:27:22 -07:00
Mateo Wang
b09b7d3eb8
Merge pull request #39980 from BerriAI/litellm_lit_7048_batch_cost_row_once
fix(batches): account a batch's cost once, from the first retrieve that sees it final
2026-09-06 01:27:10 -07:00
mateo-berri
904542a559 fix(ci): stop guessing at names built from contexts the sweep cannot read
Three ways the sweep could fail a workflow GitHub would publish fine.

`github.workflow` and `github.job` were counted as fixed for the whole run, so
two jobs naming themselves after the workflow they sit in were reported as a
collision. `runner` and `vars` were wrong the same way. Drop the exception
entirely: a name still holding an expression is one GitHub resolves per job, so
it is nothing to compare, which is what the rest of the module already does.

`format()` was resolved with Python's semantics, so an attribute lookup crashed
the script and a width specifier padded a name GitHub never pads. Fill `{0}`
holes and escaped braces, and treat anything richer as unresolved.

A matrix `include` or `exclude` row holding a value that is not a scalar lost
that key and became an empty row, which excludes every combination. Report the
row instead of quietly reshaping the matrix around it.
2026-09-06 01:27:03 -07:00
mateo-berri
5ca9e26050 fix(ci): leave check-run names the sweep cannot resolve out of the comparison
A job name holding an expression the sweep could not resolve was compared as
if it were the published name. Two jobs whose names differ per matrix value or
per caller input were reported as a collision, and a matrix that was itself an
expression collapsed onto the bare job id and did the same.

Model what a job publishes as known names beside the reasons the rest stay
unknown. Anything the sweep cannot work out contributes no name and is
reported as a note instead of guessed at. An expression over contexts that are
fixed for the whole run still compares, so two jobs sharing one of those are
still caught.
2026-09-06 01:09:36 -07:00
mateo-berri
a72041b757 test(router): pin the retry skip list across attempts in a model group 2026-09-06 01:03:32 -07:00
Yuneng Jiang
b1cc1e8dae
test(ui): assert navigation through accessible links and page content 2026-09-06 00:59:18 -07:00
mateo-berri
d664ca139e chore(router): suppress the retry-skip kwargs writes and correct the filter docstring
The two writes that hand the skip list to the next attempt now carry a
`# rebind-ok` reason, which is the sanctioned escape hatch for an unavoidable
parameter mutation and matches how `log_retry` already writes into the same
kwargs dict a few lines above

`get_excluded_filtered_deployments`'s docstring said returning the unfiltered
list would re-include the deployment that just failed. The retry skip does
exactly that on purpose, so the docstring now says each caller decides what an
empty result means

The reliability registry cell the new e2e test claims is marked
`fail_before_fix: proven`: the same config returns 400 at the merge base and
200 off a sibling deployment at the tip
2026-09-06 00:53:24 -07:00
Yuneng Jiang
a08fb489cd
test(ui): accept trailing slashes in migration sidebar links 2026-09-06 00:38:07 -07:00
mateo-berri
0fcf0fe06c test(router): cover the retry skip-list narrowing helper
The router code coverage gate reads every function defined in router.py
and fails when no test file names it. _as_retry_skipped_deployment_ids
was only reached indirectly through the retry path, so the gate went red
on this PR's tip.

Test it directly instead: a tuple of strings survives, non-string items
inside the tuple are dropped, and every other shape a caller could send
narrows to an empty skip list.
2026-09-06 00:06:35 -07:00
mateo-berri
5bbc83e3de fix: model the check-run names GitHub really publishes
The collision sweep read a job's name as its `name:` or bare job id, which is
wrong for a matrix job that sets no name: GitHub publishes `build (3.12)`, one
per combination. That missed real duplicates and invented ones that don't exist.
It also crossed every matrix value while ignoring `exclude`, so it checked
combinations no job ever runs.

Four smaller gaps went with it. Boolean matrix values reached a name as `True`
rather than `true`. A `format()` whose arguments cannot fill its placeholders
raised straight out of the script instead of leaving the name unresolved. A job
calling a reusable workflow only ever chained one level, and a call outside the
repo fell back to the caller's own name, which GitHub never posts. A job whose
`name:` was not a string failed validation and silently dropped every job in
that file, so the sweep now renders any scalar and reports a file it cannot read
instead of skipping it.
2026-09-06 00:04:31 -07:00
mateo-berri
defd8661f4 refactor(spend): stop queueing a batch's claim row for a writer the proxy never builds
SPEND_LOGS_URL only diverts spend logs when db_writer_client is set, and nothing in the proxy ever assigns that global, so the queued copy was only ever skipped as a duplicate by the local insert.
2026-09-06 00:04:24 -07:00
Mateo Wang
4104868458
Merge pull request #39723 from Atharva-Kanherkar/fix/anthropic-responses-refusal-translation
Some checks failed
Unit Tests / misc (push) Waiting to run
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests / caching-local (push) Waiting to run
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / integrations (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
LiteLLM Rust / release wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
fix(anthropic_responses): preserve Responses refusal blocks in Anthropic messages translation
2026-09-06 00:03:06 -07:00
yujonglee
b9f5cd6036
ci: run unit tests on Python 3.12 (#39989) 2026-09-05 23:56:09 -07:00
mateo-berri
6866eac96f fix(router): ignore a retry skip list the caller sent itself
The retry skip travels as a request kwarg, and the router forwards keys it
does not recognize, so a client can put _retry_skipped_deployment_ids in its
own request body. The value went straight into a pydantic TypeAdapter and
then into a set(), so an int or an object raised TypeError and a string, a
list, or a dict raised a ValidationError, each of them replacing the 400 the
provider had actually returned.

Every read now goes through one narrowing function that keeps a tuple of
strings and skips nothing otherwise, so a forged value costs the caller
nothing beyond the retry landing on the same deployment again.
2026-09-05 23:53:36 -07:00
mateo-berri
fcb6d2267c fix(spend): keep a batch's claim row out of the logs a proxy was told not to write
disable_spend_logs has to keep meaning that no request gets logged, and the row
that makes a batch chargeable exactly once is the one row it cannot drop, so with
logging off that row now carries only what tells the retrieves apart. SPEND_LOGS_URL
deployments get their copy back too: the claim writes straight to this table, so the
row is queued as well when an external writer is the one that takes the spend logs.
2026-09-05 23:51:44 -07:00
mateo-berri
05cba21763 fix(anthropic): split refusal off a combined finish_reason chunk
A fake-streamed provider hands the adapter one chunk carrying both the
delta payload and the finish_reason, which is exactly what the combined
chunk splitter exists for, but its content check never listed the refusal.
The translation short-circuits on finish_reason, so that refusal text was
dropped and the client got `stop_reason: refusal` over an empty content
array, the symptom this PR set out to fix.

Both refusal accumulators also drop their `mutable-ok` lists for a plain
string attribute
2026-09-05 23:40:20 -07:00
mateo-berri
ecf7e4e766 fix(router): keep the provider's error when the retry skip empties the group
Before excluding the deployment that just refused, the retry-skip guard asked
whether another one could still answer. It asked by re-running a single routing
filter, the order filter, while deployment selection also applies cooldowns, the
context-window pre-call check, tag routing, and routing plugins.

Any filter the guard did not replicate made it answer yes while the real pick was
left with nothing. A group narrowed to one deployment by tag routing turned the
provider's own 400 into a no-deployments 429.

The skip now runs where every filter has already been applied, and it keeps the
deployments untouched when skipping would leave none. The caller gets the
provider's error either way, and a group with one eligible deployment retries in
place as it did before.
2026-09-05 23:36:28 -07:00
mateo-berri
c09d34fc4b fix(anthropic): stream refusals parked in provider_specific_fields
The first-delta guard read `delta.refusal` directly, while the translation
three lines later goes through `openai_chat_refusal_text`, which also reads
the `provider_specific_fields` LiteLLM parks unrecognized fields in. A
provider that sends the refusal that way had its only refusal delta skipped
as blank, so the client got `stop_reason: refusal` over an empty content
array, which is the symptom this PR set out to fix
2026-09-05 23:20:19 -07:00
mateo-berri
fffe0bb0dc test(azure_ai): pin the tier the messages bridge sends when astra refuses max
The /v1/messages adapter lowers a tier the entry does not accept, so dropping max from the astra
rows moves that path from Foundry's 400 to a request at xhigh. Nothing pinned that, and the guard
test's docstring named gpt-6-astra as the only gpt-5 name with an azure_ai row, which 11 rows
contradict.
2026-09-05 23:18:25 -07:00
mateo-berri
3dea1ebb32 fix(cost-map): keep the prompt cache breakpoint flag on the foundry gpt-6-astra row
The openai gpt-6-astra card carries supports_prompt_cache_breakpoint, so a Foundry deployment
reported it as true until the azure_ai row took over the lookup. The cache control hook still
honours breakpoints for that deployment through the bare name, so /model/info was the only thing
that changed, and it now agrees with the hook again.
2026-09-05 23:18:25 -07:00
mateo-berri
9f379b36b9 refactor: return the collision check's failure instead of raising it
The checker raised a custom exception and caught it two lines down in the
same module, which is the throw-then-catch the repo's coding guide rules
out. `main` now prints the same message and returns the exit code, so the
collision list stays a value the whole way out
2026-09-05 23:18:18 -07:00
Mateo Wang
0318b4acdc
Merge pull request #30856 from emerzon/litellm_vertex_lyria_models
feat(vertex): add Lyria model support
2026-09-05 23:12:25 -07:00
mateo-berri
a2b21b323a Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_7048_batch_cost_row_once 2026-09-05 23:03:41 -07:00
mateo-berri
cb1ec76e46 fix(router): keep retry skips on the active order-fallback target
The retry-skip guard checks that some other deployment could still answer
before it excludes the one that just refused, so a single-deployment group
keeps the old retry-in-place behavior. It asked that question at the group's
minimum order, but the router picks the retry's deployment at the order the
request has already escalated to.

So a group with a primary at order 1 and a backup at order 2 answered "yes,
order 1 still has a candidate" while the retry was pinned to order 2, and the
exclusion left order 2 with nothing. The caller got a no-deployments error in
place of the provider's own 400.

The helper now takes the active target order and filters by it, which is the
same value async_get_healthy_deployments reads off the request.
2026-09-05 23:02:44 -07:00
mateo-berri
d6a727fe0f fix: keep matrix include rows whole when expanding job names
The guard read each matrix key's values independently and crossed them, so
a job name reading two keys off one include row published pairs no job ever
runs, which could fail a valid workflow on a required check

It now builds the combinations GitHub builds: the listed keys crossed, each
include row folded into the combinations it overwrites nothing in, and a row
that fits nowhere standing on its own
2026-09-05 23:02:32 -07:00
mateo-berri
11e45ad953 fix(vertex_ai): mark the Lyria 3 catalog entries text-only
`vertex_ai/lyria-3-clip-preview` and `vertex_ai/lyria-3-pro-preview` were
registered with `supports_vision`, `supports_image_input`, and an `image`
modality, which contradicts their `gemini/lyria-3-*` siblings and makes
/model/info advertise image input on text-to-music models.
2026-09-05 23:00:01 -07:00