Commit graph

3614 commits

Author SHA1 Message Date
tin-berri
5a8c4f48e4
feat(router): native compact-to-fit across conversation APIs (#42074)
* feat(router): native compact-to-fit across conversation APIs

* fix(router): preserve compaction admission and shared client boundaries

* fix(router): honor compaction fit fallbacks and router-scoped access

* fix(router): charge compaction usage to caller token limits

* test(http): keep FastAPI inside proxy tests

* fix(router): check compactor capacity before skipping escalation
2026-09-21 22:52:29 -07:00
devin-ai-integration[bot]
7956dd6e8c
fix(anthropic_adapter): keep reasoning_effort a string for targets that stay on chat completions (#42401)
* fix(anthropic_adapter): keep reasoning_effort a string for targets that stay on chat completions

* fix(anthropic_adapter): judge the summary bridge with the deployment's api_base

The adapter's bridge check now resolves the provider and base the same
way completion() does, passing the deployment's api_base and api_key
into get_llm_provider and the resolved base into the bridge check, so a
Foundry deployment lands on the same route in both places and a bare
model name routed by its api_base still gets the plain tier.

A litellm_proxy target keeps the dict, since the upstream gateway makes
its own bridge decision and needs the summary to make it.

* refactor(anthropic_adapter): return the plain effort instead of writing it inside the helper

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-21 21:08:28 -07:00
devin-ai-integration[bot]
418561991b
fix(bedrock): price bedrock/mantle/<model> deployments from the base model row (#42402)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-21 20:59:36 -07:00
devin-ai-integration[bot]
5cf17f9ce8
fix(responses): drop client_metadata and merge system messages for Databricks chat-only models (#42390)
* fix(responses): drop client_metadata before bridging to chat completions

Codex CLI sends client_metadata on every /v1/responses call. For a
provider with no native Responses config the chat-completions bridge
forwarded the raw kwargs, so client_metadata reached the provider as a
chat body field and Databricks rejected the request with an unknown
field 400. The bridge now drops the Responses-only request fields
before calling completion while still passing every other kwarg
through, so deployment-level params such as chat_template_kwargs keep
reaching providers without a native config.

* fix(databricks): merge consecutive system messages for chat-template models

Codex sends instructions plus a leading developer item, which the Responses
bridge and the developer-to-system translation turn into two consecutive
system messages that Databricks chat-template models reject with "System
message must be at the beginning". Each run of consecutive system messages
is now merged into one before the request is built for non-Claude models.

Also keep client_metadata out of the bridged chat request even when
allowed_openai_params names it, so both bridge branches drop the same set.

* fix(databricks): skip empty system messages when merging consecutive ones

Databricks drops empty content before the merge, so a system message in a
run could carry no content key and the merge iterated None. Those messages
are now skipped; a run with no content at all keeps its first message.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-21 20:09:54 -07:00
ryan-crabbe-berri
e7f3f58f96
refactor(agentic-loop): build follow-up kwargs in one place so no executor can repeat a request param (#42307)
* refactor(agentic-loop): build follow-up kwargs in one place so no executor can repeat a request param

The Responses and both chat completions follow-up executors each rebuilt the follow-up kwargs by hand and then expanded them next to the request params, so a plan whose kwargs repeated a request param raised a duplicate keyword TypeError. They now share build_agentic_followup_kwargs, which drops any key already sent as a request param (and the explicitly passed model/input/messages) from both the request kwargs and the plan kwargs. Each executor keeps its own internal-key filter unchanged, and the /v1/messages executor is untouched because it merges into a single dict and cannot hit this.

* test(agentic-loop): move follow-up regressions into their mapped test files

Greptile review: the executor regressions belong in test_llm_http_handler.py and test_chat_completion_agentic_loop.py rather than a split-off file, and the builder test helper returned a read-only mapping while promising a dict. The Responses overlap test is dropped because #41560 already added the same one to the mapped file.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-21 20:06:01 -07:00
devin-ai-integration[bot]
3252852b0f
fix(auth): fail closed when the JWT single-team fallback or compact editor membership read hits a DB outage (#42344)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-21 20:04:30 -07:00
devin-ai-integration[bot]
0fd1c191ca
feat(fal_ai): add queue-only /fal_ai pass-through route with spend tracking (#42360) 2026-09-22 02:59:58 +00:00
devin-ai-integration[bot]
e7cd97c6b6
fix(proxy): release unclaimed budget reservations at request end (#42304)
* fix(proxy): release unclaimed budget reservations at request end

* fix(proxy): release unclaimed budget reservations of websocket sessions too

* test(proxy): drop the structural middleware inheritance check

* fix(proxy): claim the budget reservation on streaming pass-through before its cost callback

The SSE chunk processor hands its success handler to the logging worker
after the response, so the request-end release freed the reservation
first and left the key unguarded until the worker drained. Claim it at
both end-of-stream hand-offs, the immediate enqueue and the coroutine
parked for deferred dispatch.

Give the xai realtime test double the litellm_params attribute every
real Logging object carries, since the wrapper now reads it.

* test(pass-through): give the vertex streaming test doubles a litellm_params dict

The spec'd Logging mocks in test_vertex_ai_anthropic_streaming_cost_injection.py
lacked the instance attribute the chunk processor now reads to claim the budget
reservation. Also restores main's _lazy_openapi_snapshot.json: the branch's copy
had been regenerated under Python 3.14, which dedents one docstring description
that the CI regeneration on Python 3.12 keeps indented, and the PR adds no lazily
loaded route, so main's file is the correct one.

* fix(pass-through): claim the budget reservation only after its cost callback is enqueued

Every pass-through success hand-off stamped callback_bound before handing the
coroutine to the logging worker. When that enqueue raised, the reservation stayed
claimed with no callback left to reconcile it, so the request-end release skipped it
and the reserved cost stayed pinned on the key's counter. Enqueue first, then claim,
so a failed hand-off leaves the reservation for the request-end release.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-09-21 19:51:12 -07:00
devin-ai-integration[bot]
5a764205a5
fix(fal_ai): price non-canonical image sizes from the nearest row and honour dump options (#42336)
* fix(fal_ai): price non-canonical image sizes from the nearest row and honour dump options

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(fal_ai): drop monkeypatched mixed pricing case

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(fal_ai): use the default dimensions constant directly

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal_ai): forward nested include and exclude when dumping image data

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal_ai): honour pydantic item selectors in image data serializer

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal_ai): match negative item selectors in image data serializer

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:19:18 -07:00
devin-ai-integration[bot]
b833e1fc4c
feat(fal_ai): add flux-lora-depth image edits and moondream3 chat completions (#42334)
* feat(fal_ai): add flux-lora-depth image edits and moondream3 chat completions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(fal_ai): retrigger codecov processing

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal_ai): reject multi-turn and system messages for moondream3 chat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal_ai): return 400 for invalid moondream3 chat requests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal_ai): reject moondream3 responses missing output or usage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(fal_ai): reject streaming moondream3 requests before dispatch

stream never reaches optional_params, so the transform_request check could not fire; reject in _complete_fal_ai on ctx.stream

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:01:28 -07:00
devin-ai-integration[bot]
5dc6261ebb
fix(bedrock): sign batch S3 requests with s3_access_key_id and s3_secret_access_key (#42342)
* fix(bedrock): sign batch S3 requests with s3_access_key_id and s3_secret_access_key

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(bedrock): keep S3 signer test additions scoped to new cases

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(bedrock): drop e2e suite changes from the S3 signing fix

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): build S3 credentials directly from the s3_* pair so ambient AWS_* env never mixes in

Restores the split-identity e2e coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:01:18 -07:00
devin-ai-integration[bot]
b720909dac
fix(bedrock): send every Mantle beta in the anthropic-beta header on the bedrock/mantle route (#42376)
- fix(bedrock): send every Mantle beta in the anthropic-beta header on the bedrock/mantle route
- refactor(bedrock): type the Mantle header helper and build the header fields in one comprehension
2026-09-21 18:52:29 -07:00
Mateo Wang
9fad216030
Merge pull request #42295 from BerriAI/litellm_fix_azure_cancellederror_cooldown
fix(azure): propagate asyncio.CancelledError instead of raising a 500
2026-09-21 17:24:22 -07:00
yucheng-berri
12f79308cf
Merge pull request #42354 from BerriAI/litellm_fix_video_key_guardrails_lit_6685
* fix(guardrails): scan video prompts for key-attached guardrails on /v1/videos

/v1/videos dispatches call_type avideo_generation, which CallTypes did not
know and no guardrail translation handler covered, so the unified guardrail
hook returned the request unscanned. Add the video call types and an OpenAI
video guardrail translation package that scans the prompt for create, remix,
edit and extension requests

Resolves LIT-6685

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ui): regenerate api types for video call types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: skip avideo_generation in azure sdk client exhaustive check

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(e2e): retry a leaked video job until the guardrail sync deadline

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): satisfy the type-discipline gate in the video handler

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(guardrails): gate the video e2e on a chat probe so a miss starts at most one paid job

Addresses Greptile review: typed RewritingGuardrail override, dropped routine docstrings, and the e2e waits for the key guardrail to sync via /chat/completions before its single /v1/videos call

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 17:23:56 -07:00
yucheng
a44befa8c5 test: skip avideo_generation in azure sdk client exhaustive check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 22:47:14 +00:00
kerry
fee8f38ccc Merge remote-tracking branch 'origin/main' into litellm_xai_max_completion_tokens 2026-09-21 22:42:12 +00:00
ryan-crabbe-berri
a776d0d722
Merge pull request #41561 from chopratejas/fix-responses-guardrail-custom-tool-output-writeback
fix(responses): patch custom_tool_call_output in place on guardrail write-back
2026-09-21 15:40:34 -07:00
kerry
32c63e332e Revert "fix(xai): accept max_completion_tokens as a supported param"
This reverts commit 4ff251e6b1.
2026-09-21 22:38:29 +00:00
kerry
4ff251e6b1 fix(xai): accept max_completion_tokens as a supported param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 22:37:40 +00:00
kerry
5c0589207c fix(xai): accept max_completion_tokens as a supported param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 22:30:13 +00:00
Mateo Wang
7ae61b62ed
Merge pull request #42323 from BerriAI/litellm_stream_usage_keep_explicit_zero
fix(streaming): keep an explicit provider prompt_tokens=0 or completion_tokens=0 in streamed usage
2026-09-21 15:13:21 -07:00
ryan-crabbe-berri
ddb236b7e2 Merge remote-tracking branch 'origin/main' into pr-41560
# Conflicts:
#	tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py
2026-09-21 15:04:57 -07:00
Yassin Kortam
f6c69af427
Merge pull request #41101 from hMED22/litellm_add_edenai_provider
feat(edenai): add Eden AI provider across chat, Responses, Messages, embeddings, audio, images and video
2026-09-21 16:16:28 -05:00
Mateo Wang
fc82f6e8fa
Merge pull request #42288 from BerriAI/litellm_safeguards_bedrock_vertex_messages
fix(anthropic): forward Claude Code safeguards and dangerous-tool-use beta to Bedrock Invoke and Vertex on /v1/messages
2026-09-21 14:15:36 -07:00
mateo-berri
a64febb3e7 fix(streaming): keep an explicit provider prompt_tokens=0 or completion_tokens=0 in streamed usage
The stream chunk builder started its per-chunk accumulators at 0 and adopted only nonzero counts, then fell back to litellm's tokenizer whenever the accumulated value was falsy, so a provider that reported an explicit 0 for prompt or completion tokens was billed the estimate instead. The accumulators now start at None, a usage chunk that reports a count marks it reported (a later chunk's 0 never replaces a reported nonzero), and the estimate only runs when no chunk reported the count. The Anthropic message_start cursor reset now yields None so the estimate still covers a cancelled stream, and Ollama chat streaming only attaches usage on the done chunk when both counts are present instead of inventing 0/0 on every chunk
2026-09-21 14:01:08 -07:00
kerry-berri
89de508d6e
Merge pull request #42306 from BerriAI/litellm_fal_ai_surface_video_errors
fix(fal_ai): surface fal errors in video status and content instead of completed and generic 500
2026-09-21 13:59:04 -07:00
kerry
0b9035b48f fix(fal_ai): handle transient result errors and inject clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 20:45:58 +00:00
kerry
eaa6936f13 fix(fal_ai): carry fal response into content errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 20:27:08 +00:00
mateo-berri
61fcfd986d fix(anthropic): map the dangerous-tool-use beta for Bedrock Mantle so safeguards never reach it without the beta 2026-09-21 13:24:52 -07:00
Mateo Wang
662e5b6e32
Merge pull request #42284 from BerriAI/litellm_qianwen_ai_platform_rename
fix: rename the mainland China brand to Qianwen AI Platform
2026-09-21 13:18:20 -07:00
kerry
a909a7908e fix(fal_ai): surface fal errors in video status and content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 20:17:29 +00:00
kerry-berri
5216844c40
Merge pull request #42286 from BerriAI/litellm_fal_ai_minimax_h3
feat(fal_ai): add MiniMax H3 text-to-video and reference-to-video
2026-09-21 13:16:08 -07:00
mateo-berri
b0651d52ec Merge remote-tracking branch 'origin/main' into litellm_safeguards_bedrock_vertex_messages 2026-09-21 13:15:33 -07:00
kerry-berri
246a6ea54a
Merge pull request #42282 from BerriAI/litellm_fal_price_from_response_dims
fix(fal_ai): price images from the dimensions fal returns
2026-09-21 13:08:49 -07:00
yucheng-berri
42519a7680
Merge pull request #42262 from BerriAI/litellm_bedrock_batch_s3_bucket_owner
* fix(bedrock): send s3BucketOwner on batch input and output data config

Resolve s3_bucket_owner from litellm_params, then optional_params, then
AWS_S3_BUCKET_OWNER and emit it on both S3 data configs so cross-account
batch buckets pass Bedrock ownership validation. Omitted when unset

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(bedrock): build batch output config with explicit returns

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 12:59:08 -07:00
kerry
b53f9ad658 fix(fal_ai): validate returned image dimensions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:58:59 +00:00
yassin
9dee1d86e7 fix(edenai): advertise reasoning_effort only for models the price map flags as reasoning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:58:58 +00:00
mateo-berri
bd8df0ed8f test: keep the Qwen brand missing-key tests off the network and split the ChatGPT fields test back out 2026-09-21 12:54:23 -07:00
mateo-berri
21b5e2d8a1 fix(anthropic): add the dangerous-tool-use beta when safeguards arrive without it on Bedrock Invoke and Vertex 2026-09-21 12:52:38 -07:00
Mateo Wang
2e35ae1065
Merge pull request #42049 from BerriAI/litellm_mantle_native_anthropic_messages
feat(bedrock_mantle): serve /v1/messages for Claude models on Mantle's native Anthropic Messages API
2026-09-21 12:52:16 -07:00
yassin
594d97e7e7 fix(edenai): restore unrelated files the squashed PR commit had reverted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:42:37 +00:00
yassin
7dc9609109 Merge branch 'main' into litellm_add_edenai_provider 2026-09-21 19:34:49 +00:00
mateo-berri
deca6aea79 test(bedrock): assert safeguard_results survive message_start and check the beta by membership 2026-09-21 12:30:11 -07:00
mateo-berri
0e16050100 fix(anthropic): forward Claude Code safeguards and dangerous-tool-use beta to Bedrock Invoke and Vertex on /v1/messages
Claude Code's server-side auto-mode classifier sends a `safeguards` body field
together with the `dangerous-tool-use-2026-09-03` beta. PR #42152 made the
first-party anthropic route pass them through, but the beta header mapping
left the other two Claude platforms at null, so Bedrock Invoke dropped both
(classifier silently disabled) and Vertex forwarded the body field without
the beta, which the platform rejects with "safeguards: Extra inputs are not
permitted" (a 400 Claude Code hides by retrying without them).

Map the beta for bedrock and vertex_ai in the beta headers config and add
`safeguards` to the Bedrock Invoke request allowlist so the pair reaches
both platforms unchanged. Nothing is injected: a client that sends
`safeguards` without the beta still gets the platform's 400, exactly as
api.anthropic.com answers it.
2026-09-21 12:21:50 -07:00
Mateo Wang
e7e4df9098
Merge pull request #42041 from BerriAI/litellm_azure_ai_gpt5_tools_responses_bridge
fix(azure_ai): bridge gpt-5.4+ function tools with reasoning to the Foundry Responses API
2026-09-21 12:21:02 -07:00
mateo-berri
cf00ab1bf8 fix: rename the mainland China brand to Qianwen AI Platform 2026-09-21 12:13:43 -07:00
kerry
9b54c4b077 refactor(fal_ai): bill flux dev per 1024x1024 megapixel and drop ImageResponse retyping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:10:33 +00:00
kerry
adc4e6a132 fix(fal_ai): price images from the dimensions fal returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:10:32 +00:00
kerry
8d73ce756a feat(fal_ai): add MiniMax H3 text-to-video and reference-to-video
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-21 19:03:00 +00:00
Mateo Wang
9a90adad32
Merge pull request #42275 from BerriAI/litellm_claude_platform_messages_beta_passthrough
fix(bedrock): forward anthropic-beta headers verbatim on the Claude platform messages path
2026-09-21 12:01:39 -07:00