Commit graph

3149 commits

Author SHA1 Message Date
Mateo Wang
1ce66e98a2
Merge pull request #40989 from BerriAI/litellm_responses_bridge_hoist_additional_tools
fix(responses): hoist Codex additional_tools input items into the chat bridge tools
2026-09-14 23:33:38 -07:00
mateo-berri
e3152c011d fix(responses): classify streamed tool calls on the chat name and strip guardrail edits around the grammar block
The streaming bridge restored the namespace before deciding whether a tool call was a custom tool, so a namespaced function sharing a short name with a nested custom tool streamed back as a custom_tool_call. Classify on the raw chat tool name first, the way the non-streaming path already does.

The guardrail merge only stripped the namespace prefix and grammar suffix from the ends of the edited description, so a guardrail appending text after the grammar block left the block in the member description and the chat conversion appended it a second time. Strip the first occurrence of each instead.
2026-09-14 22:58:25 -07:00
yucheng
fe0fb97fd2 fix(guardrails): record not_run when a skipped role mixes text and images
Only image-only unscoped content stays unrecorded; text or tool content
removed by scoping is recorded as not_run even when an image sits beside it.
Also keeps the type-discipline budget flat by returning the reason from the
helper and annotating the accumulator lists _extract_inputs requires.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 05:20:56 +00:00
yucheng
714b113c5f fix(guardrails): leave scoped-out image-only input unrecorded and split the not_run helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 05:10:32 +00:00
yucheng
9acdebf563 chore: merge origin/main into litellm_lit6314_guardrail_metadata_transfer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 05:03:31 +00:00
Mateo Wang
c16c172a65
Merge pull request #40939 from BerriAI/litellm_responses_per_message_guardrail_rewrite
fix(guardrails): write per-message guardrail rewrites back onto Responses input items
2026-09-14 21:57:29 -07:00
kerry
7761d04450 test(vertex): cover malformed batch usage details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 02:37:51 +00:00
mateo-berri
16c326537f fix(guardrails): define UnappliableRequestRewrite in the shared guardrail translation utils
The three guardrail translation handlers imported the exception from the proxy policy engine through a function-local import, which CodeQL flagged as a cyclic import. The exception and its helper now live next to the handlers in the shared guardrail translation utils, and the tests import it from there.

The Prompt Security modify-mode helper is also restructured into early-return TypedDict displays so the LIT002 budget stays at its limit
2026-09-14 18:39:54 -07:00
kerry
27a486e4d3 test(cost): cover modality guards and image detection fallbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:34:17 +00:00
mateo-berri
6574d83eae Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts:
#	litellm/proxy/guardrails/guardrail_hooks/prompt_security/prompt_security.py
2026-09-14 18:19:38 -07:00
kerry
4a8ec7b9d8 fix(cost): bill gemini-embedding-2-preview per token like the GA entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:18:50 +00:00
kerry
6a18105275 fix(vertex): only bill image rate without modality details when every input is an image
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:16:00 +00:00
kerry
ac8e1a355c test(vertex): load local pricing in embedding billing tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:00:32 +00:00
kerry
e5845c17ff fix(vertex): bill image inputs at the image rate when usage lacks modality details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:51:49 +00:00
kerry
0c91d9157c refactor(vertex): drop unused resolved_files from embed response parsing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:45:08 +00:00
kerry
d4f2119b03 fix(cost): bill gemini-embedding-2 per token and stop double charging audio
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:41:46 +00:00
yucheng
5ceec4c21e fix(guardrails): cite message scoping only when an unscoped pass finds content
The not_run reason now says after message scoping only when the same messages carry text or tool calls without the skip flags applied. A request that is empty to begin with, whatever the flags, records no scannable content

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:03:02 +00:00
yucheng
52da64a45b fix(guardrails): only cite message scoping in the not_run reason when scoping is on
A request whose messages carry no scannable content at all, with no skip flag set, now records the neutral reason no scannable content instead of blaming configuration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 23:56:32 +00:00
yucheng
0519d86346 fix(guardrails): stop labelling image-only input as a not_run scoping skip
Images without text were never dispatched to guardrails before this change, so that gap is not a message scoping skip and must not get a not_run entry

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 23:49:03 +00:00
yucheng
0d0b96ed06 Revert "refactor(guardrails): rename scoped-out evaluation status from not_run to skipped"
This reverts commit b37ce94075.
2026-09-14 23:46:09 +00:00
yucheng
bd9a87ea76 Revert "fix(guardrails): keep legacy not_run neutral and stop labelling image-only input as skipped"
This reverts commit f78dd921c9.
2026-09-14 23:46:09 +00:00
tin-berri
c626ff098b
Merge pull request #40877 from BerriAI/litellm_lit7658_cache_cost_v0_fresh
feat(proxy): predict prompt-cache costs across deployments
2026-09-14 16:25:41 -07:00
yucheng
f78dd921c9 fix(guardrails): keep legacy not_run neutral and stop labelling image-only input as skipped
Some checks failed
ai-gateway image / ai-gateway release image (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Usage tracking, compliance and the dashboard now treat both not_run (older spend logs) and skipped as unevaluated through a shared UNEVALUATED_GUARDRAIL_STATUSES set, so old records stop counting as passed. The skipped record is no longer written when the request carried images, since images without text were never dispatched to guardrails before this change and that gap is not a message-scoping skip

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:34:33 +00:00
yucheng
b37ce94075 refactor(guardrails): rename scoped-out evaluation status from not_run to skipped
The per-guardrail status a scoped-out evaluation records is now skipped, matching the
skip_*_in_guardrail settings that cause it. Request-level rollup still maps it to not_run
so the StandardLoggingPayload status contract is unchanged

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 22:15:53 +00:00
yassin
fbc1011d27 fix(bedrock): end the realtime session when the client disconnects instead of waiting for Nova Sonic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:42:42 +00:00
yassin
b4d0f4ad26 refactor(realtime): move session ownership marker keys into constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:33:48 +00:00
yassin
d2342f06ce fix(bedrock): stamp the realtime success ownership marker when Nova Sonic spend is logged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 21:13:29 +00:00
yassin
771430df7b fix(bedrock/realtime): keep partial spend on cancelled output task and make the committed-session refusal non-retryable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:07:21 +00:00
yassin
5c04ec2b93 fix(bedrock/realtime): keep the pending session.update until a provider stream is committed
Peek at the pending session.update instead of popping it, so an eager fallback failure before the bridge starts does not lose the replay for the next attempt. Move the websocket scope keys to constants.py

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:07:20 +00:00
yassin
cb5d901774 fix(bedrock/realtime): propagate deferred Nova Sonic stream failures to the router
Bedrock realtime caught every exception inside both forwarding tasks and
gathered them with return_exceptions=True, so a provider failure surfacing
after the websocket handshake (lazy duplex stream: 503/429/validation only
show up on await_output or the input publisher) made async_realtime return
normally and the router recorded a success instead of running fallbacks and
cooldown accounting. session.updated is now acked only after Bedrock is
ready, provider failures escape as BedrockError with the AWS status code,
a failure after the client disconnected is not reported as a provider
failure, and a fallback attempt on the same websocket replays the pending
session.update instead of emitting a second session.created.

Resolves LIT-6484

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 19:07:20 +00:00
yucheng
b4e3d7c533 chore: merge main into litellm_lit6314_guardrail_metadata_transfer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 17:33:04 +00:00
mateo-berri
7ea19eccc7 fix(responses): keep namespace custom tools through guardrail merges and Mantle params identity 2026-09-13 03:02:03 -07:00
mateo-berri
37447c98f7 fix(guardrails): reject per-message texts that cannot land on a string input or a Messages request 2026-09-13 02:28:43 -07:00
mateo-berri
d26e007229 Merge remote-tracking branch 'origin/main' into litellm_responses_per_message_guardrail_rewrite 2026-09-13 01:05:36 -07:00
mateo-berri
09314f239c fix(guardrails): hand per-message rewrites back as structured_messages
A guardrail that rewrites text per chat message now returns the rewritten
rows as structured_messages instead of only texts, so the Responses and
chat handlers write the rewrite back through the structured path. The
generic guardrail API response accepts an optional structured_messages
list, Prompt Security modify builds one from modified_messages, and rows a
server echoes back exactly as shown are restored to the original row
objects because the request model drops undeclared keys. Texts-only
per-message answers keep the named rejection on both endpoints.
2026-09-13 00:54:29 -07:00
Mateo Wang
c2c2a623c0
Merge pull request #39846 from BerriAI/litellm_bedrock_mantle_govcloud_cost_row
Some checks are pending
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / caching-local (push) Waiting to run
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / integrations (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
Unit Tests: Documentation Validation / documentation (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests / misc (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
fix(bedrock_mantle): price GovCloud regions from the regional cost row and accept region-prefixed model names
2026-09-12 21:13:58 -07:00
devin-ai-integration[bot]
77dc1a6c03
fix(anthropic-adapter): surface mid-stream provider errors as Anthropic error events (#33352)
* fix(anthropic-adapter): surface mid-stream provider errors as Anthropic error events

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* style(anthropic-adapter): drop added comments per repo convention

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-09-12 21:13:35 -07:00
Mateo Wang
70e3f5a02e
Merge pull request #39836 from BerriAI/litellm_lit_6975_bedrock_files_delete_list
feat(bedrock): support file delete and list for S3-backed managed files
2026-09-12 21:13:27 -07:00
kerry-berri
9ae727bc8e
Merge pull request #40929 from BerriAI/litellm_fireworks_short_key_lookup
fix(fireworks): resolve short model names to long cost map keys
2026-09-12 20:49:44 -07:00
Devin AI
a519d805bb refactor(fireworks): resolve cost map key through a provider config hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-13 03:29:11 +00:00
mateo-berri
a635d7be6a fix(guardrails): write per-message guardrail rewrites back onto Responses input items
A guardrail that answers one rewritten text per message it saw no longer
matches the texts the Responses handler extracted once the request carries
instructions or tool items, so the rewrite was rejected with a 500. Spread
such an answer over the structured messages' text slots and write it back
through the structured path, have Prompt Security modify return
structured_messages directly, and give the chat completions pairing the same
named rejection instead of a silent misalignment when the counts differ.
2026-09-12 20:29:09 -07:00
Devin AI
0904051fda refactor(fireworks): move cost map key construction under llms/
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-13 03:24:23 +00:00
Mateo Wang
9d984371fd
Merge pull request #40909 from BerriAI/litellm_databricks_reasoning_effort_thinking
fix(databricks): translate reasoning_effort to thinking for Gemini 2.5
2026-09-12 17:46:44 -07:00
ryan-crabbe-berri
c134fb7a38
Merge pull request #39395 from seyeong-han/litellm_meta_muse_voice_realtime
Some checks failed
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests / caching-local (push) Waiting to run
Unit Tests / core-utils (push) Waiting to run
Unit Tests / enterprise-package (push) Waiting to run
Unit Tests / enterprise-routing (push) Waiting to run
Unit Tests / integrations (push) Waiting to run
Unit Tests / All Other Providers (push) Waiting to run
Unit Tests / Vertex AI (push) Waiting to run
Unit Tests / misc (push) Waiting to run
Unit Tests / proxy-auth (push) Waiting to run
Unit Tests / proxy-endpoints (push) Waiting to run
Unit Tests / proxy-extras (push) Waiting to run
Unit Tests / proxy-infra (push) Waiting to run
Unit Tests / proxy-server (push) Waiting to run
Unit Tests / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
feat(realtime): add Meta Muse Voice transcription
2026-09-12 15:58:24 -07:00
Mateo Wang
fe5ff9d3b0
Merge pull request #40771 from BerriAI/litellm_regression_coverage_followup
test: tighten regression tests added in #37974
2026-09-12 15:29:36 -07:00
Mateo Wang
c040061f15
Merge pull request #40766 from BerriAI/litellm_fix_v1_messages_disconnect_partial_cost
fix(anthropic): price recovered tokens when a /v1/messages client disconnects mid-stream
2026-09-12 15:21:43 -07:00
ryan-crabbe-berri
4647cd1215 fix(realtime): keep a Muse turn active for turnless partials after speechEnd
Muse partials carry no turnId and belong to the most recent speechStart,
and the docs say the model may keep post processing a turn after speechEnd
until speechComplete. Releasing the active turn on speechEnd made any
partial arriving in that window raise and get dropped in ENDPOINTING mode.
The turn now stays active until its speechComplete or final transcript.
2026-09-12 15:13:44 -07:00
Tin Chi Lo
cba843cc16 feat(proxy): predict prompt-cache costs across deployments 2026-09-12 14:02:45 -07:00
mateo-berri
c7b607c46e fix(databricks): keep the Claude fallback when gating the anthropic thinking payload
Some checks failed
ai-gateway image / ai-gateway release image (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Gate the reasoning_effort translation on the cost-map flag or the model name containing
claude, so unmapped Claude serving endpoints keep translating. Flag the newer Claude
entries that were missing it. Expose supports_anthropic_thinking_payload as a public
helper next to the other supports_* wrappers instead of importing the private factory.
Drop the adaptive-only guard, since the adaptive flags only ever match Claude ids, and
add regression tests for an unmapped Claude endpoint and an adaptive Claude model
2026-09-12 13:13:38 -07:00
mateo-berri
10a0da7a32 Merge litellm_internal_staging into devin/1784568628-databricks-gemini-reasoning-effort 2026-09-12 13:08:22 -07:00