Commit graph

16707 commits

Author SHA1 Message Date
Tin Chi Lo
398300c4e7 fix(router): honor team and key provider weights 2026-09-14 23:31:52 -07:00
Mateo Wang
c93708b2a5
Merge pull request #40228 from AaronHowell/litellm_fix_responses_credentials_affinity
fix(responses): preserve provider affinity
2026-09-14 23:09:55 -07:00
yucheng-berri
91588221cd
Merge pull request #39050 from BerriAI/litellm_lit6314_guardrail_metadata_transfer
fix(guardrails): record not_run evaluation when scoping leaves nothing to scan
2026-09-14 22:31:16 -07:00
Mateo Wang
a78b24c195
Merge pull request #41172 from BerriAI/litellm_azure_spend_log_zero_cost
fix(proxy): log the provider usage on deferred /v1/messages calls and price cache writes without a creation rate
2026-09-14 22:26:22 -07:00
yucheng
fe0fb97fd2 fix(guardrails): record not_run when a skipped role mixes text and images
Only image-only unscoped content stays unrecorded; text or tool content
removed by scoping is recorded as not_run even when an image sits beside it.
Also keeps the type-discipline budget flat by returning the reason from the
helper and annotating the accumulator lists _extract_inputs requires.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 05:20:56 +00:00
yucheng
714b113c5f fix(guardrails): leave scoped-out image-only input unrecorded and split the not_run helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 05:10:32 +00:00
yucheng
9acdebf563 chore: merge origin/main into litellm_lit6314_guardrail_metadata_transfer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 05:03:31 +00:00
Mateo Wang
c16c172a65
Merge pull request #40939 from BerriAI/litellm_responses_per_message_guardrail_rewrite
fix(guardrails): write per-message guardrail rewrites back onto Responses input items
2026-09-14 21:57:29 -07:00
mateo-berri
d7a2bdd441 test(cost): type the cache rate cases of the base cost test 2026-09-14 21:56:23 -07:00
mateo-berri
8573241c49 fix(cost): resolve a missing 1h cache write rate after off-peak pricing
The one-hour cache write fallback now takes the applied cache write rate, so an off-peak write price carries into it instead of the input rate

The cost estimate test for a cost-map model without cache prices now expects writes at the input rate, which is what the proxy bills

The recording logger in the deferred guardrail test types its callback parameters
2026-09-14 21:46:33 -07:00
mateo-berri
f8c2539ba7 Merge remote-tracking branch 'origin/main' into litellm_azure_spend_log_zero_cost 2026-09-14 21:37:19 -07:00
yuneng-jiang
26a13132f8
Merge pull request #41149 from BerriAI/litellm_strict_provider_identity
test: add strict stateless provider replay identity
2026-09-14 21:31:25 -07:00
mateo-berri
e01d97ea08 fix(guardrails): read Prompt Security modified rows with the slot count's own predicate
A chat row whose content carried an empty text part counted two slots in the
chat completions handler while Prompt Security read one text out of the
modified row, so the structured rewrite was dropped and the request got the
named rejection. One shared helper now lists a row's slot texts and both the
slot count and the modified-row reader use it.
2026-09-14 21:31:22 -07:00
Yuneng Jiang
16fb44f23a
ci: run provider replay harness in CircleCI 2026-09-14 21:03:58 -07:00
Mateo Wang
2bdafa1206
Merge branch 'main' into fix/headroom-protect-cache-control-rows 2026-09-14 20:53:07 -07:00
yuneng-jiang
9d7f2aad04
Merge branch 'main' into litellm_strict_provider_identity 2026-09-14 20:43:55 -07:00
yuneng-jiang
15bd8b0e4a
Merge pull request #40892 from BerriAI/litellm_jwt_management_callers
test: bind management E2E callers and isolate JWT actors
2026-09-14 20:30:52 -07:00
kerry-berri
03a4c806c4
Merge pull request #41157 from BerriAI/litellm_gemini_embedding_2_per_token_billing
fix(cost): bill gemini-embedding-2 per token and stop double charging audio
2026-09-14 19:49:11 -07:00
kerry
954dfa6ba7 test(utils): allow modality batch cost fields in cost map schema test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 02:38:53 +00:00
kerry
7761d04450 test(vertex): cover malformed batch usage details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 02:37:51 +00:00
mateo-berri
22b377fe2a fix(proxy): log the provider usage on deferred /v1/messages calls and price cache writes without a creation rate
With a post-call guardrail the proxy defers async success logging, and every nested wrapper on a
/v1/messages call bridged to the Responses API overwrote the stored closure, so the spend log was
built from the outermost Anthropic-shaped reply under Responses semantics and recorded the prompt
tokens without the cache hit. The first wrapper to exit now keeps the slot, which is the innermost
provider response, the same one the non-deferred path logs.

The flat cost path also billed cache-creation tokens at 0 when the model had no
cache_creation_input_token_cost. It now falls back to the input rate, and the 1h rate to the
creation rate, matching the tiered path and the custom pricing helper.
2026-09-14 19:21:03 -07:00
kerry
c0c5044c45 fix(batches): keep modality token details in raw vertex batch usage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:57:50 +00:00
kerry
ce83fac351 fix(cost): bill batch embeddings per modality token rate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:50:21 +00:00
mateo-berri
16c326537f fix(guardrails): define UnappliableRequestRewrite in the shared guardrail translation utils
The three guardrail translation handlers imported the exception from the proxy policy engine through a function-local import, which CodeQL flagged as a cyclic import. The exception and its helper now live next to the handlers in the shared guardrail translation utils, and the tests import it from there.

The Prompt Security modify-mode helper is also restructured into early-return TypedDict displays so the LIT002 budget stays at its limit
2026-09-14 18:39:54 -07:00
Mateo Wang
d2859e18d7
Merge pull request #40988 from BerriAI/litellm_responses_stream_error_exception_mapping
fix(responses): route mid-stream error events through exception_type so content_policy_fallbacks fire
2026-09-14 18:36:20 -07:00
kerry
a28ea22ec1 fix(cost): move gemini-embedding-2-preview to per-token rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:34:22 +00:00
kerry
27a486e4d3 test(cost): cover modality guards and image detection fallbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:34:17 +00:00
Mateo Wang
1ba97665b2
Merge pull request #41046 from BerriAI/devin/1789368982-cli-strenum-py310
fix(cli): drop enum.StrEnum so the CLI imports on Python 3.10
2026-09-14 18:32:30 -07:00
mateo-berri
931bdb8c0b fix(compression): protect part-level cache_control rows in compress() too
compress() scores text-only copies of the rows, so a content-part cache_control marker was gone by the time get_protected_indices ran and the pinned row could still be stubbed. Read protection from the original rows, which are index-aligned with the normalized copies, and add a regression test that fails without the change.
2026-09-14 18:19:51 -07:00
mateo-berri
6574d83eae Merge remote-tracking branch 'origin/main' into HEAD
# Conflicts:
#	litellm/proxy/guardrails/guardrail_hooks/prompt_security/prompt_security.py
2026-09-14 18:19:38 -07:00
kerry
4a8ec7b9d8 fix(cost): bill gemini-embedding-2-preview per token like the GA entries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:18:50 +00:00
kerry
6a18105275 fix(vertex): only bill image rate without modality details when every input is an image
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:16:00 +00:00
kerry
ac8e1a355c test(vertex): load local pricing in embedding billing tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 01:00:32 +00:00
Mateo Wang
dad6e9be4a
Merge branch 'main' into fix/headroom-protect-cache-control-rows 2026-09-14 17:58:17 -07:00
yucheng-berri
8481bc27f9
Merge pull request #41132 from BerriAI/litellm_bedrock_grounding_from_plain_messages
fix(bedrock guardrails): derive contextual grounding source and query from plain messages
2026-09-14 17:56:58 -07:00
kerry
e5845c17ff fix(vertex): bill image inputs at the image rate when usage lacks modality details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:51:49 +00:00
kerry
0c91d9157c refactor(vertex): drop unused resolved_files from embed response parsing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:45:08 +00:00
kerry
e26a4970dd fix(test): complete synthetic model metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:42:21 +00:00
kerry
d4f2119b03 fix(cost): bill gemini-embedding-2 per token and stop double charging audio
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:41:46 +00:00
Yuneng Jiang
37bde0bdbe
test: keep provider request snapshots immutable 2026-09-14 17:27:18 -07:00
yucheng-berri
b67137b67f
Merge pull request #40702 from BerriAI/litellm_model_armor_logging_only_streaming
feat(model_armor): logging_only mode scans completed streams after delivery
2026-09-14 17:23:32 -07:00
Yuneng Jiang
51db940514
test: relocate strict replay harness coverage 2026-09-14 17:20:58 -07:00
Yuneng Jiang
b9c194076a
test: preserve strict replay numeric spelling 2026-09-14 17:11:56 -07:00
yucheng
5ceec4c21e fix(guardrails): cite message scoping only when an unscoped pass finds content
The not_run reason now says after message scoping only when the same messages carry text or tool calls without the skip flags applied. A request that is empty to begin with, whatever the flags, records no scannable content

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 00:03:02 +00:00
yucheng
52da64a45b fix(guardrails): only cite message scoping in the not_run reason when scoping is on
A request whose messages carry no scannable content at all, with no skip flag set, now records the neutral reason no scannable content instead of blaming configuration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 23:56:32 +00:00
Yuneng Jiang
6604c78120
test: add strict stateless provider replay identity 2026-09-14 16:55:43 -07:00
yucheng
0519d86346 fix(guardrails): stop labelling image-only input as a not_run scoping skip
Images without text were never dispatched to guardrails before this change, so that gap is not a message scoping skip and must not get a not_run entry

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-14 23:49:03 +00:00
Yassin Kortam
d1fc231545
Merge pull request #41145 from BerriAI/litellm_health_check_skip_save_on_failed_read_main 2026-09-14 16:48:47 -07:00
yucheng
0d0b96ed06 Revert "refactor(guardrails): rename scoped-out evaluation status from not_run to skipped"
This reverts commit b37ce94075.
2026-09-14 23:46:09 +00:00
yucheng
bd9a87ea76 Revert "fix(guardrails): keep legacy not_run neutral and stop labelling image-only input as skipped"
This reverts commit f78dd921c9.
2026-09-14 23:46:09 +00:00