Commit graph

46489 commits

Author SHA1 Message Date
Mateo Wang
9cd7696156
Merge pull request #37229 from BerriAI/litellm_comprehend_medical_passthrough
feat(proxy): add Amazon Comprehend Medical passthrough provider
2026-08-17 17:25:01 -07:00
mateo-berri
8ba2263d4c perf(guardrails): aggregate usage units in one sorted pass
The flush and the usage endpoints summed units with a scan per distinct key,
quadratic in rows times keys; group sorted rows instead. Skip payloads without
a request_id like the metrics path, type the flush key as a NamedTuple, and drop
the (guardrail_id, date) index that the primary key already covers
2026-08-17 17:19:05 -07:00
tin-berri
5277dab4f2
fix(shadow_eval): copy messages before router call and raise judge output cap (#37232)
* fix(shadow_eval): copy messages before router call and raise judge output cap

* fix(shadow_eval): lead failure detail with location and pin post-failure continuation
2026-08-17 17:10:52 -07:00
mateo-berri
7f42c84f57 fix(passthrough): dispatch Comprehend Medical logging on the provider tag only
Config-driven pass_through_endpoints pointed at a comprehendmedical.*.amazonaws.com
target were being claimed by the Comprehend Medical logging handler through the
hostname arm, which overrode their operator-set cost_per_request and relabeled
their spend rows. Only the built-in /comprehendmedical routes tag the provider,
so match on that alone.

Also mirror /comprehendmedical into the helm ingress and terraform gateway
prefix lists that hand-copy gateway/routes/allowlist.py
2026-08-17 17:07:31 -07:00
Mateo Wang
e81cedb13a
Merge pull request #37231 from BerriAI/litellm_lit5696_system_hoist_writeback
fix(anthropic): fold guardrail-modified leading system rows into top-level system param
2026-08-17 17:05:42 -07:00
mateo-berri
5437139b94 chore(guardrails): sync lazy OpenAPI snapshot and dashboard types for usage units
Regenerates the guardrails and policy_engine fragments of the lazy OpenAPI
snapshot for the usage-unit fields, regenerates schema.d.ts from it, and
exports DailyGuardrailUsageUnitsRepository next to its sibling repositories
2026-08-17 16:43:58 -07:00
mateo-berri
d9bd678f64 fix(gateway): expose comprehendmedical passthrough routes on the gateway component 2026-08-17 16:34:08 -07:00
Mateo Wang
b70df5bdf6
Merge pull request #37194 from BerriAI/litellm_azure_di_native_ocr_format
feat(ocr): return Azure Document Intelligence's native payload from /v1/ocr via req_format=native
2026-08-17 16:30:27 -07:00
mateo-berri
0cbec3f05c refactor(anthropic): drop bare generics and Any from new guardrail fold helpers 2026-08-17 16:17:00 -07:00
mubashir1osmani
4401b85855 Merge remote-tracking branch 'berri/litellm_internal_staging' into litellm_managed_batches_observability 2026-08-17 19:08:12 -04:00
Mateo Wang
50e71313b7
Merge pull request #37219 from BerriAI/litellm_internal_copy_37077
fix(batches): price a retrieved batch from its deployment's model and rates (internal copy of #37077)
2026-08-17 16:07:39 -07:00
mateo-berri
b7593a99c7 fix(guardrails): keep remaining usage upserts when one write fails
Per-row guards in the daily metrics and usage unit flush so a single DB error no longer drops the rest of the batch, plus removal of narrating comments flagged in review
2026-08-17 16:01:53 -07:00
mateo-berri
a972f172d7 fix(anthropic): fold guardrail-modified leading system rows into top-level system param 2026-08-17 15:57:46 -07:00
Mateo Wang
77b7c6c40c
Merge pull request #37198 from BerriAI/litellm_lit5660_batches_limit_400
fix(proxy): reject out-of-range limit on GET /v1/batches with OpenAI-parity 400
2026-08-17 15:53:46 -07:00
mateo-berri
7ba78c9ea9 fix(ocr): reject invalid req_format values as 400 on the SDK path 2026-08-17 15:49:18 -07:00
Mateo Wang
96d2ceef3b
Merge pull request #31435 from kingdoooo/litellm_bedrock_output_bucket
fix(bedrock): validate file-content retrieval against the configured output bucket (#26335)
2026-08-17 15:48:56 -07:00
mateo-berri
915a1cabcd feat(proxy): add Amazon Comprehend Medical passthrough provider 2026-08-17 15:44:06 -07:00
Mateo Wang
a6de0736e1
Merge pull request #37201 from BerriAI/litellm_fix_batches_404
fix(proxy): return 404 instead of 500 for unresolvable batch and file ids on /v1/batches
2026-08-17 15:43:49 -07:00
Mateo Wang
6858fda3ee
Merge pull request #37218 from BerriAI/litellm_lit5675_rust_param_leak
fix: stop rust flag from leaking into upstream provider request bodies
2026-08-17 15:43:31 -07:00
Yuneng Jiang
a7baf1d8e2
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/resolve-stacked-pr-conflicts-e95c06 2026-08-17 15:41:52 -07:00
Yuneng Jiang
3cc16de0d2
test(ui): settle the in-flight search before the loading tests end
Both loading tests mock searchToolQueryCall as a promise that resolves on a
timer, assert the loading affordance, then return with that promise still in
flight. When the worker outlives the file's jsdom environment, the component's
setIsLoading(false) runs against a torn-down window, and React reports
"ReferenceError: window is not defined" as an unhandled rejection. Vitest counts
that as an error, so ui-unit-tests fails the job while reporting every one of
its 7351 tests as passed.

Awaiting the settled state keeps both assertions and leaves nothing pending at
teardown.
2026-08-17 15:41:31 -07:00
mateo-berri
e736b59802 test(cost): type the batch_cost_calculator model_info literals instead of suppressing 2026-08-17 15:40:29 -07:00
mateo-berri
91c12ec810 test(proxy): run managed passthrough limit tests in CI 2026-08-17 15:39:46 -07:00
Mateo Wang
2975265f0b
Merge pull request #35013 from BerriAI/litellm_openrouter_stream_usage_cost
fix(streaming): track provider-reported cost when caller omits include_usage
2026-08-17 15:38:12 -07:00
Itai Modiano
fbd09ca27d
perf(guardrails): stop sending the conversation twice in the noma v2 payload (#36764)
The Noma guardrail sends the conversation to the scanner in `inputs`. It
also forwarded `request_data` whole, which repeats that same conversation
under `messages` (or `input` on the responses API), and attached
`logging_obj.model_call_details`, which repeats it a third time.

For image-heavy calls that duplication is most of the request. A
production scan of a request carrying base64 images measured 100MB total,
of which 94.8MB was `request_data` against 5.1MB of `inputs` - the proxy
was uploading ~95% redundant bytes, and paying to serialize them.

Drop `messages` and `input` from `request_data` and from
`model_call_details`. This is a denylist rather than an allowlist on
purpose: every other key is still forwarded untouched, so a scanner-side
change that starts reading a new `request_data` key needs no matching
release of this hook. The removed keys are ones the scanner never reads -
it takes context only from metadata, litellm_metadata,
provider_specific_header, litellm_session_id/trace_id/call_id, stream,
response/responses ids, and litellm_logging_obj.complete_streaming_response,
all of which still pass through.

The conversation still reaches the scanner in full via `inputs`, so no
detection coverage changes.

Trimming happens before serialization, so the duplicate is never encoded.

Existing payload tests asserted the duplication; they now assert the trim
while keeping what they originally guarded - deep-copy semantics and the
unpicklable-object (uvloop.Loop) regression.
2026-08-17 15:37:17 -07:00
mateo-berri
5a11fe141e fix(batches): price poller-tracked batches from the deployment's registered rates 2026-08-17 15:36:39 -07:00
Mateo Wang
ce20c282e9
Merge pull request #37197 from BerriAI/litellm_gemini_36_flash_introductory_pricing
fix(gemini): price gemini 3.6 flash at Google's introductory rates on every service tier
2026-08-17 15:32:08 -07:00
Mateo Wang
67635d832e
Merge pull request #36638 from cu-aaii/litellm_redaction_deepcopy_aborts_success_logging
fix(logging): stop deepcopying results redaction cannot redact
2026-08-17 15:22:07 -07:00
mateo-berri
55e80849d1 feat(guardrails): track bedrock guardrail usage units per invocation 2026-08-17 15:18:53 -07:00
yuneng-jiang
6a0bd0dffb
Merge pull request #37019 from BerriAI/litellm_mock_assertion_strength
test(ui): assert what collaborators are called with, not merely that they were
2026-08-17 15:18:21 -07:00
tin-berri
c1aae9d723
feat(shadow-eval): name the shadowed key in job responses and the UI headline (#37221) 2026-08-17 22:12:59 +00:00
Yuneng Jiang
6db82c0d15 test(ui): assert what collaborators are called with, not merely that they were
A bare toHaveBeenCalled() passes no matter what the caller passed, so the CSV
export could serialize the wrong rows, write the wrong content type, and name
the file wrong while its test stayed green.

Strengthens the load-bearing cases in three files onto the arguments that carry
the behaviour: the rows handed to the CSV serializer, the blob content type, the
anchor that gets attached and cleaned up, and the specific message each
validation failure shows the user. Two of the discount and margin tests
previously asserted the same bare call for different validation failures, so
neither could tell the two apart.

Each rewrite was proven by mutating the source it covers and confirming the test
goes red where the bare assertion stayed green.
2026-08-17 15:09:06 -07:00
yuneng-jiang
0d3dd8b4e1
Merge pull request #37018 from BerriAI/litellm_testing_library_lint_rules
test(ui): gate dashboard test assertions with testing-library and jest-dom rules
2026-08-17 15:09:02 -07:00
yucheng-berri
7984ae4636
fix(mcp): scope authorization server issuer for named MCP servers (#37204)
* fix(mcp): scope authorization server issuer

Generated with AI

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(mcp): keep the bare-origin issuer when no server was named

The scoped issuer must key off whether the request actually carried a server
name. _build_oauth_authorization_server_response rebinds mcp_server_name when
root discovery resolves the single configured OAuth2 server, so gating on the
rebound value also scoped /.well-known/openid-configuration, whose document is
served from the bare origin and whose issuer must stay the bare origin

Adds the named-server regression test for the reported mismatch, restores the
bare-origin assertion, and covers the OIDC document

* test(mcp): type the delegate_auth_to_upstream helper parameter

* refactor(mcp): bind the discovery issuer to a local before building the response

---------

Co-authored-by: Irosh <15094153+irosh-colombage-ZocDoc2@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-08-17 15:06:12 -07:00
Mateo Wang
e8154c3e4c
Revert "fix(mcp): scope authorization server issuer" (#37220) 2026-08-17 15:06:01 -07:00
ryan-crabbe-berri
427ed9e2d1
fix(ui): search every user in the Usage page user filter (#37206)
The User Usage view handed EntityUsage a static entityList holding only
the first /user/list page, so its filter could only find the 50 most
recently created users and anyone beyond that page, including users
with spend in the selected period, was unreachable

Add a self-contained UserDropdown that owns useInfiniteUsers (server-side
search plus load-more, mirroring TeamDropdown) and use it both as the
User Usage filterSlot and for the Global Usage user filter. Resolve a
selected user that is outside the loaded page by id so its label
survives view round-trips. Drop the now dead single-select branch from
UsageExportHeader
2026-08-17 15:00:38 -07:00
mateo-berri
678a636846 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_internal_copy_37077
# Conflicts:
#	tests/test_litellm/batches/test_batch_utils.py
2026-08-17 14:57:29 -07:00
mateo-berri
fbc5c1c3d4 refactor(streaming): copy usage-only chunk instead of mutating the parameter 2026-08-17 14:52:14 -07:00
Mateo Wang
de3c587d0a
Merge branch 'litellm_internal_staging' into litellm_openrouter_stream_usage_cost 2026-08-17 14:52:06 -07:00
mateo-berri
ba680246ee Merge pre-rebase history of litellm_bedrock_output_bucket; the branch was rebuilt on current litellm_internal_staging and the rebuilt commits supersede the old ones 2026-08-17 14:51:30 -07:00
Kent
9ff9f77137 test(router): cover s3_output_bucket_name surviving the trusted credential snapshot
The field itself landed on staging via 0c5c9c79d7; these are the regression tests from PR #31435 for the retrieval-facing half.

(cherry picked from commit a9a322d63f6d4658b1f28d1622335775e94736a4)
2026-08-17 14:51:10 -07:00
Kent
7bea4def27 fix(bedrock): validate file-content retrieval against the configured output bucket
Bedrock batch jobs write their results to s3_output_bucket_name when it differs
from the input bucket, but the file-content retrieval path validated the file id
only against the input bucket (s3_bucket_name). A deployment that configures a
separate output bucket therefore could not retrieve its own batch outputs: the
id validated against the input bucket and was rejected as a foreign bucket.

Resolve the trusted output bucket alongside the input bucket from the immutable
credential snapshot (or AWS_S3_OUTPUT_BUCKET_NAME), and try the file id against
each configured bucket, returning the first that validates. The SSRF guard is
preserved: only server-configured buckets are tried, never a request param, and
an id outside both is still rejected.

(cherry picked from commit 1d407c2f26)
2026-08-17 14:51:09 -07:00
Mateo Wang
3f4810b8f2
Merge pull request #37078 from cu-aaii/litellm_fix_bedrock_converse_batch_usage
fix(bedrock): read batch usage by payload shape, not by provider name
2026-08-17 14:51:02 -07:00
mateo-berri
8a43a8c7e3 fix(logging): merge deployment pricing onto a copy of the cached model info 2026-08-17 14:48:50 -07:00
mateo-berri
ce82a440d3 fix: register rust as a litellm param so it never leaks into provider request bodies 2026-08-17 14:48:01 -07:00
Mateo Wang
68d8cbeaf9
Merge pull request #37205 from BerriAI/litellm_lit5663_terminal_batch_output_cost
fix(proxy): bill cancelled and failed batches that still produced an output file
2026-08-17 14:47:18 -07:00
Yuneng Jiang
41d6dbb50c build(ui): gate dashboard test assertions with testing-library and jest-dom
Enables 14 rules at error with zero violations, so the vacuous-assertion class
the previous commit fixed cannot come back. No budget file and no suppressions
baseline: a rule is on only if it is already at zero.

prefer-to-have-value stays off. It matches any attribute whose name contains
"value", so it rewrites toHaveAttribute("aria-valuenow", n) into toHaveValue(n),
and jest-dom's toHaveValue supports only form controls, which breaks every
role="meter" element the dashboard renders.

Records the enabled set, the rules left off with their measured counts, and the
seven ways these plugins' autofixers produce broken output.
2026-08-17 14:46:55 -07:00
Yuneng Jiang
6670061a33 test(ui): assert presence and state so dashboard tests fail when behaviour breaks
One test could not fail: it asserted on the Promise returned by an unawaited
findAllByText, and a pending Promise is always truthy, so the organization
detail view had no working coverage at all.

Rewrites 316 assertions across 128 files onto matchers that describe what the
user perceives, and awaits the queries that were being compared as Promises.

Most of this was mechanical, but the fixers behind these rules are not
trustworthy, so every site they damaged was repaired by hand. The quiet one
worth naming: prefer-to-have-text-content wraps strings in new RegExp() without
escaping, turning toContain("100K+ requests") into a pattern meaning "100
followed by one-or-more K". That compiles, lints clean, and keeps passing while
no longer asserting what it claims.
2026-08-17 14:46:55 -07:00
yuneng-jiang
32e6346455
Merge pull request #37014 from BerriAI/litellm_/ui-ux-patterns-audit-1d04bc
test(ui): query antd controls accessibly instead of by internal CSS class
2026-08-17 14:46:50 -07:00
mateo-berri
5cf291a3ef Merge branch 'litellm_internal_staging' into litellm_gemini_36_flash_introductory_pricing 2026-08-17 14:46:26 -07:00