Commit graph

13345 commits

Author SHA1 Message Date
Mateo Wang
67635d832e
Merge pull request #36638 from cu-aaii/litellm_redaction_deepcopy_aborts_success_logging
fix(logging): stop deepcopying results redaction cannot redact
2026-08-17 15:22:07 -07:00
mateo-berri
55e80849d1 feat(guardrails): track bedrock guardrail usage units per invocation 2026-08-17 15:18:53 -07:00
tin-berri
c1aae9d723
feat(shadow-eval): name the shadowed key in job responses and the UI headline (#37221) 2026-08-17 22:12:59 +00:00
yucheng-berri
7984ae4636
fix(mcp): scope authorization server issuer for named MCP servers (#37204)
* fix(mcp): scope authorization server issuer

Generated with AI

Co-Authored-By: Claude Code <noreply@anthropic.com>

* fix(mcp): keep the bare-origin issuer when no server was named

The scoped issuer must key off whether the request actually carried a server
name. _build_oauth_authorization_server_response rebinds mcp_server_name when
root discovery resolves the single configured OAuth2 server, so gating on the
rebound value also scoped /.well-known/openid-configuration, whose document is
served from the bare origin and whose issuer must stay the bare origin

Adds the named-server regression test for the reported mismatch, restores the
bare-origin assertion, and covers the OIDC document

* test(mcp): type the delegate_auth_to_upstream helper parameter

* refactor(mcp): bind the discovery issuer to a local before building the response

---------

Co-authored-by: Irosh <15094153+irosh-colombage-ZocDoc2@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
2026-08-17 15:06:12 -07:00
Mateo Wang
e8154c3e4c
Revert "fix(mcp): scope authorization server issuer" (#37220) 2026-08-17 15:06:01 -07:00
mateo-berri
678a636846 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_internal_copy_37077
# Conflicts:
#	tests/test_litellm/batches/test_batch_utils.py
2026-08-17 14:57:29 -07:00
Mateo Wang
de3c587d0a
Merge branch 'litellm_internal_staging' into litellm_openrouter_stream_usage_cost 2026-08-17 14:52:06 -07:00
Kent
9ff9f77137 test(router): cover s3_output_bucket_name surviving the trusted credential snapshot
The field itself landed on staging via 0c5c9c79d7; these are the regression tests from PR #31435 for the retrieval-facing half.

(cherry picked from commit a9a322d63f6d4658b1f28d1622335775e94736a4)
2026-08-17 14:51:10 -07:00
Kent
7bea4def27 fix(bedrock): validate file-content retrieval against the configured output bucket
Bedrock batch jobs write their results to s3_output_bucket_name when it differs
from the input bucket, but the file-content retrieval path validated the file id
only against the input bucket (s3_bucket_name). A deployment that configures a
separate output bucket therefore could not retrieve its own batch outputs: the
id validated against the input bucket and was rejected as a foreign bucket.

Resolve the trusted output bucket alongside the input bucket from the immutable
credential snapshot (or AWS_S3_OUTPUT_BUCKET_NAME), and try the file id against
each configured bucket, returning the first that validates. The SSRF guard is
preserved: only server-configured buckets are tried, never a request param, and
an id outside both is still rejected.

(cherry picked from commit 1d407c2f26)
2026-08-17 14:51:09 -07:00
Mateo Wang
3f4810b8f2
Merge pull request #37078 from cu-aaii/litellm_fix_bedrock_converse_batch_usage
fix(bedrock): read batch usage by payload shape, not by provider name
2026-08-17 14:51:02 -07:00
mateo-berri
8a43a8c7e3 fix(logging): merge deployment pricing onto a copy of the cached model info 2026-08-17 14:48:50 -07:00
mateo-berri
ce82a440d3 fix: register rust as a litellm param so it never leaks into provider request bodies 2026-08-17 14:48:01 -07:00
Mateo Wang
68d8cbeaf9
Merge pull request #37205 from BerriAI/litellm_lit5663_terminal_batch_output_cost
fix(proxy): bill cancelled and failed batches that still produced an output file
2026-08-17 14:47:18 -07:00
mateo-berri
5cf291a3ef Merge branch 'litellm_internal_staging' into litellm_gemini_36_flash_introductory_pricing 2026-08-17 14:46:26 -07:00
mateo-berri
97b7eaad5c fix(mcp): keep issuer-anchored slots on reload, skip discovery for stamped M2M challenge
Registry-swap reconciliation used bool(server.url) while registration uses _requires_oauth_discovery, dropping slots for issuer-anchored servers without a url. The preemptive 401 loop awaited discovery before the stamped client_credentials continue, so a deferred discovery failure could 503 requests whose challenge decision never reads metadata
2026-08-17 14:44:01 -07:00
Mateo Wang
c6fad3683d
Merge pull request #36032 from Scott-Wilson-ZocDoc/litellm_fix_responses_tool_choice_auto
fix(responses): unwrap object-form tool_choice before calling the Responses API
2026-08-17 14:37:38 -07:00
Marty Sullivan
bc977b76dc fix(batches): own deployment pricing per token direction, not per field
Filling each cost field independently let a published batch rate outrank a
standard rate the deployment configured itself: a deployment declaring only
input_cost_per_token had its batches billed at the model's published batch price
rather than half its own rate. Measured on a model that publishes both, that
billed $0.001500 where the deployment's own rate meant $0.000500.

Declaring either rate for a direction now claims that whole direction, so nothing
published can displace it, and a direction the deployment is silent on still
inherits both published rates.
2026-08-17 14:28:38 -07:00
Marty Sullivan
e7c2ce8624 test(batches): cover the deployment with no resolvable model at all
Codecov's remaining uncovered patch line was the early return taken when no model
is available to look a published entry up by, which leaves a deployment's own
declared rates standing alone. Measuring the patch lines against the coverage
report now leaves none uncovered.
2026-08-17 14:28:38 -07:00
Marty Sullivan
800e1d4f35 fix(cost): treat a batch rate configured as zero as free, not unset
batch_cost_calculator gated the batch rate fields on truthiness, so a deployment
that configures input_cost_per_token_batches or its output twin as 0.0 was read
as having configured nothing and that token direction fell through to half the
standard rate. Layering declared rates over published ones made this reachable:
a deployment declaring only a zero batch rate previously kept a fabricated zero
on the standard field, which happened to bill nothing.

The two batch fields are now gated on presence. Verified no cost-map entry
changes behavior: the only three carrying a zero batch rate are embeddings, whose
standard output rate is also 0.0, so both paths yield the same zero.

Adds a parametrized regression over an explicit zero, an explicit non-zero, and
unset, plus coverage for the deployment id get_model_info cannot resolve, which
were the lines Codecov flagged.
2026-08-17 14:28:38 -07:00
Marty Sullivan
b593cef758 fix(batches): keep published rates for a side the deployment leaves unset
Substituting a deployment's pricing wholesale billed the token direction it did
not configure at zero: get_model_info fills an absent cost with 0, and any
non-None pricing field suppressed the global fallback. A deployment declaring
only input_cost_per_token therefore billed output at nothing.

Each of the four batch cost fields now falls back to the model's published rate
when the deployment leaves it unset, so a one-sided override applies to the side
it configures and only that side.

Adds a parametrized regression over input-only, output-only, and both-zero, plus
coverage for a deployment whose model has no published entry. Annotates the new
test helpers per the repo's type-coverage rule and drops the narrative banner
comment from the batch tests.
2026-08-17 14:28:38 -07:00
Marty Sullivan
727905dfc3 fix(batches): only use deployment pricing when the deployment declares it
The router registers a model_info entry for every deployment, priced or not, and
get_model_info fills absent costs with 0. Resolving deployment pricing through it
therefore reported a free deployment for any ordinary one, which priced its batches
at $0 while usage stayed correct: the same silent under-count this branch set out
to remove, widened from bedrock to every provider.

Caught by a live batch run, where four vertex batches that price correctly today
came back at $0. The raw registration is now what decides: pricing is used only
when the deployment actually declares one of the batch cost fields, so ordinary
deployments fall back to the global cost map exactly as before.

The earlier test missed this by using a deployment id that was never registered,
where get_model_info does raise; a real deployment is always registered.
2026-08-17 14:28:37 -07:00
Marty Sullivan
f74c72eedb fix(batches): price a retrieved batch from its deployment's model and rates
Retrieving a completed batch computed its cost with no model identity: neither the
deployment's model nor its configured pricing reached the batch cost calculation.

For bedrock that left the cost model falling back to the provider's own response
model (e.g. "claude-sonnet-4-6"), which does not resolve under a bedrock provider,
so the lookup missed and cost silently became $0 while usage stayed correct.
Dropping the deployment's model info separately discarded any rates configured on
that deployment, billing a zero-cost deployment at the public rate instead.

Both are the same omission at the call site, so both are fixed by passing the
logging object's own model and the pricing the router registered for the
deployment.
2026-08-17 14:28:37 -07:00
mateo-berri
127b426991 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_azure_di_native_ocr_format 2026-08-17 14:28:17 -07:00
Mateo Wang
c1fc5983ca
Merge pull request #36482 from irosh-colombage-ZocDoc2/fix/mcp-oauth-scoped-issuer
fix(mcp): scope authorization server issuer
2026-08-17 14:27:38 -07:00
yuneng-jiang
77c8a6452f
Merge pull request #36258 from BerriAI/litellm_/elastic-ishizaka-db31e6
feat(proxy): let USE_V2_MIGRATION_RESOLVER select the v2 migration resolver
2026-08-17 14:13:04 -07:00
mateo-berri
5ddff616dc fix(mcp): keep origin issuer on the openid-configuration alias 2026-08-17 13:41:36 -07:00
Mateo Wang
5c15c097b6
Merge pull request #36033 from Scott-Wilson-ZocDoc/litellm_fix_session_resume_thinking
fix(anthropic): stop emitting empty thinking blocks on the Responses adapter
2026-08-17 13:32:34 -07:00
Mateo Wang
e11fe1d6cc
Merge pull request #36781 from daniel-meismer-zocdoc/feature/request-logs-user-id-filter
feat(ui): add user ID request log filter
2026-08-17 13:32:26 -07:00
Mateo Wang
47eaff19d3
Merge pull request #36979 from Scott-Wilson-ZocDoc/fix/anthropic-responses-optional-tool-props
fix(anthropic): preserve optional Responses tool properties
2026-08-17 13:26:14 -07:00
mateo-berri
d5a4c14577 docs(proxy): pre-fix passthrough streams omitted content-type, not octet-stream
Some checks failed
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
2026-08-17 13:21:28 -07:00
Mateo Wang
cde5465e6f
Merge pull request #37203 from BerriAI/litellm_batches_metadata_type_400
fix(proxy): return 400 for non-object metadata and litellm_metadata instead of silent drop or 500
2026-08-17 13:19:33 -07:00
Mateo Wang
3b6e56716a
Merge pull request #36978 from Scott-Wilson-ZocDoc/fix/mcp-guardrail-usage-monitor
fix(guardrails): record MCP tool guardrail evaluations and blocks in …
2026-08-17 13:18:03 -07:00
mateo-berri
3cdf041827 Merge branch 'litellm_internal_staging' into fix/anthropic-responses-optional-tool-props 2026-08-17 13:10:29 -07:00
mateo-berri
f86dc8f54e test(proxy): assert non-Bedrock passthrough stream emits no content-type header 2026-08-17 13:09:17 -07:00
mateo-berri
98bfbb99d2 refactor(responses): drop commentary from the tool_choice fix 2026-08-17 13:04:45 -07:00
mateo-berri
8ced9f56a1 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_4561_bedrock_passthrough_content_type 2026-08-17 13:04:18 -07:00
mateo-berri
11e2341fc9 Merge branch 'litellm_internal_staging' into feature/request-logs-user-id-filter 2026-08-17 13:01:19 -07:00
mateo-berri
c7b17b6615 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_pr36482_head 2026-08-17 12:54:14 -07:00
mateo-berri
cba076715d Merge remote-tracking branch 'origin/litellm_internal_staging' into pr36032_drive 2026-08-17 12:52:12 -07:00
mateo-berri
ce4eaa16e8 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_pr36033_drive 2026-08-17 12:50:32 -07:00
mateo-berri
09c8d1f1f5 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit5660_batches_limit_400 2026-08-17 12:46:15 -07:00
mateo-berri
a21de07b3f fix(proxy): drop every invalid metadata field before raising so failure hooks never see them 2026-08-17 12:44:46 -07:00
mateo-berri
73ae9c65dd test(proxy): give the x-litellm-model fallback test deterministic openai env creds 2026-08-17 12:42:57 -07:00
mateo-berri
2f9d331b4c fix(proxy): retire terminal batches whose advertised output file 404s instead of retrying 2026-08-17 12:36:39 -07:00
mateo-berri
f3b0cdca43 fix(proxy): keep ProxyException status codes on /v1/moderations instead of wrapping into 500 2026-08-17 12:32:30 -07:00
mateo-berri
21984101e5 fix(proxy): bill cancelled and failed batches that still produced an output file 2026-08-17 12:23:59 -07:00
mateo-berri
c894697a2a fix(proxy): keep ProxyException status codes on /v1/messages instead of wrapping into 500 2026-08-17 12:21:12 -07:00
Mateo Wang
badd737526
Merge pull request #37199 from BerriAI/litellm_lit5657_batches_400_missing_fields
fix(proxy): return 400 naming the missing required param on POST /v1/batches
2026-08-17 12:20:25 -07:00
mateo-berri
6e55a21ebb fix(proxy): enforce batch list limit bounds on managed passthrough listings 2026-08-17 12:15:22 -07:00
mateo-berri
57e1bf41c6 fix(proxy): return 400 for non-object metadata and litellm_metadata instead of silent drop or 500 2026-08-17 12:08:56 -07:00