Commit graph

46534 commits

Author SHA1 Message Date
Marty Sullivan
b593cef758 fix(batches): keep published rates for a side the deployment leaves unset
Substituting a deployment's pricing wholesale billed the token direction it did
not configure at zero: get_model_info fills an absent cost with 0, and any
non-None pricing field suppressed the global fallback. A deployment declaring
only input_cost_per_token therefore billed output at nothing.

Each of the four batch cost fields now falls back to the model's published rate
when the deployment leaves it unset, so a one-sided override applies to the side
it configures and only that side.

Adds a parametrized regression over input-only, output-only, and both-zero, plus
coverage for a deployment whose model has no published entry. Annotates the new
test helpers per the repo's type-coverage rule and drops the narrative banner
comment from the batch tests.
2026-08-17 14:28:38 -07:00
Marty Sullivan
964c9a3ca2 fix(batches): resolve the deployment model from model_call_details
On a batch retrieve both self.model and litellm_params[model] come back None, so
the cost model fell through to the provider's own response model (an Anthropic id
like claude-opus-4-5-20251101) which does not resolve under a bedrock provider,
leaving bedrock batches at $0 with correct usage.

model_call_details carries the deployment's provider-qualified model
(bedrock/global.anthropic.claude-opus-4-5-20251101-v1:0), confirmed by
instrumenting a live retrieve, so it is preferred with the previous two sources
kept as fallbacks.
2026-08-17 14:28:38 -07:00
Marty Sullivan
0797e266cd fix(batches): price against the deployment model, not the router alias
self.model can carry the router's model_group alias, which no cost map resolves,
so a bedrock batch still priced at $0 after the model name started being passed.
The deployment's own litellm_params model is used when present.

Verified against the local (image-bound) cost map that dev and prod both force:
alias 'claude-opus-4-5' prices $0.000000 while
'bedrock/global.anthropic.claude-opus-4-5-20251101-v1:0' prices $0.017000
2026-08-17 14:28:38 -07:00
Marty Sullivan
727905dfc3 fix(batches): only use deployment pricing when the deployment declares it
The router registers a model_info entry for every deployment, priced or not, and
get_model_info fills absent costs with 0. Resolving deployment pricing through it
therefore reported a free deployment for any ordinary one, which priced its batches
at $0 while usage stayed correct: the same silent under-count this branch set out
to remove, widened from bedrock to every provider.

Caught by a live batch run, where four vertex batches that price correctly today
came back at $0. The raw registration is now what decides: pricing is used only
when the deployment actually declares one of the batch cost fields, so ordinary
deployments fall back to the global cost map exactly as before.

The earlier test missed this by using a deployment id that was never registered,
where get_model_info does raise; a real deployment is always registered.
2026-08-17 14:28:37 -07:00
Marty Sullivan
f74c72eedb fix(batches): price a retrieved batch from its deployment's model and rates
Retrieving a completed batch computed its cost with no model identity: neither the
deployment's model nor its configured pricing reached the batch cost calculation.

For bedrock that left the cost model falling back to the provider's own response
model (e.g. "claude-sonnet-4-6"), which does not resolve under a bedrock provider,
so the lookup missed and cost silently became $0 while usage stayed correct.
Dropping the deployment's model info separately discarded any rates configured on
that deployment, billing a zero-cost deployment at the public rate instead.

Both are the same omission at the call site, so both are fixed by passing the
logging object's own model and the pricing the router registered for the
deployment.
2026-08-17 14:28:37 -07:00
mateo-berri
127b426991 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_azure_di_native_ocr_format 2026-08-17 14:28:17 -07:00
Mateo Wang
c1fc5983ca
Merge pull request #36482 from irosh-colombage-ZocDoc2/fix/mcp-oauth-scoped-issuer
fix(mcp): scope authorization server issuer
2026-08-17 14:27:38 -07:00
Yuneng Jiang
b0f00e3a19
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/ui-ux-patterns-audit-1d04bc 2026-08-17 14:19:30 -07:00
Yuneng Jiang
75abe09f47
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_/ui-ux-patterns-audit-1d04bc
The auto-router tests moved onto accessible queries here while staging added a
"Lite" preset and a default-model pin, so the option-label expectations take
staging's list read through this branch's visibleOptions helper.

Staging's new pin tests reached for antd's internal classes, which the lint rule
this branch enables rejects. The edit-modal cases now read the rendered
selection through one selectedValueIn helper, and the clear-affordance click in
ComplexityRouterConfig keeps a reasoned suppression since antd marks that icon
aria-hidden.
2026-08-17 14:14:08 -07:00
yuneng-jiang
77c8a6452f
Merge pull request #36258 from BerriAI/litellm_/elastic-ishizaka-db31e6
feat(proxy): let USE_V2_MIGRATION_RESOLVER select the v2 migration resolver
2026-08-17 14:13:04 -07:00
yuneng-jiang
a7be39336b
Merge pull request #36975 from BerriAI/litellm_/gateway-outage-root-cause-dcba7e
fix(helm): bound the migrations Job so a blocked migration cannot stall the release
2026-08-17 14:12:58 -07:00
yuneng-jiang
4503efc8b6
Merge pull request #37061 from BerriAI/litellm_/invite-button-shadcn-decouple-1463e5
refactor(ui): decouple bulk invite from the invite user button
2026-08-17 14:03:54 -07:00
mateo-berri
5ddff616dc fix(mcp): keep origin issuer on the openid-configuration alias 2026-08-17 13:41:36 -07:00
Mateo Wang
f1e1c89600
Merge pull request #36790 from daniel-meismer-zocdoc/litellm_searchable_usage_report_user_filter
fix(ui): make per-user usage filter searchable
2026-08-17 13:39:30 -07:00
Mateo Wang
5c15c097b6
Merge pull request #36033 from Scott-Wilson-ZocDoc/litellm_fix_session_resume_thinking
fix(anthropic): stop emitting empty thinking blocks on the Responses adapter
2026-08-17 13:32:34 -07:00
Mateo Wang
e11fe1d6cc
Merge pull request #36781 from daniel-meismer-zocdoc/feature/request-logs-user-id-filter
feat(ui): add user ID request log filter
2026-08-17 13:32:26 -07:00
Mateo Wang
47eaff19d3
Merge pull request #36979 from Scott-Wilson-ZocDoc/fix/anthropic-responses-optional-tool-props
fix(anthropic): preserve optional Responses tool properties
2026-08-17 13:26:14 -07:00
mateo-berri
d5a4c14577 docs(proxy): pre-fix passthrough streams omitted content-type, not octet-stream
Some checks failed
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
2026-08-17 13:21:28 -07:00
Mateo Wang
cde5465e6f
Merge pull request #37203 from BerriAI/litellm_batches_metadata_type_400
fix(proxy): return 400 for non-object metadata and litellm_metadata instead of silent drop or 500
2026-08-17 13:19:33 -07:00
Mateo Wang
3b6e56716a
Merge pull request #36978 from Scott-Wilson-ZocDoc/fix/mcp-guardrail-usage-monitor
fix(guardrails): record MCP tool guardrail evaluations and blocks in …
2026-08-17 13:18:03 -07:00
mateo-berri
d857ae3844 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_pr36790_drive
# Conflicts:
#	ui/litellm-dashboard/src/app/(dashboard)/usage/_components/components/EntityUsage/EntityUsage.tsx
#	ui/litellm-dashboard/src/components/EntityUsageExport/UsageExportHeader.tsx
2026-08-17 13:13:25 -07:00
mateo-berri
3cdf041827 Merge branch 'litellm_internal_staging' into fix/anthropic-responses-optional-tool-props 2026-08-17 13:10:29 -07:00
mateo-berri
5295da055c refactor(anthropic): take Mapping in _open_block to satisfy LIT001 2026-08-17 13:10:21 -07:00
mateo-berri
f86dc8f54e test(proxy): assert non-Bedrock passthrough stream emits no content-type header 2026-08-17 13:09:17 -07:00
mateo-berri
9288d96401 refactor(mcp): drop redundant issuer comment 2026-08-17 13:08:14 -07:00
mateo-berri
98bfbb99d2 refactor(responses): drop commentary from the tool_choice fix 2026-08-17 13:04:45 -07:00
mateo-berri
8ced9f56a1 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_4561_bedrock_passthrough_content_type 2026-08-17 13:04:18 -07:00
ryan-crabbe-berri
d0a815d2e1
fix(ui): stop pairing key spend with the team budget when a key has no budget (#37196)
The key overview card and the Virtual Keys spend meter fell back to the parent team's max_budget as the denominator while the numerator stayed the key's own spend, so a $0.50 key on a $1,200 team read as "$0.50 of $1,200 (Team)" and drew a meter against a limit that governs the whole team's aggregate spend, not this key. Both surfaces now show Unlimited for a budgetless key and, when the parent team or organization does carry a budget, a hover hint listing those inherited caps so the reader knows what still gates the key
2026-08-17 13:03:05 -07:00
mateo-berri
11e2341fc9 Merge branch 'litellm_internal_staging' into feature/request-logs-user-id-filter 2026-08-17 13:01:19 -07:00
mateo-berri
c7b17b6615 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_pr36482_head 2026-08-17 12:54:14 -07:00
mateo-berri
cba076715d Merge remote-tracking branch 'origin/litellm_internal_staging' into pr36032_drive 2026-08-17 12:52:12 -07:00
mateo-berri
ce4eaa16e8 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_pr36033_drive 2026-08-17 12:50:32 -07:00
mateo-berri
b34cbb18e2 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit5663_terminal_batch_output_cost 2026-08-17 12:46:46 -07:00
mateo-berri
09c8d1f1f5 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit5660_batches_limit_400 2026-08-17 12:46:15 -07:00
mateo-berri
a21de07b3f fix(proxy): drop every invalid metadata field before raising so failure hooks never see them 2026-08-17 12:44:46 -07:00
mateo-berri
73ae9c65dd test(proxy): give the x-litellm-model fallback test deterministic openai env creds 2026-08-17 12:42:57 -07:00
Mateo Wang
f04b1dadfb
Merge pull request #37200 from BerriAI/litellm_bump_sqlparse_osv
fix(ci): bump sqlparse to 0.6.0 to resolve osv-scan CVEs
2026-08-17 12:40:43 -07:00
mateo-berri
2f9d331b4c fix(proxy): retire terminal batches whose advertised output file 404s instead of retrying 2026-08-17 12:36:39 -07:00
mateo-berri
f3b0cdca43 fix(proxy): keep ProxyException status codes on /v1/moderations instead of wrapping into 500 2026-08-17 12:32:30 -07:00
mateo-berri
21984101e5 fix(proxy): bill cancelled and failed batches that still produced an output file 2026-08-17 12:23:59 -07:00
mateo-berri
c894697a2a fix(proxy): keep ProxyException status codes on /v1/messages instead of wrapping into 500 2026-08-17 12:21:12 -07:00
Mateo Wang
badd737526
Merge pull request #37199 from BerriAI/litellm_lit5657_batches_400_missing_fields
fix(proxy): return 400 naming the missing required param on POST /v1/batches
2026-08-17 12:20:25 -07:00
mateo-berri
6e55a21ebb fix(proxy): enforce batch list limit bounds on managed passthrough listings 2026-08-17 12:15:22 -07:00
Thijmen Stavenuiter
4e3e7d8cf6 test: cover soft budget row creation and windows without max_budget 2026-08-17 21:12:36 +02:00
mateo-berri
57e1bf41c6 fix(proxy): return 400 for non-object metadata and litellm_metadata instead of silent drop or 500 2026-08-17 12:08:56 -07:00
Thijmen Stavenuiter
c7abb519a1 Merge branch 'litellm_internal_staging' of https://github.com/BerriAI/litellm into key-budget-window-usage 2026-08-17 21:04:01 +02:00
mateo-berri
4a7dfd75fc fix(proxy): return 404 instead of 500 for unresolvable batch and file ids on /v1/batches 2026-08-17 12:02:15 -07:00
mateo-berri
4ef7c3f41c fix(ci): bump sqlparse to 0.6.0 to resolve osv-scan CVEs 2026-08-17 12:02:14 -07:00
mateo-berri
a9bb09905d test(batches): drop redundant section banner 2026-08-17 12:00:40 -07:00
Thijmen Stavenuiter
ffd23ab131 style: apply ruff format to budget limits comprehension 2026-08-17 21:00:24 +02:00