Commit graph

42314 commits

Author SHA1 Message Date
Devin AI
1c9c6b61e0 fix(spend_tracking): record the selected deployment on failed request spend logs 2026-08-07 10:32:03 +00:00
Mateo Wang
b66d4e6965
Merge pull request #35137 from BerriAI/litellm_fix_responses_cost_router_35131
fix(proxy): fetch background responses through the router in CheckResponsesCost
2026-08-06 03:26:36 -07:00
Mateo Wang
972c0d0b04
Merge pull request #35140 from BerriAI/litellm_fix_file_content_placeholder_cost
fix(cost): stop token-pricing the placeholder input on file content calls
2026-08-06 03:04:36 -07:00
Mateo Wang
729bec69f5
Merge pull request #36014 from BerriAI/litellm_scan_only_tool_results
feat(guardrails): add scan_only_tool_results to scope unified guardrails to tool results
2026-08-06 02:17:29 -07:00
mateo-berri
14d4897e55 fix(guardrails): refuse scan_only_tool_results combos that scan nothing
Prompt Security drops tool and function rows unless check_tool_results
is on, so it now reports scan-only support from that setting and the
registry refuses the pairing at boot. Pairing scan_only_tool_results
with skip_tool_message_in_guardrail excludes every message, so guardrail
initialization now rejects that combination too.
2026-08-06 01:58:06 -07:00
Mateo Wang
a0e627f99d
Merge pull request #35468 from elinacse/bugfix/managed-batch-cost-not-logged
fix(batch): track cost for managed batches with no attributable key/u…
2026-08-06 01:51:23 -07:00
mateo-berri
28ff7f3f0b fix(guardrails): scan function-role results and dedupe returned tools
Under scan_only_tool_results, legacy OpenAI function-role messages now count as tool results, and duplicate names among guardrail-returned tools keep only the first occurrence. CustomGuardrail.structured_messages_cover_full_request lets CrowdStrike AIDR declare that its writeback already rebuilds the whole conversation, so handlers install it as-is instead of merging it into the full message list a second time and duplicating out-of-scope rows. Lint budget ceilings ratchet down to match the tree
2026-08-06 00:52:16 -07:00
mateo-berri
0b24a9ab05 merge: litellm_internal_staging into litellm_fix_file_content_placeholder_cost 2026-08-06 00:09:27 -07:00
mateo-berri
7d745521bf fix(guardrails): merge synthesized tools under scan_only_tool_results and reject role-filtered no-op combos at init 2026-08-05 23:46:40 -07:00
mateo-berri
e0c4c7cee0 Merge branch 'litellm_internal_staging' into bugfix/managed-batch-cost-not-logged 2026-08-05 23:42:18 -07:00
Devin AI
55c392bda1 merge litellm_internal_staging 2026-08-06 06:15:17 +00:00
mateo-berri
4f7d1fce3a fix(proxy): fall back to the SDK when a queued response's deployment is missing 2026-08-05 23:13:15 -07:00
Mateo Wang
0acca3e86a
Merge pull request #24548 from mpcusack-altos/fix/bedrock-batch-credential-fields
fix(router): include Bedrock batch/S3 fields and model in deployment credentials
2026-08-05 22:39:39 -07:00
tin-berri
34fc8d2ee7
fix: expired-miss share over all measured turns + cost-optimization tab labels (#36037)
* fix(ui): make the expired-miss stat row a focusable tooltip trigger

* fix: auto-router expired-miss percentage and cost-optimization tab labels

- change expired-miss percentage denominator from return-to-tier misses to
  all measured turns (same_model + first_visit + return_to_tier). when
  auto-routers flip tiers rapidly within TTL, return-to-tier turns become
  hits and disappear from the miss count; the old metric reported only the
  rare failure population. the new metric contextualizes that population as
  a share of overall coverage
- rename usage tab from 'Usage' to 'Overall'
- rename auto-router-usage tab from 'Auto-Router Usage' to 'Auto-Router'
- update component and unit tests to match new semantics
2026-08-05 22:34:55 -07:00
tin-berri
86890654c5
fix(proxy): include today's UTC bucket when a daily activity range ends at the caller's current day (#36051)
* fix(proxy): include today's UTC bucket when a daily activity range ends at the caller's current day

* fix(proxy): gate the current-UTC-day extension behind an opt-in param sent by the cost optimization dashboard

* fix(ui): label cost optimization savings dates as UTC days
2026-08-05 22:33:54 -07:00
Michael Cusack
3d275d97fe fix(router): return model and Bedrock batch fields in deployment credentials
get_deployment_credentials_with_provider dropped s3_region_name,
s3_encryption_key_id, and aws_batch_role_arn because
CredentialLiteLLMParams never declared them, and it never returned the
deployment's model, so proxy batch creation against Bedrock failed with
"LiteLLM doesn't support custom_llm_provider=bedrock for 'create_batch'"
or "AWS IAM role ARN is required" (#25104)

Provider-only file and batch calls keep their no-model contract:
get_team_provider_credentials strips the model key so a provider-scoped
request is not pinned to an arbitrary matching deployment
2026-08-05 21:49:31 -07:00
Mateo Wang
ba91768146
Merge pull request #35925 from BerriAI/litellm_tier_aware_reasoning_token_cost
fix(cost): bill reasoning tokens at the service tier output rate
2026-08-05 21:09:55 -07:00
Mateo Wang
b45b4b7300
Merge pull request #35923 from BerriAI/litellm_dated_variant_tier_pricing_sync
fix(pricing): sync flex/priority tier keys to dated OpenAI snapshot variants
2026-08-05 21:09:37 -07:00
tin-berri
7c621b3141
fix(auto-router): accept every reminder marker pair a harness emits (#36029)
* fix(auto-router): accept every reminder marker pair a harness emits

reminder_markers held one (open, close) pair, so a harness that wraps
injected context differently per agent type only got the slice of traffic
using the configured envelope stripped. Every other agent type kept hitting
the original bug: its reminder-only turn never stripped to empty, won
"newest human ask", and the harness blob got classified in place of the
real question, choosing the tier and therefore the spend.

The field now takes a list of ReminderMarkerPair, following the
KeywordTierRule pattern already in this file so each pair validates itself
and errors point at reminder_markers.N.close rather than a bare index.

Blocks from different pairs can nest, which the gap construction could not
handle: resuming the kept text at an inner block's end walks back inside
the enclosing block and leaks its remainder. Running the block ends through
a maximum collapses nested and overlapping spans without a separate merge
pass, and stays linear in block count, which a fold over a growing tuple
of merged spans would not.

A single pair's ends already increase, so the maximum is the identity and
the default path is byte-identical: verified against the shipped function
over 200k generated inputs, and every existing reminder test passes
unchanged. The prior single-pair config shape is rejected loudly at
startup and at /model/new rather than silently stripping nothing.

* docs(auto-router): document reminder_markers in the complexity router README

* chore(ui): regenerate dashboard API types for the reminder_markers shape

---------

Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
2026-08-05 21:03:36 -07:00
devin-ai-integration[bot]
0bae9708a7
fix(arize_phoenix): lowercase OTLP/gRPC auth metadata key (#34883) 2026-08-05 20:57:50 -07:00
mateo-berri
c2998dea75 fix(guardrails): guard tools write-back under scan_only_tool_results and warn on role-filtered no-op scans 2026-08-05 20:49:15 -07:00
mateo-berri
1b30b1bc20 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_tier_aware_reasoning_token_cost
# Conflicts:
#	litellm/types/utils.py
2026-08-05 20:45:34 -07:00
Mateo Wang
d26ef670e2
Merge pull request #36031 from BerriAI/litellm_b13_unscoped_files_list
fix(managed_files): return unified ids from unscoped file listing
2026-08-05 20:11:15 -07:00
mateo-berri
4b9872e7e8 fix(managed_files): return unified ids from unscoped file listing 2026-08-05 19:53:44 -07:00
Mateo Wang
b617e672e3
Merge pull request #36024 from BerriAI/litellm_anthropic_sse_keepalive
fix(proxy): send keepalive pings on anthropic messages SSE streams during upstream silence
2026-08-05 19:41:35 -07:00
Mateo Wang
ae53de36e8
Merge pull request #36019 from BerriAI/litellm_deterministic_output_file_ids
fix(managed_files): derive unified output file ids deterministically so concurrent registrations converge
2026-08-05 19:29:58 -07:00
tin-berri
ece652f6a7
feat(ui): add the auto-router usage tab to cost optimization (#35995) 2026-08-06 02:15:35 +00:00
mateo-berri
2434c1b904 Merge origin/litellm_internal_staging into litellm_deterministic_output_file_ids 2026-08-05 19:04:27 -07:00
mateo-berri
28a277e99e refactor(guardrails): drop dead tool extraction and an Any annotation, ratchet lint budgets 2026-08-05 18:56:50 -07:00
mateo-berri
3c808f9c8f Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_scan_only_tool_results 2026-08-05 18:55:43 -07:00
yuneng-jiang
f01a4fc023
Merge pull request #36026 from BerriAI/litellm_dead_locals_5_8
Some checks are pending
Unit Tests: LLM Provider Transformations / All Other Providers (push) Waiting to run
Unit Tests: MCP, Secrets, Containers & Misc / misc (push) Waiting to run
Unit Tests: Proxy Auth & Key Management / proxy-auth (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy API Endpoints / proxy-endpoints (push) Waiting to run
Unit Tests: Proxy API Endpoints / proxy-server (push) Waiting to run
Unit Tests: Proxy Infrastructure / proxy-infra (push) Waiting to run
Unit Tests: Proxy Legacy Tests / auth-and-jwt (push) Waiting to run
Unit Tests: Proxy Legacy Tests / key-generation (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-config (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-response-and-misc (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server-extras (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-token-counter (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-user-auth-and-spend (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-utils (push) Waiting to run
Unit Tests: Responses, Caching & Types / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
refactor(ui): drop dead locals and unused React state across the dashboard
2026-08-05 18:43:27 -07:00
mateo-berri
00cbebf503 fix(managed_files): source the unified input file id from the response so retrieve-time mints converge with the cost job 2026-08-05 18:33:20 -07:00
mateo-berri
d70e10982a fix(guardrails): keep tool-results-only scans off function definitions and merge scoped write-backs
Gate the OpenAI handler's tools forwarding behind scan_only_tool_results,
matching the Anthropic handler, so a tool-results-only scan can no longer
evaluate or rewrite trusted function definitions.

When a guardrail returns a replacement structured_messages list, substitute
the returned messages back into the positions their scoped originals came
from instead of installing the scoped list as the whole conversation, so
out-of-scope messages (system prompt, prior turns) survive redaction on
both the OpenAI and Anthropic paths.
2026-08-05 18:21:58 -07:00
Yuneng Jiang
51d499f138
Merge branch 'litellm_internal_staging' into litellm_dead_locals_5_8 2026-08-05 18:14:44 -07:00
Mateo Wang
60d9e6012c
Merge pull request #35999 from BerriAI/litellm_guardrails_v1_messages_tool_traffic
fix(guardrails): scan /v1/messages tool traffic
2026-08-05 18:08:47 -07:00
Mateo Wang
b9b239b0fb
Merge pull request #35980 from BerriAI/litellm_content_filter_post_mcp_call
fix(guardrails): allow litellm_content_filter to run on post_mcp_call
2026-08-05 18:08:02 -07:00
Yuneng Jiang
6e434c926b
Merge branch 'litellm_internal_staging' into litellm_dead_locals_5_8 2026-08-05 18:07:23 -07:00
Yuneng Jiang
ce5c4c1bf9
refactor(ui): drop dead locals and unused React state across the dashboard
Removes declarations nothing reads, along with the writes that fed them, so
the remaining code says what it actually does.

Where a declaration was dead but its initializer had a real effect, the call
survives and only the binding goes: spies stay installed, renders still run,
and every awaited request keeps its await. Pure computations are deleted
whole rather than left as statements that build a value and throw it away.

Dead useState pairs are removed outright instead of being elided to
const [, setX], which would keep a hook and every write to a value nothing
reads. Three chains turned out to be dead end to end and are removed with
their fetches: the tool detail team list, the Teams MCP access group load,
and the user dashboard proxy settings load.

ColumnMeta's declaration merging in columnMeta.ts and view_logs/table.tsx is
a false positive; TypeScript requires those type parameters to match the
upstream signature exactly, so both get a scoped suppression instead.
2026-08-05 18:07:17 -07:00
yuneng-jiang
624fa11d71
Merge pull request #36025 from BerriAI/litellm_dead_locals_3_tests_and_destructures
refactor(ui): drop unreferenced locals from tests and narrow destructures
2026-08-05 17:54:12 -07:00
Yuneng Jiang
888f911133
refactor(ui): drop unreferenced locals from tests and narrow destructures
Third and fourth slices of the sweep, combined because they raise nearly the
same question and neither changes what runs.

Nine test files plus one source file lose symbols whose only mention was
their own declaration. Ten more narrow a destructure to the keys actually
read, so `const { accessToken, userRole, userId: userID, premiumUser } =
useAuthorized()` keeps only `accessToken`. Aliases are preserved as written.

ignoreRestSiblings stays on so the omit idiom `const { tags, ...rest } =
metadata` is left alone; dropping `tags` there would fold it back into rest.

ToolDetail is held back again. Its unread binding only looks like a plain
deletion on the first pass, because the dead useMemo still reads it; one more
pass exposes a useQuery that issues a real request. That belongs with the
slices that get QA'd.

Part of LIT-5162.
2026-08-05 17:46:39 -07:00
yuneng-jiang
c2c795fad5
Merge pull request #35821 from BerriAI/litellm_dead_locals_2_components
refactor(ui): drop unreferenced locals from shared dashboard components
2026-08-05 17:44:38 -07:00
mateo-berri
f3bfa19ce5 fix(managed_files): resolve model_name identically across all output file registration paths so full unified ids converge 2026-08-05 17:36:22 -07:00
mateo-berri
097c03eebb fix(proxy): tolerate non-scalar sse keepalive interval config shapes 2026-08-05 17:33:09 -07:00
ryan-crabbe-berri
f2690aa60e
fix(ui): opening a project now pushes ?project= so back and deep links work (#36001)
* fix(ui): drive project detail selection from the ?project= url param

Opening a project kept selectedProjectId in useState, so the URL never changed; the detail view could not be linked or reloaded and browser Back skipped past the Projects page entirely.

Selection now lives in the ?project= query param via nuqs with history: push, matching how Teams, Organizations and Virtual Keys already work.

* fix(ui): project detail close replaces history to match the other detail pages

Adopts the close semantics from PR #36013 so browser Back after an
in-page close leaves the Projects page instead of reopening the
dismissed detail; the close test now pins the replace mode
2026-08-05 17:31:22 -07:00
Yuneng Jiang
80627c0477
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_dead_locals_2_components
# Conflicts:
#	ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx
2026-08-05 17:29:54 -07:00
yuneng-jiang
1b059f472d
Merge pull request #35819 from BerriAI/litellm_dead_locals_1_app_routes
refactor(ui): drop unreferenced locals from dashboard route components
2026-08-05 17:28:15 -07:00
ryan-crabbe-berri
1dab133d24
fix(ui): link project page keys to their virtual key detail (#36002) 2026-08-05 17:26:13 -07:00
ryan-crabbe-berri
6a2e4e6c36
fix(ui): sync projects list page index to ?page= so back and reload keep the page (#36003)
* fix(ui): sync projects list page index to ?page= so back and reload keep the page

Paging the Projects list only moved TanStack's internal page index, so the URL
never changed: reload dropped you on page 1, browser Back left the page entirely,
and the page could not be shared.

The page index now comes from a nuqs ?page= query state with history: "push".
Pagination stays controlled off that value and the footer writes the URL
directly, because TanStack resets its page index whenever the data array
identity changes; letting it own the state would clear a deep-linked page as
soon as the projects query resolved. A page outside the current row set falls
back to page 1, which covers both a hand-typed ?page=99 and a search that
narrows the list below the current page.

* fix(ui): carry page_size in the url so restored history entries show the same rows

Greptile flagged that a history entry restoring ?page=N under a changed
local page size displays different projects than it originally showed.
Page size now rides the same query string via useQueryStates, size
changes reset the page inside a single history entry, and values outside
the offered options fall back to the default
2026-08-05 17:25:42 -07:00
mateo-berri
6ca120a674 fix(proxy): coerce and validate the sse keepalive ping interval from config 2026-08-05 17:23:39 -07:00
Mateo Wang
eabcafc1df
perf(pre-commit): fetch basedpyright base counts from CI artifacts (#35970) 2026-08-05 17:21:16 -07:00