Commit graph

42318 commits

Author SHA1 Message Date
mateo-berri
ca7453bc69
chore(lint): recompute budget ceilings after merging base
The base branch ratcheted the same limits in 28a277e9, so the conflicting
files were reset to base and the ratchet re-run against the new merge-base
rather than resolved by hand. Each limit is now the base value minus this
branch's own delta, so both ratchets survive: basedpyright -653 across 48
rules, strict ruff -80, LIT -85.
2026-08-06 12:04:38 +00:00
mateo-berri
f9d48bd47c
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_reduce_any_types
# Conflicts:
#	basedpyright-code-budget.json
#	type-discipline-budget.json
2026-08-06 11:48:17 +00:00
Mateo Wang
b66d4e6965
Merge pull request #35137 from BerriAI/litellm_fix_responses_cost_router_35131
fix(proxy): fetch background responses through the router in CheckResponsesCost
2026-08-06 03:26:36 -07:00
Mateo Wang
972c0d0b04
Merge pull request #35140 from BerriAI/litellm_fix_file_content_placeholder_cost
fix(cost): stop token-pricing the placeholder input on file content calls
2026-08-06 03:04:36 -07:00
Mateo Wang
729bec69f5
Merge pull request #36014 from BerriAI/litellm_scan_only_tool_results
feat(guardrails): add scan_only_tool_results to scope unified guardrails to tool results
2026-08-06 02:17:29 -07:00
mateo-berri
14d4897e55 fix(guardrails): refuse scan_only_tool_results combos that scan nothing
Prompt Security drops tool and function rows unless check_tool_results
is on, so it now reports scan-only support from that setting and the
registry refuses the pairing at boot. Pairing scan_only_tool_results
with skip_tool_message_in_guardrail excludes every message, so guardrail
initialization now rejects that combination too.
2026-08-06 01:58:06 -07:00
Mateo Wang
a0e627f99d
Merge pull request #35468 from elinacse/bugfix/managed-batch-cost-not-logged
fix(batch): track cost for managed batches with no attributable key/u…
2026-08-06 01:51:23 -07:00
mateo-berri
28ff7f3f0b fix(guardrails): scan function-role results and dedupe returned tools
Under scan_only_tool_results, legacy OpenAI function-role messages now count as tool results, and duplicate names among guardrail-returned tools keep only the first occurrence. CustomGuardrail.structured_messages_cover_full_request lets CrowdStrike AIDR declare that its writeback already rebuilds the whole conversation, so handlers install it as-is instead of merging it into the full message list a second time and duplicating out-of-scope rows. Lint budget ceilings ratchet down to match the tree
2026-08-06 00:52:16 -07:00
mateo-berri
0b24a9ab05 merge: litellm_internal_staging into litellm_fix_file_content_placeholder_cost 2026-08-06 00:09:27 -07:00
mateo-berri
7d745521bf fix(guardrails): merge synthesized tools under scan_only_tool_results and reject role-filtered no-op combos at init 2026-08-05 23:46:40 -07:00
mateo-berri
e0c4c7cee0 Merge branch 'litellm_internal_staging' into bugfix/managed-batch-cost-not-logged 2026-08-05 23:42:18 -07:00
Devin AI
55c392bda1 merge litellm_internal_staging 2026-08-06 06:15:17 +00:00
mateo-berri
4f7d1fce3a fix(proxy): fall back to the SDK when a queued response's deployment is missing 2026-08-05 23:13:15 -07:00
Mateo Wang
0acca3e86a
Merge pull request #24548 from mpcusack-altos/fix/bedrock-batch-credential-fields
fix(router): include Bedrock batch/S3 fields and model in deployment credentials
2026-08-05 22:39:39 -07:00
tin-berri
34fc8d2ee7
fix: expired-miss share over all measured turns + cost-optimization tab labels (#36037)
* fix(ui): make the expired-miss stat row a focusable tooltip trigger

* fix: auto-router expired-miss percentage and cost-optimization tab labels

- change expired-miss percentage denominator from return-to-tier misses to
  all measured turns (same_model + first_visit + return_to_tier). when
  auto-routers flip tiers rapidly within TTL, return-to-tier turns become
  hits and disappear from the miss count; the old metric reported only the
  rare failure population. the new metric contextualizes that population as
  a share of overall coverage
- rename usage tab from 'Usage' to 'Overall'
- rename auto-router-usage tab from 'Auto-Router Usage' to 'Auto-Router'
- update component and unit tests to match new semantics
2026-08-05 22:34:55 -07:00
tin-berri
86890654c5
fix(proxy): include today's UTC bucket when a daily activity range ends at the caller's current day (#36051)
* fix(proxy): include today's UTC bucket when a daily activity range ends at the caller's current day

* fix(proxy): gate the current-UTC-day extension behind an opt-in param sent by the cost optimization dashboard

* fix(ui): label cost optimization savings dates as UTC days
2026-08-05 22:33:54 -07:00
mateo-berri
4ab7a33d2c
chore(lint): ratchet lint budgets down by what this branch fixed
Lowers the committed ceilings so the headroom shrinks by exactly what was
cleared instead of leaving stale slack for the next change to spend.

basedpyright -653 errors across 48 rules, with reportAny 29204 -> 28842
and reportExplicitAny 9227 -> 9105. Strict ruff -80 violations, led by
ANN401 -59. LIT rules -85, led by LIT001 -76.
2026-08-06 04:50:01 +00:00
Michael Cusack
3d275d97fe fix(router): return model and Bedrock batch fields in deployment credentials
get_deployment_credentials_with_provider dropped s3_region_name,
s3_encryption_key_id, and aws_batch_role_arn because
CredentialLiteLLMParams never declared them, and it never returned the
deployment's model, so proxy batch creation against Bedrock failed with
"LiteLLM doesn't support custom_llm_provider=bedrock for 'create_batch'"
or "AWS IAM role ARN is required" (#25104)

Provider-only file and batch calls keep their no-model contract:
get_team_provider_credentials strips the model key so a provider-scoped
request is not pinned to an arbitrary matching deployment
2026-08-05 21:49:31 -07:00
mateo-berri
e2cb01c87c
fix(types): correct annotations that were false about their runtime values
An adversarial review of the previous commit found annotations that
described what the code wished were true rather than what flows through.
A false annotation is worse than the Any it replaced, since it launders a
wrong assumption past the type checker.

- purview: `_resolve_user_id` claimed every request-body value was a
  Mapping, contradicting `_resolve_trusted_user_id` one method over, which
  types the same argument `Mapping[str, object]`. `_should_block` claimed
  every Graph response value was a sequence of str->str mappings and was
  not assignable from its own producer's return type.
- cato: `_CatoAnalyzeResponse.required_action` was required and
  non-nullable while the API returns null, as seven fixtures in the
  guardrail's own suite assert. `analysis_result` had the same problem.
  The streaming hook narrowed an override parameter below what
  `ProxyLogging` actually passes it.
- marketplace: `_PluginRecord.manifest_json` was `str` against a nullable
  column. Making it honest surfaced a latent crash, covered below.
- ownership: two functions took an attribute Protocol while their own
  bodies branch on `isinstance(response, dict)`, which no Protocol can
  satisfy.
- openapi generator: `paths` claimed every path-item value was an
  operation, though path items also carry `parameters`, `summary` and
  `$ref`.
- custom openapi spec: a TypedDict asserted a shape that the function
  returns raw Pydantic sub-schemas out of. Reverted to Any, which is
  imprecise but not false.

`get_marketplace` did an unguarded `json.loads` on the nullable
`manifest_json` inside an `except json.JSONDecodeError`, which cannot
catch the TypeError a NULL raises, so one NULL row 500s the endpoint. It
now skips the plugin like the file's other two read sites already do, with
a regression test that fails without the guard.

Where honesty cost precision, precision lost. `_should_block` went back to
its original signature entirely: the narrowing needed to type it turned a
fail-closed DLP control fail-open, because the TypeError it used to raise
on a malformed response reached `except Exception` and became a 400.
2026-08-06 04:37:48 +00:00
Mateo Wang
ba91768146
Merge pull request #35925 from BerriAI/litellm_tier_aware_reasoning_token_cost
fix(cost): bill reasoning tokens at the service tier output rate
2026-08-05 21:09:55 -07:00
Mateo Wang
b45b4b7300
Merge pull request #35923 from BerriAI/litellm_dated_variant_tier_pricing_sync
fix(pricing): sync flex/priority tier keys to dated OpenAI snapshot variants
2026-08-05 21:09:37 -07:00
tin-berri
7c621b3141
fix(auto-router): accept every reminder marker pair a harness emits (#36029)
* fix(auto-router): accept every reminder marker pair a harness emits

reminder_markers held one (open, close) pair, so a harness that wraps
injected context differently per agent type only got the slice of traffic
using the configured envelope stripped. Every other agent type kept hitting
the original bug: its reminder-only turn never stripped to empty, won
"newest human ask", and the harness blob got classified in place of the
real question, choosing the tier and therefore the spend.

The field now takes a list of ReminderMarkerPair, following the
KeywordTierRule pattern already in this file so each pair validates itself
and errors point at reminder_markers.N.close rather than a bare index.

Blocks from different pairs can nest, which the gap construction could not
handle: resuming the kept text at an inner block's end walks back inside
the enclosing block and leaks its remainder. Running the block ends through
a maximum collapses nested and overlapping spans without a separate merge
pass, and stays linear in block count, which a fold over a growing tuple
of merged spans would not.

A single pair's ends already increase, so the maximum is the identity and
the default path is byte-identical: verified against the shipped function
over 200k generated inputs, and every existing reminder test passes
unchanged. The prior single-pair config shape is rejected loudly at
startup and at /model/new rather than silently stripping nothing.

* docs(auto-router): document reminder_markers in the complexity router README

* chore(ui): regenerate dashboard API types for the reminder_markers shape

---------

Co-authored-by: Abhimanyu Kapur <38531241+akapur99@users.noreply.github.com>
2026-08-05 21:03:36 -07:00
devin-ai-integration[bot]
0bae9708a7
fix(arize_phoenix): lowercase OTLP/gRPC auth metadata key (#34883) 2026-08-05 20:57:50 -07:00
mateo-berri
c2998dea75 fix(guardrails): guard tools write-back under scan_only_tool_results and warn on role-filtered no-op scans 2026-08-05 20:49:15 -07:00
mateo-berri
1b30b1bc20 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_tier_aware_reasoning_token_cost
# Conflicts:
#	litellm/types/utils.py
2026-08-05 20:45:34 -07:00
mateo-berri
6b5c7f92ce
refactor(types): replace implicit and explicit Any across 11 modules
Types the values that were flowing through as Any in the highest-density
modules, using shapes the code already assumes: TypedDicts for the JSON
payloads read by literal key, Protocols for the prisma rows, existing
litellm types where they were already modeled, and `object` where a value
is only stored and forwarded.

Annotation-level only, no runtime behavior change. New annotations use
read-only views (Mapping / Sequence / tuple) rather than dict / list, so
LIT001 drops alongside the Any counts instead of trading one budget for
another. No suppressions, casts, or type guards were added.

basedpyright across the touched files: 1547 -> 856 errors, with reportAny
down 399 and reportExplicitAny down 134, and no rule increasing.
2026-08-06 03:34:06 +00:00
Mateo Wang
d26ef670e2
Merge pull request #36031 from BerriAI/litellm_b13_unscoped_files_list
fix(managed_files): return unified ids from unscoped file listing
2026-08-05 20:11:15 -07:00
mateo-berri
4b9872e7e8 fix(managed_files): return unified ids from unscoped file listing 2026-08-05 19:53:44 -07:00
Mateo Wang
b617e672e3
Merge pull request #36024 from BerriAI/litellm_anthropic_sse_keepalive
fix(proxy): send keepalive pings on anthropic messages SSE streams during upstream silence
2026-08-05 19:41:35 -07:00
Mateo Wang
ae53de36e8
Merge pull request #36019 from BerriAI/litellm_deterministic_output_file_ids
fix(managed_files): derive unified output file ids deterministically so concurrent registrations converge
2026-08-05 19:29:58 -07:00
tin-berri
ece652f6a7
feat(ui): add the auto-router usage tab to cost optimization (#35995) 2026-08-06 02:15:35 +00:00
mateo-berri
2434c1b904 Merge origin/litellm_internal_staging into litellm_deterministic_output_file_ids 2026-08-05 19:04:27 -07:00
mateo-berri
28a277e99e refactor(guardrails): drop dead tool extraction and an Any annotation, ratchet lint budgets 2026-08-05 18:56:50 -07:00
mateo-berri
3c808f9c8f Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_scan_only_tool_results 2026-08-05 18:55:43 -07:00
yuneng-jiang
f01a4fc023
Merge pull request #36026 from BerriAI/litellm_dead_locals_5_8
Some checks are pending
Unit Tests: LLM Provider Transformations / All Other Providers (push) Waiting to run
Unit Tests: MCP, Secrets, Containers & Misc / misc (push) Waiting to run
Unit Tests: Proxy Auth & Key Management / proxy-auth (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy API Endpoints / proxy-endpoints (push) Waiting to run
Unit Tests: Proxy API Endpoints / proxy-server (push) Waiting to run
Unit Tests: Proxy Legacy Tests / auth-and-jwt (push) Waiting to run
Unit Tests: Proxy Legacy Tests / key-generation (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-config (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-response-and-misc (push) Waiting to run
Unit Tests: Proxy Infrastructure / proxy-infra (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server-extras (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-token-counter (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-user-auth-and-spend (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-utils (push) Waiting to run
Unit Tests: Responses, Caching & Types / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
refactor(ui): drop dead locals and unused React state across the dashboard
2026-08-05 18:43:27 -07:00
mateo-berri
00cbebf503 fix(managed_files): source the unified input file id from the response so retrieve-time mints converge with the cost job 2026-08-05 18:33:20 -07:00
mateo-berri
d70e10982a fix(guardrails): keep tool-results-only scans off function definitions and merge scoped write-backs
Gate the OpenAI handler's tools forwarding behind scan_only_tool_results,
matching the Anthropic handler, so a tool-results-only scan can no longer
evaluate or rewrite trusted function definitions.

When a guardrail returns a replacement structured_messages list, substitute
the returned messages back into the positions their scoped originals came
from instead of installing the scoped list as the whole conversation, so
out-of-scope messages (system prompt, prior turns) survive redaction on
both the OpenAI and Anthropic paths.
2026-08-05 18:21:58 -07:00
Yuneng Jiang
51d499f138
Merge branch 'litellm_internal_staging' into litellm_dead_locals_5_8 2026-08-05 18:14:44 -07:00
Mateo Wang
60d9e6012c
Merge pull request #35999 from BerriAI/litellm_guardrails_v1_messages_tool_traffic
fix(guardrails): scan /v1/messages tool traffic
2026-08-05 18:08:47 -07:00
Mateo Wang
b9b239b0fb
Merge pull request #35980 from BerriAI/litellm_content_filter_post_mcp_call
fix(guardrails): allow litellm_content_filter to run on post_mcp_call
2026-08-05 18:08:02 -07:00
Yuneng Jiang
6e434c926b
Merge branch 'litellm_internal_staging' into litellm_dead_locals_5_8 2026-08-05 18:07:23 -07:00
Yuneng Jiang
ce5c4c1bf9
refactor(ui): drop dead locals and unused React state across the dashboard
Removes declarations nothing reads, along with the writes that fed them, so
the remaining code says what it actually does.

Where a declaration was dead but its initializer had a real effect, the call
survives and only the binding goes: spies stay installed, renders still run,
and every awaited request keeps its await. Pure computations are deleted
whole rather than left as statements that build a value and throw it away.

Dead useState pairs are removed outright instead of being elided to
const [, setX], which would keep a hook and every write to a value nothing
reads. Three chains turned out to be dead end to end and are removed with
their fetches: the tool detail team list, the Teams MCP access group load,
and the user dashboard proxy settings load.

ColumnMeta's declaration merging in columnMeta.ts and view_logs/table.tsx is
a false positive; TypeScript requires those type parameters to match the
upstream signature exactly, so both get a scoped suppression instead.
2026-08-05 18:07:17 -07:00
yuneng-jiang
624fa11d71
Merge pull request #36025 from BerriAI/litellm_dead_locals_3_tests_and_destructures
refactor(ui): drop unreferenced locals from tests and narrow destructures
2026-08-05 17:54:12 -07:00
Yuneng Jiang
888f911133
refactor(ui): drop unreferenced locals from tests and narrow destructures
Third and fourth slices of the sweep, combined because they raise nearly the
same question and neither changes what runs.

Nine test files plus one source file lose symbols whose only mention was
their own declaration. Ten more narrow a destructure to the keys actually
read, so `const { accessToken, userRole, userId: userID, premiumUser } =
useAuthorized()` keeps only `accessToken`. Aliases are preserved as written.

ignoreRestSiblings stays on so the omit idiom `const { tags, ...rest } =
metadata` is left alone; dropping `tags` there would fold it back into rest.

ToolDetail is held back again. Its unread binding only looks like a plain
deletion on the first pass, because the dead useMemo still reads it; one more
pass exposes a useQuery that issues a real request. That belongs with the
slices that get QA'd.

Part of LIT-5162.
2026-08-05 17:46:39 -07:00
yuneng-jiang
c2c795fad5
Merge pull request #35821 from BerriAI/litellm_dead_locals_2_components
refactor(ui): drop unreferenced locals from shared dashboard components
2026-08-05 17:44:38 -07:00
mateo-berri
f3bfa19ce5 fix(managed_files): resolve model_name identically across all output file registration paths so full unified ids converge 2026-08-05 17:36:22 -07:00
mateo-berri
097c03eebb fix(proxy): tolerate non-scalar sse keepalive interval config shapes 2026-08-05 17:33:09 -07:00
ryan-crabbe-berri
f2690aa60e
fix(ui): opening a project now pushes ?project= so back and deep links work (#36001)
* fix(ui): drive project detail selection from the ?project= url param

Opening a project kept selectedProjectId in useState, so the URL never changed; the detail view could not be linked or reloaded and browser Back skipped past the Projects page entirely.

Selection now lives in the ?project= query param via nuqs with history: push, matching how Teams, Organizations and Virtual Keys already work.

* fix(ui): project detail close replaces history to match the other detail pages

Adopts the close semantics from PR #36013 so browser Back after an
in-page close leaves the Projects page instead of reopening the
dismissed detail; the close test now pins the replace mode
2026-08-05 17:31:22 -07:00
Yuneng Jiang
80627c0477
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_dead_locals_2_components
# Conflicts:
#	ui/litellm-dashboard/src/components/add_model/add_auto_router_tab.tsx
2026-08-05 17:29:54 -07:00
yuneng-jiang
1b059f472d
Merge pull request #35819 from BerriAI/litellm_dead_locals_1_app_routes
refactor(ui): drop unreferenced locals from dashboard route components
2026-08-05 17:28:15 -07:00