Commit graph

1329 commits

Author SHA1 Message Date
mateo-berri
8b56bc88f9 Merge remote-tracking branch 'origin/main' into HEAD
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
LiteLLM Rust DB / db-queries (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
# Conflicts:
#	litellm/completion_extras/litellm_responses_transformation/transformation.py
#	tests/unit/integrations/test_anthropic_cache_control_hook.py
2026-11-13 23:53:50 -05:00
mika-lt
7ab37a1529
fix(count_tokens): answer 400, not 500, for a message without role or content on Anthropic and Azure AI (#51284)
* fix(count_tokens): answer 400, not 500, for a message without role or content

Anthropic, Azure AI Anthropic and Bedrock Claude counters now reject a
message missing role or content before calling the provider, with the
same 400 invalid_request_error the provider answers, instead of a 500
from the count transform.

* fix(count_tokens): count a chat message without content the way chat inference sends it, and check every Mantle count

* refactor(bedrock): leave message validation to the Bedrock count handlers

Main's Bedrock runtime and Mantle count handlers already answer 400 for a message without role or content before any AWS call, so the counter's own check was a second copy

* fix(count_tokens): name the chat message chat translation cannot send

A contentless system, developer, or tool message, or a message without a role, now answers "Message N must have a 'content' field" or "'role' field" instead of the translator's raw error text, which echoed the whole conversation back

* fix(bedrock): keep converse Claude deployments answering 400 for a chat message without content

Converse inference drops a non-assistant message without content, and for a last turn Claude then refuses the assistant prefill, so the count has no inference request to match. Invoke and Mantle deployments use Anthropic's chat translation and count the message the way their inference sends it

* fix(count_tokens): keep base counting for null-content tool and named turns on Bedrock converse

The converse check now covers only messages with just role and content, the
ones the merge base rejected. A content null message is never blamed for a
translation error, and a body whose messages all drop in translation answers
400 instead of a local estimate

* fix(count_tokens): reject only contentless user turns before counting on Bedrock converse

A system turn with content null counted on the merge base and converse
inference serves it, so it goes through chat translation again

* fix(count_tokens): keep the merge base's converse handling and leave null turns chat translation cannot send as given

* fix(count_tokens): keep a non-string message role from crashing the Anthropic count checks

* test(integration): expect the 400 for a count whose leading system message has no content

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 04:21:29 +00:00
moe-berri
752eb7c5b7
fix(opencode): stream Go Messages-protocol models through chat completions (#51715)
* test(opencode): cover Go Messages chat streaming

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): delegate Go Messages chat streaming to Anthropic

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(opencode): preserve Messages stream usage and cleanup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 22:21:52 -05:00
norkelbyte
5849906ddb
fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079)
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route

/v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with
tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which
runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends.
A body inference would reject now answers the same 400 before any provider call.

DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on
DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning
mapping inference uses, and falls back to the estimate only when that route is missing.

* perf(proxy): delete count drop paths from one copy of the options

additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear

* fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing

Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills

Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400

additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500

The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models

* fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's

A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference

A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does

The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any

* fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude

A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status

/utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none

* fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking

A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts

A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it

* fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking

A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count

* fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle

* fix(bedrock): send the thinking beta headers on Mantle counts

A Mantle count with thinking display updates or block_binding now sends the
same anthropic-beta header Mantle inference sends

* fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test

* fix(deepseek): count only the tools DeepSeek chat inference sends

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 02:29:16 +00:00
devin-ai-integration[bot]
03f14051b9
test(otel): keep proxy startup out of the per-test timeout in the excluded services matrix (#51759)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 01:05:05 +00:00
Ankit Jha
5f381ad472
fix(proxy): keep carried end-user spend when a budget reset is retried (#51143)
* fix(proxy): keep carried end-user spend when a budget reset is retried

End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.

Fixes #50898

* fix(proxy): leave never-reset tiers unguarded and test each tier's own window

A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.

* test(proxy): give the end-user ledger its own fake prisma client

* fix(proxy): pin a reset time on tiers without one before end-user pages commit

* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced

* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset

* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration

* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window

* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time

A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users

* fix(proxy): name each end-user reset window by the tier's last write

Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due

* fix(proxy): name end-user reset windows by the tier and its advance count

Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.

* fix(proxy): keep carried spend for customer teams and members on a retried reset

Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.

The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.

* test(integration): list every budget column and relation in the budget update field check

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 01:03:19 +00:00
yucheng-berri
8cf7376df6
test(s3_v2): give the slow-sink ramp test a callback limit that outlasts its burst (#51744) 2026-11-13 23:23:55 +00:00
yujonglee
6523ab920a
fix(cost-map): restore backward compatibility after the Rust loader move (#51545)
A cost map with a top-level value that is not an object made a reload answer 500
after the new table was already published, so one worker billed new prices while
the others kept the old ones. The document is now refused before anything changes,
which also lets boot fall back to the bundled map instead of failing the import.

Also restores behavior the Rust cost map move changed for SDK callers:

- a document that starts with a UTF-8 or UTF-16 byte order mark loads again
- an alias_of row that carries other fields is served as its target
- register_model(url) fetches nothing when LITELLM_OFFLINE_MODE is set
- register_model(None) is a no-op again
- the public names RESERVED_TOP_LEVEL_KEYS, FALLBACK_GENERALIZATIONS_KEY,
  RETRYABLE_FETCH_STATUS_CODES and git_blob_id import again, and so does
  utils._runtime_registered_model_cost, which downstream test code replaces

litellm_core_utils/AGENTS.md now says which removed names come back.
2026-11-14 08:12:17 +09:00
Mateo Wang
ac3277dce4
fix(dynamic_rate_limiter_v3): publish a team's share gauges for requests served while the model is idle (#51709) 2026-11-13 14:58:54 -08:00
moe-berri
62cf355d9f
fix(spend-logs): make token redaction opt-in (#51729) 2026-11-13 14:41:12 -08:00
Kerry Lu
db75214bdc
fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry (#51694)
* fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry

The reset job now evicts the customer_org_alias entries of the organizations it resets, and a
hierarchy load that began before an eviction can no longer cache what it read. The cache hands
a lease out on a miss and refuses the write when the key was evicted since, on this worker and,
with enable_redis_auth_cache, on any worker through a generation key checked in Redis.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop auth cache memory before the generation write and skip an empty alias eviction

delete cleared memory only after its new Redis round trip, so a deleted entry stayed readable on
the handling worker while Redis was slow. The reset job also broadcast an empty alias eviction for
a cascade with no customer organizations.

* fix(proxy): delete an auth cache entry and void its leases in one script on the bulk path too

The bulk delete wrote the generation and sent the DEL as two round trips, so a failed generation
write still let the DEL run and an in-flight load could re-cache what it read.

* fix(proxy): delete the auth cache entry before the generation write inside the script

A Redis at its memory limit with noeviction refused the script at its leading SET, so the DEL
never ran and a deleted entry stayed in Redis. With the DEL first the script still deletes it.

* test(proxy): mark the new reset cache tests' bindings Final and use rstest for the new Rust test

* fix(proxy): refuse a leased auth cache write for a key no generation key can guard

* test(proxy): undo the unrelated reformat of the spend ledger parity tests

* test(redis-store): mark the updated memory-only auth cache test with rstest

* chore(proxy): drop two pointer comments from the cache lease code

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 14:21:20 -08:00
Mateo Wang
c7747002e5
test(integration): measure dynamic limiter shares after saturation (#51714)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:46:06 +00:00
devin-ai-integration[bot]
49ea54ee7a
test(streaming): make unbuffered Noma secret exposure deterministic (#51707)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 21:25:53 +00:00
Kerry Lu
a920b5be6e
fix(proxy): delete a customer team's override budget when the team has no overall budget (#51703)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 13:05:19 -08:00
devin-ai-integration[bot]
45b726fdab
test(security): resolve budget request route identifiers from their own fixtures (#51659)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 19:49:35 +00:00
Kerry Lu
bad8b1dae8
fix(proxy): drop a customer override expiry sent without an override (#51667)
* fix(proxy): drop a customer override expiry sent without an override

* test(proxy): clear the override before deleting the row so the integration test leaves no budget behind

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:24:05 -08:00
Kerry Lu
6549e114cc
fix(proxy): /customer/new budgets reset on the same schedule as other budgets (#51689)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 19:22:14 +00:00
yucheng-berri
02d6fce6f7
Use strict supervision for worker replacement tests (#51590)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:18:52 -08:00
yucheng-berri
227855ac50
test(observability): give K3 the calibrated proxy budget instead of a fixed 300s (#51592)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:18:32 -08:00
Kerry Lu
6021db5bba
fix(budgets): reset 12months and other long unit spellings on their short form's schedule (#51672)
* fix(budgets): reset long unit spellings like 12months on the schedule they are charged for

* test(budgets): match the 3mo error message in the long month spelling test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:08:55 -08:00
Kerry Lu
367890f958
fix(proxy): refuse a key pinned to another customer org before registering an unknown org alias (#51674)
* fix(proxy): refuse a key pinned to another customer org before registering an unknown org alias

* test(proxy): a pinned key naming an unknown org alias is refused with 403 even when registration would fail

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 11:03:43 -08:00
devin-ai-integration[bot]
1defaa6cd1
fix(budgets): invalidate customer hierarchy caches through stored team IDs (#51654)
* fix(budgets): invalidate customer hierarchy caches through stored team IDs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: restore customer budget alerts after hierarchy schema changes (#51645)

* fix(proxy): upsert customer orgs and customer members safely (#51607)

* fix(proxy): upsert customer orgs and customer members safely

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): lock customer org snapshots during upsert

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): reject empty customer org upsert identifiers

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): reuse the customer org when its alias is already registered

A request for an alias that another worker registered a moment earlier got
alias_already_exists back from the create and failed. The registration now
treats that the same as an existing id and returns the registered customer org.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>

* test(spend): synchronize concurrent member resets before validation completes (#51655)

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(budgets): invalidate customer hierarchy caches through stored team IDs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(budgets): supply async customer team lookups in caller fixtures

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ishaan-j-berri <ishaan.j@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
2026-11-13 18:43:38 +00:00
devin-ai-integration[bot]
143b4b7161
test(spend): synchronize concurrent member resets before validation completes (#51655)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 18:23:36 +00:00
ishaan-j-berri
df8b8d1b51
feat(customer-portal): complete hierarchy management (#51521)
* feat(customer-portal): complete hierarchy management

* fix(ci): update Soup Sieve across supported Python versions

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 17:58:34 +00:00
ishaan-j-berri
ca4c6c0a6d
feat(proxy): customer budget rollup, reset on the budget row's window, named templates, team overrides, multi-window limits and pause_allow_increase (#51582)
* fix(proxy): reset customer org, team and member spend on their budget row's window

The customer tables carried their own budget_reset_at next to a budget_id whose
row already has one. Drop the column from the three tables and let the budget
table cascade zero the customer rows it governs, with a member following its
unexpired override when that override has a duration and its template
otherwise. Customer budget writes stamp budget_reset_at from budget_duration the
way /budget/new does, and the management responses report the governing budget's
window under the same budget_reset_at field.

* feat(budgets): support concurrent customer budget windows

* fix(budgets): resolve compatibility and validation checks

* feat(proxy): name budget rows, give customer teams an override and members a parent org

* fix(budgets): preserve admitted windows during accounting

* chore(schema): allow dropping the unreleased customer budget_reset_at columns

* fix(budgets): apply standard migration lock timeout

* feat(proxy): customer members belong to many teams, charged to the one the request names

* test(migrations): allow the unreleased customer budget_reset_at drop

* style(ui): prettier the customer dashboard files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(reset-job): name the batch bound on the customer budget IN filters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(budgets): preserve mixed limits and resolve CI regressions

* fix(budgets): reuse shared spend verification for customer windows

* refactor(ui): isolate customer budget group creation

Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild.

* fix(proxy): restore Field import and bound the member team lookup after the merge

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* test(proxy): send parent_team_ids in the budget window customer tests

Co-authored-by: Tin <tin@berri.ai>
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): keep rejecting another organization's template after the budget_template_ids rename

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* test(proxy): cover customer budget increase requests and the cap_policy field

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* chore(ui): regenerate API types for cap_policy and customer budget increase requests

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): hang the increase request back-reference on the customer member model

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): declare increase request list filters with Annotated Query

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tin <tin@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 16:05:35 +00:00
devin-ai-integration[bot]
4dad0bc513
fix(utils): register ollama deployment prices from the built-in catalog without calling the provider (#51561)
* test(providers): keep registration lookups in the model catalog

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(providers): format catalog registration lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 06:50:44 +00:00
Kerry Lu
172e16acde
feat(proxy): router_settings.fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team model_preferences (#51439)
* feat(proxy): fallback_on_saturation walks key and team fallbacks before a model saturates, replacing team_metadata.model_preferences

Team router_settings.fallbacks now fire on proxy-side rate limits: the pre-call
fallback walk reads router settings Key > Team > Global instead of the key's
column alone, and skips a fallback the caller may not call.

A key or team with router_settings.fallback_on_saturation: true walks that same
fallback list as soon as the requested model reaches saturation_threshold in
dynamic_rate_limiter_v3, and goes back to the requested model when no fallback
admits the request. Responses carry x-litellm-fallback-reason:
rate_limit:model_saturation_fallback.

Removes the team_metadata.model_preferences steering and the
x-litellm-model-steered-from header from #51350, which did the same job through
a second knob.

* fix(proxy): strict bool for fallback_on_saturation, drop a stale import and settle the pre-call tests

fallback_on_saturation only turns on for a real boolean true, so a string like "yes" in router_settings no longer enables it. The saturation unit test reads saturation through the limiter's cache, so it asserts the threshold was reached instead of an exact count. The response polling test now hands the pre-call path an awaitable router_settings lookup, as the pre-call fallback walk reads Key > Team settings before the first call.

* fix(proxy): pre-call fallbacks pass the budget and access-group checks a direct request does

A fallback the pre-call walk picks now has to clear every per-model budget layer and the budgets of the access
groups that grant it, and its spend is charged to those groups instead of the requested model's. At saturation a
fallback that fails for any reason counts as a miss and the request goes back to the requested model. The client
body can no longer opt itself into the saturation walk through router_settings_override, only the saved key or team
router_settings can. The integration test for a forbidden fallback waits for the preferred model's own 429 so it
cannot pass before saturation.

* refactor(proxy): one helper decides which refused fallbacks end the pre-call walk

_pre_call_with_fallbacks had grown past the strict C901 ceiling, so the two except clauses fold into one and _fallback_refusal_ends_the_walk carries the rule: at saturation every refusal is a miss, otherwise only a non-rate-limit failure or a hard per-model 429 reaches the caller.

* fix(proxy): keep the caller's request object through the pre-call fallback walk

Copying the request to drop a client fallback_on_saturation opt-in handed the hooks a different dict than the one the caller holds, so header metadata written into one never reached the other (test_add_litellm_metadata_from_request_headers). drop_client_saturation_opt_in now edits the request in place.

* fix(proxy): pre-call fallbacks keep the pricing and budget-reservation rules a direct request passes

A fallback is skipped when unpriced models are blocked and it has no pricing, when a zero-cost request would land on a priced model, and when the budget auth reserved for the requested model does not fit it (higher per-token prices, or reserved against an access group that does not grant it).

* refactor(proxy): read fallback prices through the router registry entry point

router_surface_check only lets the router surface shrink, so the per-token price lookup goes through registry().group() instead of the deprecated get_model_group_info.

* fix(proxy): a fallback's group budget check skips the groups the request already holds

The auth-time reservation sits in the same counter the plain budget read uses, so a fallback in a group the request already reserved against was refused by its own hold once the group was within one estimate of max_budget. Groups without a hold keep the plain read.

* test(dynamic_rate_limiter_v3): the throttle metric test from main uses fallback_on_saturation instead of the removed steering

A saturation handoff carries no rate_limited_by reason, so it is never counted as a throttle.

* fix(proxy): the saturation handoff only fires under the fallback walk

_pre_call_with_fallbacks arms the request before the hooks run, so routes that call the hooks directly (batches, fine-tuning, polling, images) keep admitting or refusing an opted-in key on a saturated model as before instead of surfacing the handoff 429.

* style(proxy): keep the cast-ok reason within the line length

* fix(proxy): never invent a metadata bucket when arming the saturation fallback walk, take main's limiter metric tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): arm the saturation fallback walk in the request context, refuse fallbacks for agent-owned keys

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep the saturation walk state out of the body, check the end user's allowlist on fallbacks, release a refused fallback's rate limit hold

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): only load the end user for fallback checks when a database is connected

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): grant pre-call fallbacks on the model an alias serves and release the rate limited model's hold before the walk

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): check a pre-call fallback's per-model budgets on its listed name and its served group, as auth does

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): assert the listed fallback name's budget blocks an aliased fallback, not the order of budget reads

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): release the dynamic limiter's reservation when the fallback walk leaves a model, and replay an exhausted walk with the requested model

The dynamic rate limiter admits through its own inner v3 limiter, so the walk's hold release never reached its team-share
reservation. It now exposes async_post_call_failure_hook, which refunds that reservation when the proxy refuses a request
after admission, and _release_pre_call_hold calls it next to the registered limiter.

An exhausted saturation walk replayed with the already aliased model name, so chained model_alias_map entries moved the
request on to a further alias. It now replays with the model argument the first attempt saw.

The hold-release test drives the real v3 limiter with one slot per key instead of recording hook calls.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): type the dynamic limiter's failure hook request dict for the type-discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 05:57:18 +00:00
devin-ai-integration[bot]
89632ca1be
test(observability): classify post-auth datastore spans by identity (#51554)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 05:11:21 +00:00
Kerry Lu
d30b2b9b26
feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489)
* feat(proxy): enforce customer org, team and member budgets and limits on the request path

Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget,
allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy
allowlists in alternate-model steering, and settle spend to the member, team and org rows.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): customer hierarchy survives virtual key auth without a legacy end user

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(auth): keys pinned to a customer organization may only act for that organization's members

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): keep the customer hierarchy spend maps within the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): check the key's customer org scope after the key is resolved

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the hierarchy cache key family and keep type counts flat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): resolve the customer hierarchy in the shared authorization gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop a member's cached hierarchy through the declared cache target

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(auth): keep the hierarchy reset within the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): type the requested models as a sequence

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(spend): add daily customer usage and budget resets (#51493)

* feat(spend): add daily customer usage and budget resets

* chore(spend): document customer reset query bounds

* fix(spend): chart customer queue metrics and update recovery test

* fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "feat(spend): add daily customer usage and budget resets (#51493)"

This reverts commit ccac47608f.

* fix(proxy): evict cached member hierarchies when team, org or template budgets change

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the auth cache dependency alias as a TypeAlias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reapply "feat(spend): add daily customer usage and budget resets (#51493)"

This reverts commit 7542471e17.

* fix(spend): reconcile the daily customer spend table with main

Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(pagerduty): tolerate failure metadata without customer hierarchy ids

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(accounting): key daily customer spend rows on main's hashed identity

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(pagerduty): read customer ids from failure metadata through a typed lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
2026-11-12 20:43:44 -08:00
yujonglee
9a1acee4a7
feat(redis): serve the router's session pins from a native store (#50870)
Deployment affinity and the complexity router's session and tier pins move
off the router's DualCache onto RouterSessionPinStore, backed by the native
redis-store. The read and claim Lua scripts run byte-identical to main, the
pin read rides the request's pre-call batch, and keys and values keep main's
format.
2026-11-13 03:22:28 +00:00
devin-ai-integration[bot]
ba200f5773
test(providers): align SageMaker wire assertions with request semantics (#51536)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 03:21:56 +00:00
devin-ai-integration[bot]
78db4f1662
test(security): finish request accounting before changing Vault config (#51515)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-13 01:41:05 +00:00
maciej-berri
b304dcfe94
test(image): point admin MCP schema push at /app/litellm/schema.prisma (#51446)
The --target builder image copies the repo root to /app, so after the
schema moved under litellm/ it lives at /app/litellm/schema.prisma there.
Only the runtime stage adds the /app/schema.prisma alias.


Claude-Session: https://claude.ai/code/session_01Bne5zS8tnYUDfKJYENWcYA

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-12 16:18:01 -08:00
jesus-berri
f0021859a7
feat(ui): set per-threshold budget alert recipients on virtual keys (#51061)
* feat(ui): set per-threshold budget alert recipients on virtual keys

Create Key and Edit Key get the threshold and recipients rows the team
member budget alerts already use, writing metadata.max_budget_alert_emails.
The helpers and the row editor move to key_team_helpers so the team form
and both key forms share them.

* test(ui): cover the empty metadata box path and drop a redundant payload case

* fix(ui): keep comma-separated string alert recipients stored through the API

The backend accepts a string of recipients per threshold, so Edit Key and the
team form now load them as rows instead of dropping them and deleting the
alert on the next save

* fix(ui): keep the stored key alert map on saves that do not edit its rows

Edit Key rebuilt max_budget_alert_emails from the rows on every save, so
API-written entries the rows cannot show (a null recipient list, a list with
a null, thresholds outside 1 to 100) were deleted by unrelated edits. The
map is now rebuilt only when the rows were edited and otherwise sent back
as stored

* fix(ui): load null and mixed alert recipient lists as rows and drop a typed map on Create Key

A null recipient list loads as an owner-only row and a mixed list keeps its
strings, matching what the backend sends. Create Key now ignores a
max_budget_alert_emails typed into the Metadata box, as Edit Key does, and
the hint says thresholds replace the default 80% alert

* fix(ui): keep Create Key alert rows when the Metadata box holds null

* fix(ui): move typed alert maps into rows, keep hidden alert entries, hide alert rows from team admins

A max_budget_alert_emails map typed into the Metadata box on Edit Key or Create Key now moves
into the threshold rows on blur with a toast, the same way a tags array moves to the Tags field.
Entries the rows cannot show stay in the JSON box.

When the rows are edited, the saved map keeps the stored entries the rows cannot show (thresholds
outside 1 to 100, non-numeric keys, unreadable values). Create Key keeps such typed entries too.

Team admins editing a member's key no longer see the alert rows, since that save only accepts
budget fields

* fix(ui): let the Metadata box own key alert entries the threshold rows cannot show

Edit Key now shows max_budget_alert_emails entries outside whole numbers 1 to 100 in the
Metadata JSON box and saves them from the box as the admin left it, so they can be seen,
edited and removed. Rows still own every entry they can show, and an untouched save sends
the stored map unchanged

* fix(ui): leave zero-padded alert thresholds to the Metadata box instead of duplicating a row

* fix(ui): keep loading zero-padded team member alert thresholds as rows

The canonical-key rule applies only to key alert maps, where the Metadata box
holds entries the rows cannot show. The team form has no such fallback, so it
keeps loading "080" as row 80 as it did before

* fix(ui): let the Metadata box own a non-object max_budget_alert_emails value

A string, list, number or null stored under max_budget_alert_emails stays in the
Edit Key Metadata box, is saved back as the box holds it, and is stored as typed
on Create Key. Threshold rows only own an object map, so rows the admin adds
replace a non-object value

* fix(ui): move typed alert thresholds into the rows when Edit Key saves without a blur

Submit now moves a max_budget_alert_emails map typed into the Metadata box
into the rows, as it already does for tags, and reads the edited fields at
submit time so the moved rows are saved

* refactor(ui): name the Edit Key submit handler
2026-11-12 15:39:44 -08:00
Kerry Lu
de3f0690e7
fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates (#51415)
* fix(dynamic_rate_limiter_v3): a borrower gets its full share back once the model saturates

The per-team share counters only count requests admitted while the model is
saturated, so what a team used while the model was idle is not charged
against its share once contention starts. A team below its share is still
admitted ahead of a borrower, and the model's own limits bound the total.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): idle fallback targets and WebSocket turns follow the same share accounting

A router fallback target attempted while idle no longer holds the team's
share, and a Responses WebSocket connection refreshes its share descriptor
on every turn, so idle turns are forgiven and saturated turns count.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): an idle fallback target still records its hold

Settlement finds the hold, gives the admitting model's reservation back and
charges nothing for the idle target, instead of charging the target's
tokens to the admitting model's share. The integration fixture builds its
configuration without mutating the loaded one.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 15:30:08 -08:00
Kerry Lu
7130a5b4fd
feat(dynamic_rate_limiter_v3): prometheus share, borrowed and throttle metrics per team and model (#51417)
* feat(dynamic_rate_limiter_v3): Prometheus share, borrowed and throttle metrics per team and model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): name the BLE001 suppressions on the metrics guards

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): count only final refusals, publish the served model's share, join the priority label

Review round 1: a steered alternate's refusal no longer counts as a throttle when the preferred model then admits
the request, a router fallback's settlement publishes the fallback target's own share under its model group, the
gauges carry the same priority label the 429 carries, the gauges aggregate with livemostrecent so a new window
replaces an old worker's value, the share read is skipped when no Prometheus logger is configured, the unit tests
restore the process registry, and the integration test polls for the settled totals.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(prometheus): type the gauge setter and the label factory so the type gate holds

_set_labeled_gauge takes a _MetricLike and sets the series only when it is a real Gauge, prometheus_label_factory
declares the dict[str, str | None] it returns, the priority resolver goes back to its original shape, and the
label test tolerates the now visible None values.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(prometheus): keep the label test diff to the two None-tolerant assertions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 15:28:51 -08:00
jesus-berri
f87d0a98df
feat(cost-map): load LITELLM_MODEL_COST_MAP_URL from gs:// with Google application default credentials (#51108)
* feat(cost-map): load LITELLM_MODEL_COST_MAP_URL from gs:// with Google application default credentials

* fix(cost-map): bound the gs:// token wait by the fetch timeout and accept a bare STORAGE_EMULATOR_HOST

* fix(cost-map): retry a gs:// token that is not ready and send no credentials to a storage emulator

* fix(cost-map): spend one timeout across the gs:// token wait and the object request
2026-11-12 15:22:23 -08:00
mrinal-berri
40debbb9c0
fix(ui): serve navigation payloads under the ui prefix (#50758)
* test(ui): cover prefixed static navigation payloads

* fix(ui): strip ui prefix from static navigation payloads

* fix(ui): refresh auth state after login soft navigation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Use a shared dashboard login token store

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "Use a shared dashboard login token store"

This reverts commit 7fd0dcb8eb.

* Revert "fix(ui): refresh auth state after login soft navigation"

This reverts commit 5b6ebe8918.

* Reload the dashboard after login

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): validate login return URLs and read the image export root

* fix(ui): constrain post-login navigation to the UI origin

* refactor(ui): centralize guarded document navigation

* refactor(ui): compute navigation destination once

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 14:10:10 -08:00
Kerry Lu
98dbe80ada
fix(proxy): refuse blocked and over-limit keys before reading the request body (#51428)
* fix(proxy): refuse blocked and over-limit keys before reading the request body

A client retrying 429s with multi-megabyte payloads made every worker buffer each payload in full before the refusal went out, so the rejections themselves drove memory. Auth now checks the cached key for a block or an exhausted key-level rate limit from the headers alone and answers before the body is read; admitted requests and cache misses take the unchanged path.

* fix(proxy): pick the effective key header for the early refusal and keep its failure log

The headers-only check now reads x-litellm-api-key when present, skips a missing Authorization header instead of raising, stamps the route on the cached identity so the failure is logged, and honors an exception a callback returns from the failure hook.

* fix(proxy): honor litellm_key_header_name in the early refusal

The headers-only check now reads the key from the header general_settings.litellm_key_header_name names when the request carries it, the same precedence the full auth path applies.

* fix(proxy): keep the early-rejection path inside the type gate ceilings

Read the configured key header through general_settings_view() instead of importing the untyped
general_settings global, and drop the isinstance on an exception post_call_failure_hook already types as
HTTPException.

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 14:05:05 -08:00
Nate Armstrong
4c9d72ded6
test(messages): expect rust to drop a null thinking signature on retry (#51461) 2026-11-12 22:04:53 +00:00
devin-ai-integration[bot]
5ebc284053
test(integration): point database tests at the moved litellm/proxy-extras package (#51453)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 20:27:00 +00:00
Krrish Dholakia
32425e7f9a
fix(spend-logs): mask token-shaped secrets in stored request and resp… (#48968)
* fix(spend-logs): mask token-shaped secrets in stored request and response content

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend-logs): mask Linear API keys in spend logs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(spend): assert stored spend row masks a github token inside tool output

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(spend-logs): gate token redaction behind redact_tokens_in_spend_logs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(spend-logs): redact tokens by default with explicit opt-out

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend-logs): bound token scanning and offload large payloads

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Moe Khalil <moe@berri.ai>
2026-11-12 18:45:06 +00:00
Nate Armstrong
97b7fbfc3f
test(providers): expect rust for anthropic messages after rust-only cutover (#51351)
Anthropic /v1/messages always runs on Rust since #51119, so the safeguards
contracts no longer assert those calls stay on Python, and the idle reconnect
stream row expects the Rust route header like the keepalive row after #51346.
Vertex and DeepSeek safeguards calls keep their Python assertions.
2026-11-12 10:44:13 -08:00
moe-berri
d2711ac559
feat(alerts): add management controls and opt-in daily summaries (#51418)
* feat(alerts): add alert controls and daily notification summaries

* fix(alerts): reserve daily notifications before delivery
2026-11-12 10:06:07 -08:00
devin-ai-integration[bot]
382588a365
feat(proxy): publish opted-in pass-through endpoints on the Model Hub (#43905)
* feat(proxy): publish opted-in pass-through endpoints on the Model Hub

Pass-through endpoints gain display_name and show_in_model_hub. Endpoints that opt in appear on
/public/v1/model_hub and /public/model_hub as rows in mode passthrough, named by their display name
or their path, with the route in pass_through_path. The admin UI exposes both fields on the create
and edit forms and the public hub renders passthrough rows with a route-specific curl example

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): carry pass-through methods onto hub rows and key dashboard rows by route

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep health, cost and shell quoting correct for pass-through hub rows

Pass-through rows no longer inherit a router model's health check when they share its name, the public table shows n/a instead of Free for their token costs, and the generated curl example escapes shell characters in the route

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): quote the method in the pass-through hub curl example

A configured method string is placed after curl -X inside escaped double quotes so it cannot alter the copied command

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): build pass-through hub fixtures through a helper

Keeps the dashboard lint budget for inline object arguments under its limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): audit the pass-through Model Hub surface end to end

Cover forwarding, pagination, validation, auth, updates, duplicate names,
config-file precedence, health-name isolation, repeated reads and two chaos
cells (create/delete churn, a killed proxy worker). Build the stored
pass_through_endpoints list in one shot instead of appending to the list
the config reader handed back

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep concurrent pass-through creates from overwriting each other

Creating an endpoint appends to the general_settings list the proxy already holds instead of persisting a private copy, so two admins creating at the same time both land, as on main before this branch

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): serialize pass-through endpoint creates behind a process lock

Each create copies the stored list under an asyncio lock before persisting, so simultaneous creates in one worker no longer overwrite each other and a rejected write (config-owned path) never lands in the in-memory settings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): collect paginated pass-through rows without mutating a list

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): let a pass-through with no stored timeout save from the edit form

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(ui): type the pass-through edit fixture so a null timeout compiles

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(proxy): read published pass-throughs from the in-memory settings store

The public Model Hub routes read the pass-through list this worker already
serves from proxy_config.settings instead of querying the config table on
every anonymous hub read. Also drops a mutable-ok suppression that no longer
guards a violation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): list a config file pass-through beside database rows on the hub

Main merges database pass-throughs with config file ones on other paths since
#43962, so a database write beside a config entry is accepted rather than
rejected as config-owned

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): type the monkeypatch fixture in the pass-through hub tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): validate pass-through config entries instead of splatting unknown dicts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): list model hub rows as a tagged union of model groups and pass-throughs

Health enrichment and the no-router check now match on ModelGroupRow or
PassThroughRow instead of testing pass_through_path in each route. Both hub
routes share published_hub_rows, the v1 row helper is sync again, and the
config list route goes back to main's inline merge since the hub no longer
needs configured_pass_through_endpoints

* test(ui): find the pass-through hub row by role to stay under the no-node-access budget

* fix(proxy): keep database pass-through endpoints in the settings store across config reloads

A config reload cleared the stored database row before rebuilding the
pass-through list, so without store_model_in_db the list fell back to the
config file entries alone. The Admin UI triggers that reload on page load
through /get/config/callbacks, which dropped UI-created endpoints from the
Model Hub until the next write

* refactor(proxy): make the hub row matches exhaustive with assert_never

* refactor(proxy): return the exhaustive match fallthrough and build hub test inputs in one shot

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): drop the in-process create lock from pass-through endpoints

The lock only covered one worker, and the read-modify-write on the general_settings blob is a separate storage problem tracked as tech debt. Create goes back to main's behavior.

* fix: align pass-through model hub with current lint rules

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: share worker state for pass-through hub restart coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: show pass-through curl examples in model hub inspector

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: skip invalid pass-through hub entries and await peer refresh

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
2026-11-12 17:46:38 +00:00
tin-berri
d076cee903
feat(skills): let authors edit and withdraw pending submissions (#51312) 2026-11-12 08:30:10 -08:00
Kerry Lu
2c7a36abe9
fix(rate-limit): hold router fallbacks, nested calls, WebSocket turns and oversized prompts to a borrowing team's model share (#51367) 2026-11-12 07:52:01 -08:00
Kerry Lu
c0114e55aa
feat(dynamic_rate_limiter_v3): priority_class_share_splits_equally_among_active_teams splits a priority class share among the teams active on a model (#51366)
* feat(dynamic_rate_limiter_v3): split a priority class share equally among the teams active on a model

Adds priority_reservation_settings.priority_class_share_splits_equally_among_active_teams (default false).
When on, the class weights are renormalized over the classes with a team active on the model in the
current window, each class share is divided equally among its active teams, and once the model saturates
each team is held to its own share with a 429 whose rate_limited_by header reads
priority_class_share_saturated. Active teams are one Redis sorted set per model with the window as TTL.

* fix(proxy): keep basedpyright counts flat for the priority class share setting

PriorityReservationSettings is built with model_validate instead of **value, so a new
field no longer adds an unknown-argument diagnostic, and the limiter unit test helpers
type the handler by its implementation class.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): keep team shares usable when Redis, mixed-priority keys, WebSocket turns or tiny shares get in the way

A registry failure falls back to the class pool instead of skipping the model limits, a team's
counter is keyed by model and team so keys of different priorities spend one share, Responses
WebSocket turns are held to the class share like the team model limit, a share is floored at one
request and one token, and the registry declares the rate limits key family.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): catch only Redis and validation errors when the active team registry fails

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): keys without a team keep their class present, team ids may contain the registry separator, breaker-open counts as a registry failure

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): per-class pool markers that take no team slice, collision-free model and team counter keys, local registry pruned by last-seen time

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): settle tokens only on the stash the call owns, keep a configured class named default apart from teams with no priority

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): tag team registry members and count every unenforced request against the team share

A team id can no longer spell a pool marker, since real teams are stored under a team prefix. Below saturation the team share counter is bumped through the atomic increment at the model-wide limit, so the local over-limit short-circuit never skips a request and the counter is complete when the share grows.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): list the caller under its own class and resync the socket stash share on every frame

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): keep the connection share across a registry outage and settle fallback tokens on the served model's team counter

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 06:38:07 -08:00
Kerry Lu
52d1c4cd80
feat(dynamic_rate_limiter_v3): steer flexible teams to a permitted alternate when their preferred model is saturated (#51350)
* feat(router): steer flexible teams to a permitted alternate model when their preferred model is saturated

Teams opt in with metadata.model_preferences: [{"model": "luna", "alternates": ["opus"]}].
The dynamic rate limiter v3 pre-call hook reads the saturation cache for the preferred model
and, once it is at or above saturation_threshold, rewrites the request to the first alternate
the team may call that is below the threshold. The response carries
x-litellm-model-steered-from with the preferred model. Rate limits and spend are charged to
the model that served. Teams without model_preferences are untouched.

* fix(dynamic_rate_limiter_v3): type the hook request dicts so the steering rewrite passes the lint gates

* fix(dynamic_rate_limiter_v3): authorize alternates against the key and its model budget, send the steering header on streams

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter_v3): enforce member and project allowlists on alternates, skip the steering header for non Latin-1 names

* fix(dynamic_rate_limiter_v3): check every budget layer for an alternate, fall back to the preferred model when the alternate refuses, merge the served model's guardrails

* fix(dynamic_rate_limiter_v3): never steer agent keys, type the steering unit tests

* fix(dynamic_rate_limiter_v3): hold alternates to customer grants, access group budgets, guardrails and the zero-cost waiver

An alternate now passes every check a budget fallback passes. Customer model grants are enforced, the access groups that grant the alternate are budget checked and stamped on the token and request metadata so spend charges them, a zero-cost preferred model that waived the aggregate budget checks is never steered to a priced alternate, and an alternate whose deployments add model-level guardrails the request did not run is skipped.

* fix(dynamic_rate_limiter_v3): stamp the alternate's groups on every bucket and skip alternates the request cannot safely take

The served model's access groups now land on every metadata bucket and the auth copy inside it even when the preferred model had none, so group spend is charged. An alternate is skipped when a budget reservation already holds a different group, when it does not serve the request's endpoint, when unpriced models are blocked and it has no price, or when its guardrails differ from the request's by name or settings. A failing alternate check or alternate admission leaves the request on the preferred model instead of failing it.

* fix(dynamic_rate_limiter_v3): keep a reserved budget off pricier alternates

While the request holds a budget reservation sized for the preferred model, an alternate that costs more per token is skipped, so fail-closed budget enforcement never admits a call the reservation does not cover.

* fix(dynamic_rate_limiter_v3): an unpriced alternate never takes a reserved budget

While a budget reservation is held, an alternate with no input or output price is skipped rather than treated as free.

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 09:10:23 +00:00
Kerry Lu
ece1484bc7
feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity (#51359)
* feat(rate-limit): team_model_limit_max_borrow_multiple caps how far a team borrows idle model capacity

With team_model_limit_borrows_idle_capacity_from_other_teams on, a team could
use everything up to the model's own tpm and rpm while the model was idle.
Setting team_model_limit_max_borrow_multiple in the team's metadata caps the
team at its model_tpm_limit and model_rpm_limit times that multiple, with a 429
whose rate_limited_by header reads model_per_team_max_borrow_multiple. Unset
keeps the previous behavior. Values below 1 or non-numeric are logged and
ignored.

* fix(rate-limit): reject infinite or oversized team_model_limit_max_borrow_multiple values instead of failing the rate check

* fix(team): make team_model_limit_max_borrow_multiple proxy-controlled team metadata so team admins cannot lift their own cap

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 06:55:59 +00:00