Commit graph

87 commits

Author SHA1 Message Date
joshua-berri
9e81aa3f40
feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192)
* feat(shadow-eval): let reverse jobs skip samples the baseline group served

A reverse shadow eval judges the router's response against baseline_model.
When the router itself sent the request to the baseline's model group, the
comparison only produces ties and still pays for a shadow and a judge call.

Reverse jobs can now opt in with skip_same_model. A sampled request whose
served group (router alias resolved, fallback hops respected) is the
baseline's own group is counted in a new same_model funnel stage instead of
being judged. It makes no provider calls, writes no attempt row, and leaves
max_turns and the eval budget untouched. When the served group is unknown
the sample is judged as before. The job detail reports
same_model_skipped_count even when nothing has been judged yet.

The flag defaults to false and is rejected on forward jobs, so existing
jobs and requests behave exactly as before. Adds two columns through an
additive, schema-only migration.

* test(shadow-eval): drive same-model skip tests off explicit completion signals

Judged cases now wait on the attempt write the injected prisma reports,
then await the pipeline task that made it. Skip cases drain only the tasks
the hook created, through a task factory scoped to the hook call, before
asserting that no provider call, attempt row, or spend was recorded. This
replaces a whole-loop task diff and the assertion on the number of tasks.

Adds a skip case with tier parameter overrides, which the policy still
treats as the same group, and a case showing a lost sampling roll counts as
not_sampled before same_model. Drops the class docstring claim that a same
group comparison only produces ties.

* style(tests): remove extra shadow evaluation test spacing

* fix(ui): send skip_same_model false from the shadow eval start form

The generated request type now requires skip_same_model, which broke the
dashboard typecheck and builds. The form sends false, the API default, so
jobs started from the UI behave exactly as before.

* fix(migrations): bound shadow evaluation schema lock waits

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 21:38:00 -08:00
yuneng-jiang
47a0d50c35
test: remove dead skip-marked tests and rewrite the Redis buffer drain test (#52063)
Fourteen tests carried an unconditional skip and could never run. The unit
conftest deletes DATABASE_URL and REDIS_* before every test, so the prisma
and Redis ones would fail in tests/unit even without the marker. Each
behavior they claimed is already covered by mocked unit tests or by the
integration suite against real Postgres and Redis, and the gpt-4o token
counter test patched a function removed in #10409.

The Redis buffer drain test was the one with unique value, so it is
rewritten in test_redis_update_buffer.py against the fakeredis-backed
RedisCache fixture. It fills every queue a spend flush writes, drains the
buffer, and asserts no Redis key is left behind.
2026-11-16 10:50:52 -08:00
yujonglee
1bc3190a94
feat(db): classify errors and recycle poisoned pools like Prisma (#49801)
db::Error now reports a kind that matches PrismaDBExceptionHandler's buckets:
unreachable, capacity, deadlock, serialization failure, lock timeout, read-only
after failover, stale cached plan and rejected data. is_unavailable gates
fail-open and is_requeue_safe says whether a failed write provably never
applied

Pools::recycle_after is the native counterpart of recreating the Prisma
client: after a 25006 (the writer endpoint now points at a read-only node) or
"cached plan must not change result type" (a migration changed a table under
a prepared statement), every pooled connection opened before the error is
dropped on its next acquire, so the pool reconnects and prepares fresh

Native failures reached Python as a ValueError, so allow_requests_on_db_unavailable
treated a native outage as a permanent fault and a write on a dropped connection
was requeued. The bridge now raises NativeDatabaseFailure with the kind and the
Rust answers for is_unavailable and is_requeue_safe, and the two Python gates
return them
2026-11-14 15:28:11 -08:00
yujonglee
bd27f4e34c
feat(db): emit DB service events from native operations (#49810)
crates/db gains observe(), which records one event per round trip with the
call type, table and operation a Prisma call reports. NativeDatabase forwards
those events to a Python ServiceLoggingEvents that Database.open injects, and
it calls the same ServiceTypes.DB success and failure hooks as db_span on the
proxy's event loop. A native health_check, which pings one pooled connection
without running SQL (call type health_check), is the first observed operation

The native pool is lazy, so a pool that could not connect stayed silent until
the first ported query failed on a request path. Database.open now pings it
once in the background and logs a warning on failure, without failing or
delaying boot

The crate AGENTS.md records the Prometheus and OTel names a Rust host sink must match
2026-11-14 15:28:10 -08:00
yujonglee
f6c9c70398
feat(db): add an optional native reader pool with explicit routing (#49804)
When DATABASE_URL_READ_REPLICA is set, NativeDatabase opens a reader pool
resolved from that URL, with its own token rotation. Pools exposes writer(),
read() and read_after_write(), matching RoutingPrismaWrapper and
WriterPinnedClient. read() prefers the replica and read_after_write() the
writer. Each falls back to the other side when its pool cannot connect and
skips the failed side for 30 seconds, so reads keep working on the replica
while the writer is down

The reader pool takes its share out of the reader's connection_limit the same
way the writer does, with the same hard-coded RUST_RESERVED_CONNECTIONS. While
that is 0 the replica URL is built exactly as before and no native reader pool
opens. A replica connection_limit that is not a positive integer is left for
Prisma to judge, as it always was
2026-11-14 15:28:10 -08:00
yujonglee
cc8ae16bb4
feat(db): add the SQLx pool foundation with the writer pool (#49798)
NativeDatabase.connect opens a lazy SQLx writer pool from an Endpoint that
Python resolves from the final DATABASE_URL, read the way Prisma reads it:
Unix socket host, sslmode / sslaccept / sslcert with Prisma's verification
rules (prefer + strict asks the server once whether it offers TLS),
sslidentity / sslpassword handed over as PEM, schema, options, pgbouncer mode,
pool_timeout (0 waits forever) and socket_timeout. Options SQLx cannot send
are logged instead of failing boot

ConnectionBudget splits each worker's connection_limit between Prisma and the
native pool, so the budget an operator sized never grows. The Rust share is the
hard-coded RUST_RESERVED_CONNECTIONS and no setting names Rust, SQLx or Prisma.
It is 0 until a ported query needs the pool, so Prisma's URL is exactly what it
was, no native pool opens and nothing connects through SQLx. A connection_limit
that is not a positive integer is never split and goes to Prisma as before.
Once the share is above 0 Prisma keeps at least one connection, and pools are
tagged litellm-prisma and litellm-native in pg_stat_activity. Python keeps
minting RDS IAM and Entra tokens and hands each one to the pool

statement_cache_size sets the native statement cache (Prisma's default 100) and
pgbouncer=true turns it off. sqlx-postgres 0.9.0 still names every statement when
the cache is off, so Statements runs queries unnamed in that case, like Prisma,
until a release carries transact-rs/sqlx#4337. An application_name the operator
set is used on both pools

The SQL query files are pinned to LF so Windows checkouts hash them like .sqlx,
and the crate lists .sqlx in its package include so the core sdist builds offline

crates/db/AGENTS.md tracks parity with the Python Prisma stack

The bridge now links litellm-db, so the editable install compiles its queries.
The Rust DB workflow builds that install from the checked-in .sqlx metadata,
since it runs before the migrations that the online check needs
2026-11-14 15:28:10 -08:00
quietjules
007d1c1b5e
feat(proxy): support budget_limits windows on internal users (#51763)
* feat(proxy): support internal user budget windows

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): make user budget migration repeatable and sync API types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): bound lock wait for user budget migration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): omit unset JSON windows when creating users

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): retain personal budget windows for proxy admins

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): give user budget windows their own counters and keep them on the user

User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path

* fix(proxy): keep windows on team auto keys and write all users' windows in one transaction

* fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock

An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets.

JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do

* fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock

Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses

The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits

* test(proxy): check default window resets against the reset helper instead of the wall clock

* fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes

* fix(proxy): fail an all-users bulk update whole when its transaction fails

The per-user fallback after a failed all-users transaction wrote the newest
user and then crashed on a missing user_email attribute, so the call answered
500 with one user changed. Racing window bulk updates past the 5s transaction
limit reached it. Answer 500 with no user changed instead

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 07:44:35 -05:00
joshua-berri
88873a70d4
fix(proxy): retain unwritten rollups after database disconnects (#51706)
* fix(proxy): retain unwritten rollups after database disconnects

* fix(proxy): initialize auto-router turns per client

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 13:36:12 -08:00
joshua-berri
a166574ba4
fix(sdk): stop requiring redis to import the Messages streaming path (#51685)
* fix(sdk): stop requiring redis to import the Messages streaming path

redis_update_buffer.py and dynamic_rate_limiter_v3.py imported RedisError at module
level. Both sit on the import chain that the SDK /v1/messages stream loads, so an SDK
install without redis lost every streaming success callback. Import RedisError only
where a Redis call can actually raise it.

* style(tests): space the optional redis import fixture per ruff format

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-13 11:43:53 -08:00
ishaan-j-berri
46716fad56
fix(proxy): alert on customer org, customer team, and customer member budgets (#51579)
* fix(proxy): alert on customer org, customer team, and customer member budgets

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): scope customer alert dispatch and Redis tracing

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(alerts): escape customer identities in Slack budgets

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): dispatch customer budget events through custom loggers

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): read committed customer budget spend from writer

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): retain customer budget claims for the full period

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 09:16:33 -08:00
ishaan-j-berri
ca4c6c0a6d
feat(proxy): customer budget rollup, reset on the budget row's window, named templates, team overrides, multi-window limits and pause_allow_increase (#51582)
* fix(proxy): reset customer org, team and member spend on their budget row's window

The customer tables carried their own budget_reset_at next to a budget_id whose
row already has one. Drop the column from the three tables and let the budget
table cascade zero the customer rows it governs, with a member following its
unexpired override when that override has a duration and its template
otherwise. Customer budget writes stamp budget_reset_at from budget_duration the
way /budget/new does, and the management responses report the governing budget's
window under the same budget_reset_at field.

* feat(budgets): support concurrent customer budget windows

* fix(budgets): resolve compatibility and validation checks

* feat(proxy): name budget rows, give customer teams an override and members a parent org

* fix(budgets): preserve admitted windows during accounting

* chore(schema): allow dropping the unreleased customer budget_reset_at columns

* fix(budgets): apply standard migration lock timeout

* feat(proxy): customer members belong to many teams, charged to the one the request names

* test(migrations): allow the unreleased customer budget_reset_at drop

* style(ui): prettier the customer dashboard files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(reset-job): name the batch bound on the customer budget IN filters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(budgets): preserve mixed limits and resolve CI regressions

* fix(budgets): reuse shared spend verification for customer windows

* refactor(ui): isolate customer budget group creation

Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild.

* fix(proxy): restore Field import and bound the member team lookup after the merge

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* test(proxy): send parent_team_ids in the budget window customer tests

Co-authored-by: Tin <tin@berri.ai>
Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): keep rejecting another organization's template after the budget_template_ids rename

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* test(proxy): cover customer budget increase requests and the cap_policy field

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* chore(ui): regenerate API types for cap_policy and customer budget increase requests

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): hang the increase request back-reference on the customer member model

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): declare increase request list filters with Annotated Query

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tin <tin@berri.ai>
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 16:05:35 +00:00
ishaan-j-berri
6eda17aadb
fix(proxy): enforce settled customer budgets across workers (#51568)
* fix(proxy): enforce settled customer budgets across workers

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): sync customer budget reset API types

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(proxy): reseed customer spend through shared counters

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-13 07:33:24 -08:00
Kerry Lu
d30b2b9b26
feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489)
* feat(proxy): enforce customer org, team and member budgets and limits on the request path

Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget,
allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy
allowlists in alternate-model steering, and settle spend to the member, team and org rows.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(auth): customer hierarchy survives virtual key auth without a legacy end user

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(auth): keys pinned to a customer organization may only act for that organization's members

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): keep the customer hierarchy spend maps within the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): check the key's customer org scope after the key is resolved

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the hierarchy cache key family and keep type counts flat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): resolve the customer hierarchy in the shared authorization gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drop a member's cached hierarchy through the declared cache target

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(auth): keep the hierarchy reset within the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(auth): type the requested models as a sequence

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(spend): add daily customer usage and budget resets (#51493)

* feat(spend): add daily customer usage and budget resets

* chore(spend): document customer reset query bounds

* fix(spend): chart customer queue metrics and update recovery test

* fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "feat(spend): add daily customer usage and budget resets (#51493)"

This reverts commit ccac47608f.

* fix(proxy): evict cached member hierarchies when team, org or template budgets change

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the auth cache dependency alias as a TypeAlias

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Reapply "feat(spend): add daily customer usage and budget resets (#51493)"

This reverts commit 7542471e17.

* fix(spend): reconcile the daily customer spend table with main

Keep main's LiteLLM_DailyCustomerMemberSpend shape, make model_group nullable and add the identity unique index the bulk upsert arbitrates on in a new migration, drop the duplicate model and migration, and feed both the member-header hierarchy and the key's customer usage attribution through the shared daily spend projection so neither path double-writes.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(pagerduty): tolerate failure metadata without customer hierarchy ids

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(accounting): key daily customer spend rows on main's hashed identity

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(pagerduty): read customer ids from failure metadata through a typed lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
2026-11-12 20:43:44 -08:00
Yassin Kortam
adc4c01aa9
chore: move scripts and runtime sources under litellm (#51006)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 04:19:57 +00:00
Yassin Kortam
6efbaf539c
chore: move deployment sources under litellm/deploy (#50981)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-12 04:19:56 +00:00
moe-berri
701639ea22
feat(alerts): detect daily changes in recorded usage (#50666)
* feat(alerts): detect daily changes in usage for gateway, models, teams, users and keys

* fix(alerts): drop unused suppressions

* fix(alerts): satisfy strict lint for model usage timing

* fix(alerts): skip only resources that time out or hold invalid data

* fix(alerts): log an unfinished daily sweep at info level

* fix(alerts): keep the deployment-wide run every five minutes

* fix(alerts): record alert and sweep timestamps in UTC

* fix(alerts): repair detector tests and tracing

* refactor(alerts): build the key index in a migration instead of after startup

* refactor(alerts): build the key index after startup again

A concurrent index build inside the boot migration can exceed the 600 second migrate deploy deadline on a large LiteLLM_DailyUserSpend and stop the proxy from starting. This reverts 393db6e4e0.

* fix(alerts): keep keys pending, bound the accounting wait, resume unfinished scopes

Keys waiting for their index no longer mark the day complete. Only deliveries created during the evaluated day hold back its sweep. A day that did not finish restarts the next day from where it stopped instead of a hash boundary that never rotated non-hex IDs.

* refactor(alerts): drop the pending delivery wait, alerts are best effort on flushed usage

* test(alerts): cover detector persistence and recovery

* fix(alerts): recognize Prisma statement timeouts
2026-11-11 18:23:02 -08:00
jesus-berri
d40dedf23b
feat(proxy): run spend log cleanup from a DB-backed scheduled job table (#50131)
* feat(proxy): run spend log cleanup from a DB-backed scheduled job table

Spend log retention ran from an in-process APScheduler timer that reset on
every restart and needed Redis for leader election, so pods that restart
more often than the interval never cleaned up at all.

Add LiteLLM_ScheduledJobRun, a generic table with one row per job run.
Every pod polls it, plans the next run idempotently and claims a due run
under a transaction-scoped advisory lock, so exactly one pod runs each
occurrence. Runs carry a lease, a status and a log. Intervals count from
the last finished run and survive restarts. Schedule edits cancel and
replace the pending run. The retention and size-check jobs share a lock
group so they never overlap, and the Redis pod lock is gone.

* fix(proxy): plan an earlier cron edit from now, and cover the job runner

Moving the cleanup cron earlier queued tomorrow's run because the next
occurrence was planned after the pending run being cancelled. Plan from
finished runs only.

Register LiteLLM_ScheduledJobRun as a span table, keep the advisory lock
query inside the tick's span, move the non-string cron startup test to
the schedule parser, and add unit tests for the runner and job wiring.

* test(proxy): annotate new spend log cleanup test locals as Final

* test(proxy): type the parameters of the new spend log cleanup tests

* test(proxy): leave an hour of margin when moving a cron earlier so an hour boundary cannot flake it

* fix(proxy): reject a cron that never fires instead of failing every tick

A crontab like 0 0 31 2 * parses but has no next fire time, so next_run_at raised on every tick and the job was never scheduled again. cron_schedule now returns InvalidSchedule for it, which logs and keeps the pending run like any other bad config

* refactor(proxy): drop the informational log column from LiteLLM_ScheduledJobRun

Nothing read the column back. Run outcomes stay in status, and the reasons
already go to the proxy logs, so JobResult becomes a SUCCEEDED or FAILED
literal and the cleanup returns its RunOutcome instead of a summary report
2026-11-11 18:22:02 -08:00
devin-ai-integration[bot]
c2afb07f93
fix(spend): keep free response reads and cancels off the creation spend row (#51110)
* fix(spend): track self-heal for test_with_id_security_off_reads_and_cancels_add_no_rows_and_no_replayed_turns

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend): preserve response turn rows when id security is disabled

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-11 07:02:00 +00:00
Mateo Wang
19c48218ae
test: deflake proxy metrics and shutdown drain (#51078)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 21:05:41 -08:00
joshua-berri
076ded32be
feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066)
* feat(auto_router): report a session's whole spend on /auto_router/session

Claude Code sends its background calls straight to Haiku under the same session id, so those requests never
reached the auto-router session rollup and the status line under-reported the session. Every billed spend
row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write
resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend
row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total
without a shared Redis

* fix(auto_router): name the unsupported provider when a turn routes off Anthropic

A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model,
failed strict native validation and was logged as observation_unavailable, which reads like a broken
capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their
existing reasons, and no estimate is added, so savings for these turns stay unknown

* docs(auto_router): explain when session savings stay unestimated

saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's
first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected
for that window

* fix(auto_router): say why a system-only cache prefix has no savings estimate

Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a
body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails
only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the
empty body still count it. A live check against claude-opus-5-5 showed a differential count with a
placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated
and the savings gap for Claude Code is not fixed here

* fix(auto_router): declare the session total's Redis key family and test its expiry

The session total read and wrote Redis without a service target, so its calls showed up as bare redis
spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a
hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an
active session keeps its total and an idle one restarts

* docs(auto_router): note which turns are still estimated inside the history window

Turns that took the identical baseline path, and turns whose whole cached prefix the session had already
written, are estimated even inside the first cache TTL window, so saved_spend now says so

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 17:07:11 -08:00
joshua-berri
894552205b
feat(spend): report guardrail spend separately in daily activity (#51010)
* feat(spend): report guardrail spend separately in daily activity

Model and key spend in /user/daily/activity/aggregated include the cost of
guardrails like Bedrock guardrails, and nothing in the response says how much.
A customer that bills tenants per key and per model could not get model cost
alone, so tenants that pay AWS for their own guardrails were charged for them
twice

Daily spend rows now carry a guardrail_spend column next to spend, and every
breakdown in the daily activity responses exposes it, so model cost is
spend - guardrail_spend. spend keeps its meaning, so budgets still count
guardrail cost as before

The value comes from the request's cost_breakdown.guardrail_cost, capped at
the row's spend because an allowance discount or a repeat job poll can charge
less than the guardrail cost the breakdown records. The delivery ledger
freezes it into the accounting payload next to accounting_savings, since the
stored payload does not keep cost_breakdown, and the legacy daily queue writes
the same value

The migration only adds the column with a 0 default, so rows written before it
read 0

* fix(spend): count guardrail spend on rows billed without a cost breakdown

A request blocked before the LLM call is billed for its guardrails straight from
guardrail_information, so it carries no cost_breakdown and its guardrail_spend
read 0. Both writers now price guardrail_spend from guardrail_information with
guardrail_information_cost, the same function success and failure rows use to add
guardrail cost to spend

Also fills guardrail_spend in the client usage view built from request logs,
qualifies the field description for rows recorded before it existed, and adds
the lock timeout the migration checks require

* fix(ui): avoid a nested ternary when reading guardrail entries from request logs

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 13:02:26 -08:00
devin-ai-integration[bot]
52cd9ef0e4
fix(proxy): let the OS assign Prisma engine ports (#50850)
* fix(proxy): let the OS assign Prisma engine ports

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): clear Prisma connection state after failed startup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-10 07:28:55 +00:00
JeongCheolLee
febbe38326
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls

Anthropic server-executed tools (web search, tool search, code execution)
were returned in chat completions tool_calls with srvtoolu_ ids, so
OpenAI-protocol agent clients treated them as pending work and re-invoked
the model until their recursion limit.

Move them to provider_specific_fields.server_tool_calls on both the
non-streaming and streaming paths. History replay, the /v1/responses
bridge and the /v1/messages streaming guardrail read that field, so
multi-turn reconstruction, web_search_call output items and the
guardrail's fail-closed check on server tool blocks keep working.

* fix(responses): return Claude web_fetch calls as web_search_call open_page items

* fix(responses): keep Claude server web calls without a result as hosted items

A non-streamed Claude turn that calls web_search or web_fetch next to a
client tool can return the server call before its result. Build an
in_progress web_search_call for it, so the Responses bridge no longer
returns it as a client function_call

* fix(anthropic): keep pending server calls native and hide server calls from client tool paths

Replaying a server call that has no result yet now sends it back as
server_tool_use, so Anthropic accepts the next turn. The tool permission
guardrail checks server calls again and rewrite mode removes a denied one.
The /v1/messages adapter keeps Claude's server blocks, and the Responses
bridge never returns a server call as a client function_call.

* fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls

Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls,
so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to
output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage
and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending
Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own
server_tool_use instead of running again

* fix(anthropic): let guardrails and loggers read Claude's server tool calls again

Guardrails, tool policies, tracing loggers and the Responses guardrail
translation read only client tool calls after server calls moved to
provider_specific_fields. They now read server calls first, then client
calls, through one shared helper, and the Responses guardrail treats
Claude's web_search_call items as web_search or web_fetch calls.
Non-streaming splits server calls by block type, as streaming does.

* fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing

A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy,
streamed or not. Redaction with message logging off hides server call arguments and hosted web
call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming
guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls.

* fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge

Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use.

* fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses

Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy.

* test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions

* test(guardrails): type the fake Pillar client like its Protocol

* fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool

Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or
web_search still needs that call and its thinking replayed, or Claude loses the fetch result

* fix(guardrails): scan replayed Claude server calls and hosted web calls

Request guardrails now see and rewrite server_tool_calls replayed in chat
messages. Cato and Purview scan Claude web_search and web_fetch calls on
Responses output, and a function tool call with an extra custom key is no
longer dropped from Cisco, Purview, Repello and PANW scans.

* fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks

* fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows

* test: annotate the new server tool call test locals with Final and type their parameters

* test: type the new guardrail test helpers with litellm's tool call types instead of Any

* test: narrow the remaining new test helper parameters from bare dict and object

* fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs

Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls

* fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera

* fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-06 03:54:34 -08:00
yujonglee
b1b236826a
feat(redis): serve the proxy's common Redis call with Rust stores (#50170)
* feat(redis): route the router, rate limiters and auth through the native store families

* style(redis): satisfy the python lint gates

Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(router): give the registry's default usage store a memory-only handle

RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(redis): build stores from native handles and wait for router tasks

Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 06:01:11 +00:00
joshua-berri
a12a6cd154
feat(accounting)!: make the SpendLogs ledger the default accounting path (#49872)
* fix(accounting): keep one undecodable delivery from blocking the ledger

A request whose metadata carried a 401-digit tokens_saved stored that number in
its delivery payload. The query engine could not decode it when the worker
claimed the batch, so every later delivery on the proxy stayed pending

The worker now claims the payload as text and decodes each row in Python. A row
that still cannot be decoded is parked alone while the rest of the batch
applies. The payload builder also clamps integers to the JSON-safe range, so
caller-supplied counts can no longer reach the engine at that size

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): take spend locks in one order so the ledger cannot deadlock

A ledger delivery updates many spend tables in one transaction, so it can
deadlock with any writer that locks the same rows in another order. It took the
team advisory lock after the user and team rows, while team roster writers take
it first

The delivery now takes every team advisory lock before any row, then writes
tables in one canonical order with ids sorted. It also runs under the rollup
lock_timeout, applies 500 deliveries per batch by default, and spreads retries
with jitter

The writers that could still collide with it now follow the same order. The
budget reset cascade, model allowlist rename, global spend reset, bulk user
update, org delete and the delete-by-list paths lock their rows in key order
first, and access group endpoints update teams and keys in one sorted pass

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(accounting): prove a ledger delivery waits instead of deadlocking

Real Postgres tests hold the first lock of a team roster change, an access group
update and a budget reset cascade, start a delivery, and let the other writer
finish. The delivery must wait and then charge once. Before the lock order fix
each case ends in a Postgres deadlock

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(accounting)!: make the SpendLogs ledger the default accounting path

use_spend_logs_ledger now defaults to true, so every request is charged through
LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and
Redis spend queues, which stay in place. Tests that cover those legacy writers
pin the setting off, and the rest check the queued charge the ledger stores

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(proxy): let the mocked client open a transaction for key and team deletes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(proxy): take team locks first when a model rename rewrites allowlists

A rename locked team rows before user rows while a roster change holds the team
advisory lock and a user row, so the two could deadlock. The rename now takes the
advisory lock for every team that lists the model, sorted, before any row lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): drain a backlog in one worker run, one batch per transaction

One run applied a single fresh batch, so at 500 per batch charges fell behind any
load above that rate. The worker now keeps claiming fresh batches while they come
back full, for up to five seconds, and each batch still commits on its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): queue only the charge when a request's spend log is suppressed

With logging off for a request, the ledger still queued the full row, so a failed
flush or a shutdown could leave its prompt and response in Redis. The queue now
holds only the fields the charge needs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* style(proxy): format the rename team listing query

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): remove expired delivery receipts as fast as the worker applies them

The worker now applies several batches per run but still removed one cleanup batch
of applied receipts, so the table could outgrow its retention window. Cleanup now
repeats while each pass removes a full batch, for up to five seconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(proxy): retry a model rename when a team starts listing the model mid-lock

The team advisory locks come from a snapshot, so a team that gained the old name
before its row lock was locked without its advisory lock. That attempt now commits
without writing and the next one locks the fresh set of teams.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 09:14:10 -08:00
joshua-berri
906fa7ed28
feat(accounting): add opt-in SpendLogs ledger (#49856)
* feat(accounting): add opt-in SpendLogs ledger delivery

* fix(accounting): preserve missing organization daily spend parity

* fix(accounting): preserve ledger receipts when replaying migration

* fix(accounting): preserve callback windows and tune ledger batches

* fix(accounting): preserve window charges and bound ledger statements

* test(accounting): verify budget callback registration order

* test(accounting): verify captured charges after repeated startup

* feat(accounting): store durable payloads independently of spend logs

* fix(accounting): preserve first delivery and trace ledger writes

* fix(accounting): keep SDK imports independent of proxy tracing

* fix(accounting): honor log retention during batch takeover

* test(accounting): cover retained batch alerts and invalid settings

* test(accounting): verify downgrade command against pending work

* fix(accounting): drain retries without starving fresh deliveries

* test(accounting): cover delivery projection, metrics and window key reads

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(accounting): read delivery metrics without the shared registry

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): charge a leftover $0 batch row when its claim fails

A failed batch claim buffers the retrieve's row and reports it charged. At flush the
row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back
and the charge never reached the spend tables. The flush now takes that row over in the
same transaction and keeps the delivery.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): trace the batch takeover at flush as a SpendLogs update

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client

The export client is never created, so main always writes SpendLogs to the proxy's own
database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was
set, which dropped those request logs. It now treats a flush as an export only when the
client exists, and batch claims write local rows as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): keep every request's charge when its log row is shared, evicted or rejected

- Key non-batch deliveries on a per-request delivery_id so a reused client call id is still
  charged once per request, while buffer replays stay deduplicated
- Persist the redacted api_key instead of an opaque bearer token
- Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it
- Strip NUL from payloads and skip non-text tags so projection never parks a paid request

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch

A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): keep charging a request whose tag is too long for the daily tag index

A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): store only guardrail billing fields in the accounting payload

The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(proxy): move spend payload stamping out of update_database

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(proxy): restore the modules the import-layering guard evicts

The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(proxy): keep update_database's payload stamping on its own local

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accounting): type guardrail entries and use a public key redaction helper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): keep charging requests whose end-user id is too long for an index

An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(proxy): type the spend log queue snapshot

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): leave out any projected row Postgres cannot index or store

Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): keep the guardrail fields compression savings are counted from

Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): cap projected counters so daily totals cannot overflow

Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): let a buffered batch charge take over its zero-cost receipt

A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): read a rescued batch row's spend from its accounting payload

Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): read budget windows through the delivery transaction

Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): retry a transient failure while keeping a dropped row's charge

The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(accounting): give delivery projection payloads their charged entities

project() now reads entities only from accounting_entities, so the hand-built
delivery payloads carry them like real stored payloads do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accounting): own ledger rows by their payload and key batches on the batch id

Drop the accounting_path stamp and its SpendLogs column. A buffered row that
carries an accounting payload is ledger owned, one without it is legacy, and
spend counters get a plain windows_from_ledger flag. Window seeding tells
applied ledger rows apart by their delivery row.

A ledger batch is plain transaction A keyed by the batch id, so a repeated
completed retrieve conflicts and charges nothing. A $0 retrieve records no
delivery, a batch a legacy retrieve already charged is not charged again, and
the legacy batch path is unchanged from main apart from refusing a batch the
ledger already owns. Batch takeover and its delta bookkeeping are removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(accounting): drop a projection test duplicated while rebasing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accounting): drop previous_payload now that deliveries are never replaced

Batch takeover was the only writer of previous_payload, so the worker applies
each delivery's payload as stored and the column leaves the migration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accounting): drop the ledger downgrade checklist script

The use_spend_logs_ledger flag is the rollback path, so a separate downgrade
gate is no longer needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): name the window seed reads so their Prisma span resolves

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accounting): remove the unused accounting event types and protocols

types.py, protocols.py and measurements.py were left from the old event
scaffolding. Nothing outside the package imports them, so they go with their
tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): park only deliveries whose own data fails

A delivery that keeps failing on a database outage, lock timeout or deadlock
stays pending and retries with capped backoff, so a recovered database applies
it. Only a data error or an invalid payload parks the isolated row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): isolate a spend log row the insert statement rejects

The raw SpendLogs insert validates columns and JSON before Postgres sees the
row, so a row that fails there is now bisected out and dropped alone while its
charge is kept, instead of failing and replaying the whole flush.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): carry a job's model window on every completed poll in ledger mode

The ledger keeps the poll whose delivery claims the job, which need not be the
poll that won the limiter's in-memory charge. Every completed poll now carries
the job's window, so the delivery that lands always has it, while the live
model budget is still charged once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accounting): leave model-budget windows on the existing window writer

Model-budget windows are written by the model budget limiter rather than
derived from a request's spend, so they stay on main's window queue like live
budget counters do. This drops window capture from the accounting payload,
the deferred limiter registration and the job-window capture, and the ledger
keeps owning key and team budget windows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): keep a delivery pending when the database is unreachable

Prisma reports a can't reach database server outage as a DataError, so the
worker now checks for an unavailable database before treating a data error
as the delivery's own fault.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): isolate only rows the raw insert cannot encode

Catching every ValueError also swallowed unexpected insert errors that main
re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column
or non-JSON value, and only that is isolated as a rejected row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(accounting): store zero savings when caller data makes them uncomputable

A huge caller-supplied tokens_saved overflowed the savings math inside
build_accounting_payload, which dropped the request's charge in ledger mode.
Savings that cannot be computed are now recorded as zero and the charge lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 17:49:15 -08:00
Kerry Lu
b080c59e78
fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows (#50132)
* fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows

Postgres refuses every new connection with SQLSTATE 57P03 while it starts up, shuts down or replays WAL, and prisma surfaces that as a base DataError with no code. The spend-log writer took it for a poison row, bisected the batch down to single rows and dropped each one. Spend logs now retry and requeue on 57P03 and on server-terminated sessions (57P01, 57P02, 57P05), tool and model usage rollups requeue when the write provably never reached the server, the tool registry requeues its rows, and auth answers 503 instead of 401 while the auth database restarts.

Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>

* fix(proxy): requeue tool discovery rows only when the write never reached Postgres, keep the auth fallback to 57P03

A terminated session (57P01, 57P02, 57P05) may have committed the tool upsert, and the upsert adds call_count on conflict, so replaying it counts the calls twice. The non-Redis tool discovery flush now uses the same never-sent rule as the rollups, which also stops a permanently rejected row (54000) from coming back every tick. One terminated session does not prove the database is down, so it no longer opens the allow_requests_on_db_unavailable fallback, it still answers 503

* fix(proxy): a full server (53300, P2024) also never took the write, so counters and tool rows are sent again

Postgres refuses the connection before any statement runs when it is out of slots, and a prisma pool timeout never reaches the server either, so those failures are as safe to replay as 57P03 and P1001

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
2026-11-04 17:39:30 -08:00
yuneng-jiang
8fcb016dc6
test: make three flaky unit tests deterministic (#50114)
* test: make four flaky unit tests deterministic

The skill archive fixtures stamped the current time into zip bytes used as parametrize ids, so xdist workers collected different ids and errored the whole shard. They now use a fixed timestamp.

The usage-based v2 override test pinned no clock, so a minute boundary between rpm checks reset the counter. It now pins get_utc_datetime the way #49505 does.

The stalled-burst DB lookup test asserted a 2s wall-clock bound. It now asserts every lookup was waiting at once, which fails if lookups are serialized.

The long tool_search placement test asserted a 1s wall-clock bound. It keeps the placement assertion and drops the bound, as #49505 did for the json accumulator

* test(skills): drop the archive fixture timestamp change now that #50118 removed its test
2026-11-04 14:38:44 -08:00
tessellark
8ab0321816
fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed (#49824)
* fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed

A custom video provider can report a video's cost on the status poll that
sees it completed. Every poll of a video shared the video id as its spend
log request_id, so the paid poll's row was dropped as a duplicate and every
later poll of the finished video charged the cost again.

Video status rows now carry their own `_video_cost` request_id, and a poll
that reports a cost goes through the same settle-once claim batches use, so
the first poll that sees the video completed charges it and writes the row,
and every later poll charges nothing.

* fix(proxy): charge a video's provider-reported cost once on model, router and session budgets too

A video status poll gets the _video_cost row suffix only when it reports a cost, so a $0 poll keeps the bare video id and adds no row of its own, and the spend log writer claims a row by that suffix.

The batch charge-once marker now covers videos, keyed on the video id under {spend_key}:video:{id}, with the batch marker key unchanged. Model and router budgets use it, and per-session budgets now charge a repolled batch or video once and ignore a cost that is not a finite number

* fix(proxy): charge a finished video's own cost once, keyed by the id the client polled

Guardrail cost on a running poll no longer takes the charge-once claim, only a completed poll charges, the session marker TTL refreshes like the others, and two deployments whose provider ids match each keep their own charge.

* fix(proxy): charge repeat poll guardrails, key video charges by canonical id and survive redaction

* fix(proxy): keep video status polls in-process for the collector and give guardrail-only job polls an unclaimable row id

* fix(proxy): keep repeat batch polls free and group guardrail-only batch rows under their batch's trace

* fix(proxy): settle a finished job whose own cost is below its poll's guardrail cost under the job's charge row

* test(proxy): count charged_poll_cost as covered through the finished job guardrail poll tests

* fix(proxy): settle a finished batch that cost nothing itself on its first priced retrieve

* fix(proxy): charge no budget for a batch retrieve before its cost is final, and read the retrieved batch from before redaction

* fix(proxy): charge sessions a poll's guardrail-inclusive cost, count a job cost row's guardrail run, and keep batch_cost_is_final importable from litellm.batches

* fix(proxy): claim a job and charge the session budget in one Redis call

The session limiter claimed the job marker and then raised spend in a second Redis call. A failure between the two left the job claimed while its charge sat only in local memory, so later polls skipped it. One script now does both, and a refused script claims and charges locally

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-04 20:22:06 +00:00
joshua-berri
02b8da2f16
refactor: replace accounting event scaffolding with SpendLogs projection (#49843)
* refactor(accounting): replace event scaffolding with SpendLogs projection

* test: preserve spend timestamps without removed capture callback

* fix: guard accounting event removal across schema update paths

* refactor(accounting): retain schema and prepare durable payloads

* fix(accounting): preserve dates in serialized payloads

* fix(accounting): retain schema push compatibility

* fix(accounting): retain span registration for the legacy schema

* test(accounting): remove obsolete capture acknowledgement test

* fix(accounting): date every daily projection row the way the model rollup does

A space-separated startTime kept the whole timestamp as the daily key, while the model usage rollup took the calendar date. Every daily row now uses the same date helper

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(ui): regenerate dashboard API types after rebasing on main

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* refactor(accounting): project only from the stored accounting payload

project() now reads entities only from the payload's accounting_entities, so
the raw SpendLogs fallback branches are gone. build_accounting_payload() builds
the metadata once instead of dumping it to JSON and parsing it back. Guardrail
usage dates use a tolerant UTC parse, so a malformed startTime drops only the
guardrail rows instead of failing the whole projection.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 10:37:07 -08:00
Kerry Lu
2d0c720425
fix(proxy): cancel the Admin UI's raw spend-log reads in Postgres after 30s (#49965)
* fix(proxy): cancel the Admin UI's raw spend-log reads in Postgres after 30s

Prisma's HTTP client gives up on a statement after 30s, so the Logs, Caching,
Cost Optimization and spend report pages already fail at 30s, but Postgres kept
running the scan. Every raw LiteLLM_SpendLogs read behind those pages now runs
under SET LOCAL statement_timeout = 30000 through one shared helper, which
/team/spend/by_user also uses instead of its own copy

* fix(proxy): keep the spend report's 503 and time out the session totals read

The grouped Logs page's session totals query now runs under the same
statement timeout, /global/spend/report re-raises the helper's 503 instead
of turning it into a 400, and the hand-written prisma fakes in the endpoint
tests expose read_db so every guarded route is exercised

* fix(proxy): keep the Logs page when Postgres cancels its session totals read

A cancelled session totals read degrades like any other failed enrichment
read instead of failing the whole page. The span-name test treats the shared
statement-timeout helper as a forwarder, and the spend report test no longer
depends on a license in the environment

* test(proxy): /global/spend/report keeps the 503 hint when Postgres cancels the read

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-03 16:07:00 -08:00
Kerry Lu
391c8013c9
fix(proxy): cancel /team/spend/by_user in Postgres after 30s (#49922)
* fix(proxy): let the Admin UI cancel /team/spend/by_user at 60s with an opt-in statement timeout

GET /team/spend/by_user takes an optional timeout_seconds (1 to 600). When set, the raw
LiteLLM_SpendLogs aggregation runs under SET LOCAL statement_timeout and Postgres cancels it,
and the route returns 503 instead of letting the query run for hours. Callers that omit the
param are unchanged. The Admin UI sends timeout_seconds=60 and the Spend Per User Within Team
card shows the server's error instead of an empty table

* fix(proxy): cancel /team/spend/by_user in Postgres after 30s

Prisma's HTTP client already gives up on the statement at 30s, so the
query now runs under SET LOCAL statement_timeout = 30000 for every caller
and the opt-in timeout_seconds param is gone. The Admin UI card shows the
request error instead of the empty-state text.

* fix(proxy): keep /team/spend/by_user on the read replica inside its timeout transaction

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-03 14:03:28 -08:00
joshua-berri
be4f3d36ce
feat(accounting): capture request usage through the durable meter (#49133)
* feat(accounting): capture request usage through the durable meter

* fix(accounting): preserve native Google usage compatibility

* fix(accounting): preserve prepared usage observation on failure

* fix(accounting): exclude sync acknowledgement policy rejections from capture

* test(accounting): verify capture races and failure isolation

* fix(accounting): preserve request support with sync acknowledgements

* fix(accounting): normalize usage metadata before queue admission

* fix(accounting): preserve prior attempt links on successful fallback

* fix(accounting): preserve attempt evidence through retries and failures

* fix(accounting): depend on prior receipts without resubmitting usage

* fix(accounting): preserve pricing inputs and resolved capture attribution

* fix(accounting): persist batch usage within capture bounds

* test(accounting): cover batch dependency failure and backpressure

* fix(accounting): isolate batch persistence and preserve trusted attribution

* fix(accounting): share byte admission across batch producers and writes

* fix(accounting): reserve batch snapshot reads before admission

* fix(accounting): compact repeated batch observations without losing requests

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-10-31 07:32:25 -07:00
joshua-berri
02a6a96dec
feat(accounting): add bounded transactional usage persistence (#49190)
* feat(accounting): add bounded transactional usage persistence

* fix(accounting): preserve usage flush and shutdown on Python 3.10

* test(accounting): cover batch rejection and partial shutdown recovery

* fix(accounting): batch durable usage on a five-second interval

* fix(accounting): drain queued full batches without timer delays

* fix(accounting): isolate invalid producer submissions within usage batches

* fix(accounting): await commit prerequisites within bounded usage admission

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-10-30 21:08:43 -07:00
yujonglee
2d79d5dba7
refactor(proxy): give database shutdown one lifespan owner (#48506)
* refactor(proxy): give database shutdown one lifespan owner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(proxy): log database cleanup errors and bound shutdown waits

Database.aclose now logs a failure in either cleanup step instead of raising
out of the lifespan exit, which matches how main handled stop errors and keeps
setup_prisma_client from raising out of cleanup after a failed connect

Shutdown waits for an in-flight reconnect for a fixed five seconds instead of
the reconnect timeout plus grace, and joins engine wait threads with a fixed
limit instead of indefinitely. The threads are daemon threads, so a thread
still alive after the limit cannot keep the process from exiting

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 22:57:53 +00:00
quietjules
df47ec0171
fix(spend): keep a NaN or infinite cost from wedging daily spend writes (#48829)
* fix(spend): keep a NaN or infinite cost from wedging daily spend writes

A non-finite provider-reported cost now counts as not reported, so the token price bills the request. Any other non-finite cost is logged and recorded as 0, and spend log JSON writes NaN and Infinity as null so Postgres accepts the row.

* fix(spend): price non-finite provider costs on every route and drain parked NaN batches

Perplexity, OpenRouter pass-through, and Responses streaming now treat a NaN or infinite reported cost as unreported, so the token price bills the request. A daily spend batch parked in the Redis buffer by an older version with a NaN spend now drains as 0 instead of failing every flush. The cost header never carries NaN or Infinity, and a set holding a non-finite float no longer breaks spend log JSON

* fix(spend): record non-finite savings as 0 and warn when a parked NaN batch drains

An overflowing price on a compressed request made compression savings infinite, which still wedged the daily spend batch. Every savings column now records a non-finite value as 0 with a warning. The Redis drain warns when it reads a NaN an older version parked, since the spend summed into it is lost from daily spend. The writer test now runs the real payload builder and spend log insert

* fix(spend): keep non-finite costs out of headers, responses, and admin JSON

Writing NaN as null is now opt-in for spend log columns, the Redis buffer, and
streamed frames, so admin writes and logging integrations keep their output.
The cost calculator records an overflowing total as 0 with a warning, cost
headers leave out non-finite values, a non-streaming NaN usage.cost is dropped
instead of failing the response, and a parked NaN spend log row reads back as 0.

* fix(spend): keep caller-set call ids from splitting the non-finite cost warning

The overflow warning strips newlines from litellm_call_id, and neither
non-finite cost warning prints the raw value, which said nothing beyond
"non-finite"

* fix(spend): drop non-finite Responses and OpenRouter costs and the breakdown of an unpriceable request

* fix(types): drop a usage cost breakdown that holds a non-finite number

* fix(streaming): keep a dropped non-finite final usage cost from falling back to an earlier chunk's cost

* fix(spend): keep the dropped-cost flag out of reach of provider usage keys

A provider usage key named dropped_non_finite_cost shadowed the Usage method on the instance, so the stream builder raised and dropped the request's spend. The builder now reads the flag through the class, and Usage sets the flag after copying provider keys, so a _dropped_non_finite_cost key cannot set it either.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-30 08:27:35 -07:00
devin-ai-integration[bot]
b8283d8b61
feat(usage): show subscription-covered providers and account fees separately (#44772)
* feat(usage): show subscription-covered providers and account fees separately

* fix(usage): write only the named fee columns and sync the root schema

The fee update now writes just the columns the request names, so two admins changing
different fields at once both land, and the dashboard edit form sends only what changed.
The root schema.prisma carries the new table like the two copies, and a billing period
anchored in the calendar's last month ends on December 31, 9999 instead of failing.

* fix(usage): register the subscription table span and keep provider logic under llms

* fix(subscription_accounts): register the fee routes as management routes and reject an empty account id

* fix(usage): run the new tests in CI and close subscription card review gaps

Adds the subscription test file to its unit shard, sets the migration lock timeout, rejects infinite fees, keeps the card from rewriting login files, lists each deployment once without aliases, shows the provider badge on entity views, and asks before removing a fee

* fix(usage): resolve named-credential ChatGPT accounts and keep admin viewers read-only on the fee card

* fix(usage): count a subscription fee only for accounts behind a current deployment

Usage listed a fee row for an account no deployment uses while another ChatGPT
deployment existed and dropped it when none did. It now lists only accounts behind a
current deployment in both cases. Adds a real proxy and Postgres integration test for
the fee API and for ChatGPT Responses rows on a deployment's own login, and names the
card's multi-component test with the integration suffix.

* fix(usage): keep a stored subscription fee listed after its account's deployments are removed

The usage route lists every account behind a subscription deployment plus every account carrying a stored fee, and reads the fees on every proxy, so a fee stays visible, counted, and editable for past ranges after its last deployment goes. The integration test now polls every day the calls spanned, so a UTC midnight crossing no longer fails it.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-30 03:58:36 -07:00
devin-ai-integration[bot]
a0a8ca97d9
fix(cost): finalize synchronous response cost once for header, spend log and rollups (#48993)
* fix(cost): finalize synchronous response cost before accounting callbacks

* fix(cost): keep serving and billing when cost finalization fails

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): read the finalized cost only off real logging objects

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(guardrails): follow the renamed success dispatch and deferred sync callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): keep cache hits free and their headers finalized when logging is deferred

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): preserve completed synchronous charges after hook failures

* fix(cost): retain completed usage across cancellation and log suppression

* fix(cost): reconcile completed failures and retain cached usage metadata

* fix(cost): retain finalized request tags in failure budget updates

* fix(cost): honor disabled spend tracking for completed failures

* fix(cost): preserve pass-through accounting across cancellation

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 11:46:49 -07:00
Nate Armstrong
ead221995e
refactor(router): move router callers onto the entry points and invert the proxy imports (#48847)
Code outside the router now reaches it only through the entry points
(registry, settings, health, select, apply, observe, register_plugin, retire)
or the endpoint methods; the outside-member allowlist shrinks from 139 to 45.
litellm/router.py no longer imports litellm.proxy: it asks the installed
RouterHost (litellm/router_host.py) for vertex pass-through credentials,
member auto-router authorization and auto-router prompt compression, and the
proxy installs its host on import. Call sites where a model name can be
missing keep the old answer instead of reaching an entry point with None.
2026-10-29 17:26:33 +00:00
Kerry Lu
4bc157e281
feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations (#48961)
* feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations

* feat(proxy): roll back each migration of the dry run on its own and baseline-check databases without a ledger

* test(proxy): cover the dry run CLI and PrismaManager wrapper without a database

* fix(proxy): refuse the no-ledger dry run for a custom migration folder like boot does, and probe the rolled-back indexes directly

* ci(codecov): stop carrying forward the roi-postgres and redis-compat flags

Their carried-forward sessions were measured against older proxy_cli.py revisions (633 and 689 line maps against 710 today) and mark docstring lines of the merged file as missed, which failed codecov/patch on this PR at 70% while every unit shard reports the lines hit

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 09:39:26 -07:00
Mateo Wang
c489aa7117
feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708)
* fix(proxy): keep every startup line JSON when json_logs is on

Boot-time messages the proxy printed straight to stdout (banner, feedback
box, model list, cache and callback notices, policy table, migration
notices, CLI launch notes) and the first records of uvicorn- or
granian-spawned workers came out as plain text while json_logs was on,
breaking log pipelines that parse every line as JSON.

- print_startup_message() prints unchanged in plain mode and, with
  json_logs on, emits one INFO record on the proxy logger (ANSI
  stripped) that skips the logger and handler level checks like the
  print it replaces; show_banner() routes to it in JSON mode only
- the CLI exports JSON_LOGS=true when the config sets json_logs so
  spawned workers log JSON from their first record
- the --enforce_prisma_migration_check deprecation notice prints after
  the config is read so a config-only json_logs covers it

Resolves LIT-1624

* fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize()

Since #46803 the `lite` shell calls show_banner() against an uninitialized
litellm package, and reading litellm.json_logs inside it initialized the whole
SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR
removed on small VMs). show_banner() is back to its main version; the proxy's
initialize() now emits the JSON banner record itself and calls show_banner()
only in plain mode. The banner unit tests move to the initialize() call site.

* fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger

uvicorn applies --log_config after proxy_cli has switched JSON logging on and before
initialize() switches it on again through load_config(). A logging config that names
"LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where
initialize() emits the banner and the feedback box, so at one worker both went nowhere;
two-worker children import litellm after uvicorn's logging setup and were not affected.
print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON
handler uses, when the proxy logger has no handler of its own.

Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the
banner and the feedback box as plaintext, e2303a3925 prints neither, this head prints both as
one JSON record each. New unit test fails on e2303a3925; new integration cell
test_json_logs_with_operator_log_config_keeps_every_line_json passes at one and two workers.

* fix(proxy): keep a long startup listing whole in its JSON record

print_startup_message ran its record through the proxy logger's filters, so
in JSON mode the required-migration SQL list and the policy list were
middle-truncated at MAX_STRING_LENGTH_STDOUT_LOG (4096 by default) with the
litellm_truncated marker, where the prints they replace showed them whole.
That cap bounds echoed request payloads; a startup message is operator text,
so the record now gets the uncapped processing of a DEBUG line: secret
redaction and base64 collapse still apply, the text cap does not.

Red on 41f92b745a, green here:
- tests/unit/test_logging.py: a listing longer than the cap comes out whole;
  a credential in a startup message is still redacted
- tests/integration/configuration/test_json_logs_startup.py: a 60-policy
  config boots with json_logs on and the policy listing record (9762 chars)
  carries every policy with no truncation marker (4096 chars, marker, on
  the unfixed tree)

* fix(proxy): keep the --enforce_prisma_migration_check notice on --health and --test

This PR moved the deprecation notice after the config is read, so that a
json_logs set only in the config applies to it; --health and --test return
before that point and stopped printing it, a visible change main did not
have. The notice is now also emitted before those two returns, through the
same helper, so a deployment running `litellm --health
--enforce_prisma_migration_check` still learns the flag is dead, as on main.
A normal server start emits it once, after the config, as in the previous
commit.

Red on 2d7a2d6844, green here: tests/unit/proxy/test_proxy_cli.py asserts
the notice on --health and on --test.

* fix(proxy): keep the JSON startup records on stdout under an operator handler on the proxy logger

An operator --log_config that gives "LiteLLM Proxy" a handler of its own, a NullHandler
that silences it or a plain WARNING-level handler on stderr, is in force in the window
where the banner, the feedback box and the cache notice are emitted: uvicorn applies it
after the CLI process switched JSON on and before initialize() re-installs the JSON
handler through load_config(). 41f92b745a sent the startup records through the proxy
logger's handlers and fell back to a JSON stdout handler only when there were none, so
under such a config the records landed in the operator's handler, as plaintext on stderr,
or vanished, where the prints they replace reached stdout whatever the logging config
said.

print_startup_message() now writes every startup record through the proxy's own JSON
handler on stdout, filters included, and never through the proxy logger's handlers; the
operator's handler gets nothing from a startup message, as it got nothing from the print.
The normal JSON handler is the same class, formatter and filters, so the records a
pipeline reads are unchanged when no operator config is in play.

Tests: tests/unit/test_logging.py gets the two handler shapes (red on f6f5957d16: the
record went to stderr as plaintext or nowhere); tests/integration/configuration/
test_json_logs_startup.py gets a silencing operator --log_config cell at 1 and 2 workers
(red on f6f5957d16: banner, feedback box and cache notice missing; green here).

* fix(proxy): print the --enforce_prisma_migration_check notice when the config fails to load

f6f5957d16 printed the deprecation notice after the config is read, so a json_logs set
only in the config makes it a JSON record. A boot whose config file is missing or does
not parse then exited before that point and never printed it, where main printed it
before the error (10-20 gauntlet finding).

The notice is printed where main printed it, unless a config file can still switch
json_logs on: then it is held until the config is read and printed right after the
json_logs switch, and a click close callback prints it on the way out if the boot fails
first. functools.cache keeps it to one print either way.

Tests: tests/unit/proxy/test_proxy_cli.py covers a failed config read (red on
75043aed83), a plain config printing it once and a config-only json_logs making it one
JSON record; tests/integration/configuration/test_json_logs_startup.py adds the JSON cell
and a missing and an unparseable config cell (red on 75043aed83, green here), through
refused_boot_log, which now takes extra CLI arguments.

* feat(proxy): add suppress_startup_output to turn off startup prints

* fix(proxy): print startup messages once and without color codes off a terminal

* test(policy-engine): check the policy table keeps colors on a terminal and drops them on a pipe

* fix(proxy): keep startup warnings when suppress_startup_output is on and apply it from CONFIG_FILE_PATH and before other settings

Warnings that name something the operator must fix now print through print_startup_warning, which
skips the suppress gate. The config key is read with the CLI's json_logs handling for --config and
CONFIG_FILE_PATH, applied before the rest of litellm_settings in load_config, and can only turn the
setting on. The Prometheus config panel obeys the setting, and startup prints use the plain log
format's color rule.

* fix(proxy): read CONFIG_FILE_PATH in the CLI only when workers would load it

Workers load CONFIG_FILE_PATH only when it names an existing YAML file, so the CLI's early read of
json_logs and suppress_startup_output skips any other value instead of failing a boot that used to start.

* fix(proxy): read only suppress_startup_output from CONFIG_FILE_PATH in the CLI, without secrets

The CLI now reads CONFIG_FILE_PATH as a plain file with its includes, skips it when a config bucket is set,
and applies only suppress_startup_output, also when --config is passed. json_logs there stays with the
workers as before. Fatal boot errors drop their color codes off a terminal like the startup prints.

* fix(logging): emit startup warnings as WARNING records in JSON mode

* fix(proxy): keep one JSON record per search tool in the startup listing

* fix(proxy): keep booting with an ASCII stdout or an empty litellm_settings

Startup messages now go through click.echo, which writes UTF-8 when stdout is ASCII, as the banner did
before. An empty litellm_settings section reads as no settings in the CLI, as the workers already read it

* test(proxy): boot the CLI tests on the real ProxyConfig config readers

* test(proxy): cover the startup warnings that print with startup output suppressed

* test(proxy): boot the deprecated resolver flag test on the real ProxyConfig

* test(proxy): drop the startup output contract's narrating docstrings

* test(proxy): cover the pgbouncer start failure message off a terminal

---------

Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-29 03:47:04 -07:00
devin-ai-integration[bot]
a2ef18b257
test(unit): keep unix socket paths in unit tests under the macOS AF_UNIX limit (#45040)
* test(proxy): keep the spend sidecar socket path under the macOS AF_UNIX limit

* test(unit): share the short socket fixtures across unit tests

Move socket_dir and socket_path to tests/unit/conftest.py so the datadog
redis test's absent socket fails for the right reason on a Mac, rename
the pgbouncer helper's directory argument, and record the rule in
tests/unit/AGENTS.md

* test(unit): fall back to /tmp when TMPDIR leaves no room for a socket path

socket_dir now uses /tmp when TMPDIR plus the longest socket suffix
would pass macOS's sun_path limit, and its teardown ignores errors so a
failed test reports only its own failure

* test(unit): keep socket fixtures inside tmp_path behind a short link

socket_dir now links a short temp directory to tmp_path, so artifacts keep
pytest's retention and a killed run leaks only an empty link. The fallback
to /tmp counts the temp directory in bytes, and socket_tempdir points
tempfile at it for the pooler tests whose socket directory production picks

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-28 20:23:22 -07:00
joshua-berri
d30c702c5c
feat(accounting): add durable event ledger writer (#48883)
* feat(accounting): add durable event ledger writer

* fix(accounting): validate delegated transaction managers locally

* docs(accounting): clarify ledger integration contracts

* fix(accounting): return ledger conflicts as values and keep cleanup on the kind index

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(accounting): import assert_never from typing_extensions for Python 3.10

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(accounting): annotate ledger integration test locals as Final

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(accounting): finish ledger test conventions

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-28 16:11:51 -07:00
Mateo Wang
567383c503
feat(proxy)!: model family and access group entries in model_max_budget (#47722)
* feat(ui): let the per-model budget field take a model pattern or access group

* feat(proxy): let model_max_budget entries name a model family or access group, shared across replicas

* fix(proxy): open each shared model budget window once for every replica, and match patterns in linear time

Without Redis, the first replica to charge an entry in a new window now opens it in the window spend table and
the others join it, so no flush replaces spend another replica already added. Budget patterns match by splitting
on `*` instead of a regular expression, a refused request names the entry that refused it, and the key form no
longer offers a typed pattern another row already budgets.

* fix(proxy): read the shared budget window from the writer, and keep concurrent first charges on one replica

Opening a shared window now reads the row from the writer, since a lagging read replica could hand a replica its
own start and let a flush replace the others' spend. A charge that waited on the open no longer resets a local
window another charge on the same replica opened meanwhile. Window tests use fixed timestamps.

* fix(proxy): stop a non proxy admin from lifting a team model cap with a looser group or pattern entry

Group and pattern entries now take precedence over other entries, so a team update by anyone but a proxy admin
must keep each new or changed entry within every existing cap it can take precedence over. A pattern can match a
provider-prefixed or wildcard-routed name, so any two patterns count as overlapping, and a model or group entry
counts as outranking every pattern.

* fix(proxy): narrow model budget pattern overlap, keep team caps under key patterns, and accept only typed patterns in the UI

* fix(proxy): keep model family budgets through database blips, team models and key entries

A failed window open no longer restarts the window at this replica's clock, so it cannot wipe shared spend. Spend charged meanwhile stays local and joins the shared window once the database answers

A key entry named after a model replaces only the team entry of that name, so team pattern and group caps still apply. Team-scoped models are gated and charged under their public name, and group lookups see team deployments

The team update guard compares caps by window length and only ranks patterns that can match the same name, including Bedrock vendor patterns behind a bedrock wildcard route

* fix(proxy): refuse team entries that take precedence over a capped model_max_budget entry

A non-proxy-admin could add a carve-out (an exact model, a narrower pattern or
a group) at or under the family cap, and its own counter then multiplied the
team's spend on that family. Any new entry that takes precedence over a capped
entry now needs a proxy admin, and group members include the team's own public
model names so team deployments cannot hide a group overlap.

* test(router): cover team_public_model_names_of

* docs(proxy): name pattern and access group keys in model_max_budget docstrings

* fix(proxy): name the entry holding the cap in the team model budget raise refusal

* fix(proxy): check precedence when a team edit makes an unenforced model_max_budget entry take effect

* fix(proxy): let an org admin give a legacy model_max_budget row a budget_duration at the same or a lower cap

* fix(proxy): keep a legacy model_max_budget row's stored cap when an org admin gives it a budget_duration
2026-10-28 15:40:57 -07:00
yucheng-berri
a53ea5bd77
fix(proxy): keep every startup line JSON when json_logs is on (#47035)
* fix(proxy): keep every startup line JSON when json_logs is on

Boot-time messages the proxy printed straight to stdout (banner, feedback
box, model list, cache and callback notices, policy table, migration
notices, CLI launch notes) and the first records of uvicorn- or
granian-spawned workers came out as plain text while json_logs was on,
breaking log pipelines that parse every line as JSON.

- print_startup_message() prints unchanged in plain mode and, with
  json_logs on, emits one INFO record on the proxy logger (ANSI
  stripped) that skips the logger and handler level checks like the
  print it replaces; show_banner() routes to it in JSON mode only
- the CLI exports JSON_LOGS=true when the config sets json_logs so
  spawned workers log JSON from their first record
- the --enforce_prisma_migration_check deprecation notice prints after
  the config is read so a config-only json_logs covers it

Resolves LIT-1624

* fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize()

Since #46803 the `lite` shell calls show_banner() against an uninitialized
litellm package, and reading litellm.json_logs inside it initialized the whole
SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR
removed on small VMs). show_banner() is back to its main version; the proxy's
initialize() now emits the JSON banner record itself and calls show_banner()
only in plain mode. The banner unit tests move to the initialize() call site.

* fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger

uvicorn applies --log_config after proxy_cli has switched JSON logging on and before
initialize() switches it on again through load_config(). A logging config that names
"LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where
initialize() emits the banner and the feedback box, so at one worker both went nowhere;
two-worker children import litellm after uvicorn's logging setup and were not affected.
print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON
handler uses, when the proxy logger has no handler of its own.

Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the
banner and the feedback box as plaintext, e2303a3925 prints neither, this head prints both as
one JSON record each. New unit test fails on e2303a3925; new integration cell
test_json_logs_with_operator_log_config_keeps_every_line_json passes at one and two workers.

* fix(proxy): keep a long startup listing whole in its JSON record

print_startup_message ran its record through the proxy logger's filters, so
in JSON mode the required-migration SQL list and the policy list were
middle-truncated at MAX_STRING_LENGTH_STDOUT_LOG (4096 by default) with the
litellm_truncated marker, where the prints they replace showed them whole.
That cap bounds echoed request payloads; a startup message is operator text,
so the record now gets the uncapped processing of a DEBUG line: secret
redaction and base64 collapse still apply, the text cap does not.

Red on 41f92b745a, green here:
- tests/unit/test_logging.py: a listing longer than the cap comes out whole;
  a credential in a startup message is still redacted
- tests/integration/configuration/test_json_logs_startup.py: a 60-policy
  config boots with json_logs on and the policy listing record (9762 chars)
  carries every policy with no truncation marker (4096 chars, marker, on
  the unfixed tree)

* fix(proxy): keep the --enforce_prisma_migration_check notice on --health and --test

This PR moved the deprecation notice after the config is read, so that a
json_logs set only in the config applies to it; --health and --test return
before that point and stopped printing it, a visible change main did not
have. The notice is now also emitted before those two returns, through the
same helper, so a deployment running `litellm --health
--enforce_prisma_migration_check` still learns the flag is dead, as on main.
A normal server start emits it once, after the config, as in the previous
commit.

Red on 2d7a2d6844, green here: tests/unit/proxy/test_proxy_cli.py asserts
the notice on --health and on --test.

* fix(proxy): keep the JSON startup records on stdout under an operator handler on the proxy logger

An operator --log_config that gives "LiteLLM Proxy" a handler of its own, a NullHandler
that silences it or a plain WARNING-level handler on stderr, is in force in the window
where the banner, the feedback box and the cache notice are emitted: uvicorn applies it
after the CLI process switched JSON on and before initialize() re-installs the JSON
handler through load_config(). 41f92b745a sent the startup records through the proxy
logger's handlers and fell back to a JSON stdout handler only when there were none, so
under such a config the records landed in the operator's handler, as plaintext on stderr,
or vanished, where the prints they replace reached stdout whatever the logging config
said.

print_startup_message() now writes every startup record through the proxy's own JSON
handler on stdout, filters included, and never through the proxy logger's handlers; the
operator's handler gets nothing from a startup message, as it got nothing from the print.
The normal JSON handler is the same class, formatter and filters, so the records a
pipeline reads are unchanged when no operator config is in play.

Tests: tests/unit/test_logging.py gets the two handler shapes (red on f6f5957d16: the
record went to stderr as plaintext or nowhere); tests/integration/configuration/
test_json_logs_startup.py gets a silencing operator --log_config cell at 1 and 2 workers
(red on f6f5957d16: banner, feedback box and cache notice missing; green here).

* fix(proxy): print the --enforce_prisma_migration_check notice when the config fails to load

f6f5957d16 printed the deprecation notice after the config is read, so a json_logs set
only in the config makes it a JSON record. A boot whose config file is missing or does
not parse then exited before that point and never printed it, where main printed it
before the error (10-20 gauntlet finding).

The notice is printed where main printed it, unless a config file can still switch
json_logs on: then it is held until the config is read and printed right after the
json_logs switch, and a click close callback prints it on the way out if the boot fails
first. functools.cache keeps it to one print either way.

Tests: tests/unit/proxy/test_proxy_cli.py covers a failed config read (red on
75043aed83), a plain config printing it once and a config-only json_logs making it one
JSON record; tests/integration/configuration/test_json_logs_startup.py adds the JSON cell
and a missing and an unparseable config cell (red on 75043aed83, green here), through
refused_boot_log, which now takes extra CLI arguments.

* fix(proxy): keep startup records as printed instead of running secret redaction

In JSON mode the startup messages ran secret redaction, which the prints they replace never
had. Its sk- key pattern has no left boundary, so a model named risk-assessment-gpt4o was
logged as riREDACTED in the model list (10-21 gauntlet finding). The records now skip
redaction like the prints did; the processed record still counts as scrubbed, so the
handler's secret filter and the formatter leave it alone. The stdout cap stays off and base64
collapse stays on, as before.

Tests: tests/unit/test_logging.py checks the model name comes out whole (red on 7d17f126e5),
replacing the test that asserted startup records were redacted;
tests/integration/configuration/test_json_logs_startup.py adds a boot with a
risk-assessment-<id> model whose model list record carries the name (red on 7d17f126e5).

* refactor(proxy): meet the AGENTS.md rules Greptile flagged on the startup-message change

The policy listing builds its rows as tuples in two helpers instead of appending to a list;
plain output is unchanged (same bytes as before across 3000 random policy and attachment
configs). The explanatory comments added with this change are gone, and the docstring of
print_startup_message now says the record is whole and unredacted.

Tests: the migration-diff JSON test no longer runs the fake prisma CLI; it checks the
skipped-diff notice in-process, and the fixture extension it needed is reverted. The
json_logs export and deprecation-notice CLI tests run the real proxy module and a real
config file instead of a mocked ProxyConfig and proxy module, and the --health/--test
notice tests no longer patch our helpers. Each still fails when the export, the hold or
call_on_close is removed. New test parameters are typed.

---------

Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
2026-10-27 14:49:07 -07:00
Mateo Wang
0a1dbc0fb4
refactor(mcp): drop the experimental label from the MCP server and client paths (#48513)
* refactor(mcp): drop the experimental label from the MCP server and client paths

Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims

pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 17:47:50 +00:00
devin-ai-integration[bot]
24a7bf6af7
fix(proxy): reconcile team member total_spend with current cycle spend (#48260)
* fix(proxy): reconcile team member total_spend with current cycle spend

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): read /team/list as a list and drain the scripted provider

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(repositories): compose the membership reset writer and drop SQL text asserts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: type the cascade repair test and flatten memberships without a stacked comprehension

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: keep the reset job test file's existing typing untouched

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 17:57:09 -07:00
Kerry Lu
e6a32bbeeb
fix(proxy): buffer tool discoveries in Redis and write LiteLLM_ToolTable in one statement per drain, on its own lock (#48089)
* fix(proxy): buffer tool discoveries in redis and let the leader write LiteLLM_ToolTable in one statement

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): declare the redis family on the tool discovery buffer and drop the blind except and cast

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): satisfy LIT010 and LIT014 in the tool discovery buffer

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): mark the aggregation fold Final

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): requeue drained tool rows to redis when the write provably did not apply

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): drain tool discoveries on their own lock and scheduler job, not the spend commit

* chore(proxy): suppress the untyped apscheduler add_job on the tool discovery job

* refactor(proxy): move the LiteLLM_ToolTable bulk upsert into its own module next to the other bulk upserts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): name the LiteLLM_ToolTable bulk write span so the prisma span check resolves it

* refactor(proxy): move the tool discovery drain out of DBSpendUpdateWriter into tool_registry_writer

* refactor(proxy): share db_write_failure_is_requeue_safe from exception_handler so the spend writer stays untouched

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-24 12:34:13 -07:00
devin-ai-integration[bot]
e159005545
fix(spend): await the per-request rollup enqueues and report their failures (#48131)
* fix(spend): await the per-request rollup enqueues and report their failures

The daily and entity spend rollups ran in an unreferenced asyncio task that
nothing awaited or drained, so a shutdown after the spend log was enqueued
could drop every rollup for that request, and a malformed metadata payload
dropped the request from all daily tables at debug level. The rollups now run
inline in update_database, each helper failure is logged through
spend_log_error with the request id, and unreadable metadata only costs the
row its metadata-derived fields

Resolves LIT-6027

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend): type the rollup failure helpers for the lint gates

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend): track the rollup enqueue task so shutdown drains it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(spend): shutdown drain waits for an in-flight rollup enqueue

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(spend): register the rollup task without blocking the cost callback

Awaiting the rollups inline made a blocked queue put stall increment_spend_counters, so budget checks could run on stale counters. The task is now only registered via spawn_logging_task; drain_logging_tasks still waits for it on shutdown

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-23 18:54:34 -07:00
michelligabriele
1c6d55c7e8
fix(proxy): show the newest health check per model group on the model hub (#47552) 2026-10-20 16:42:53 -07:00