mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-18 09:45:23 +00:00
87 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
9e81aa3f40
|
feat(shadow-eval): skip redundant reverse evaluations on opt-in (#52192)
* feat(shadow-eval): let reverse jobs skip samples the baseline group served A reverse shadow eval judges the router's response against baseline_model. When the router itself sent the request to the baseline's model group, the comparison only produces ties and still pays for a shadow and a judge call. Reverse jobs can now opt in with skip_same_model. A sampled request whose served group (router alias resolved, fallback hops respected) is the baseline's own group is counted in a new same_model funnel stage instead of being judged. It makes no provider calls, writes no attempt row, and leaves max_turns and the eval budget untouched. When the served group is unknown the sample is judged as before. The job detail reports same_model_skipped_count even when nothing has been judged yet. The flag defaults to false and is rejected on forward jobs, so existing jobs and requests behave exactly as before. Adds two columns through an additive, schema-only migration. * test(shadow-eval): drive same-model skip tests off explicit completion signals Judged cases now wait on the attempt write the injected prisma reports, then await the pipeline task that made it. Skip cases drain only the tasks the hook created, through a task factory scoped to the hook call, before asserting that no provider call, attempt row, or spend was recorded. This replaces a whole-loop task diff and the assertion on the number of tasks. Adds a skip case with tier parameter overrides, which the policy still treats as the same group, and a case showing a lost sampling roll counts as not_sampled before same_model. Drops the class docstring claim that a same group comparison only produces ties. * style(tests): remove extra shadow evaluation test spacing * fix(ui): send skip_same_model false from the shadow eval start form The generated request type now requires skip_same_model, which broke the dashboard typecheck and builds. The form sends false, the API default, so jobs started from the UI behave exactly as before. * fix(migrations): bound shadow evaluation schema lock waits --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
47a0d50c35
|
test: remove dead skip-marked tests and rewrite the Redis buffer drain test (#52063)
Fourteen tests carried an unconditional skip and could never run. The unit conftest deletes DATABASE_URL and REDIS_* before every test, so the prisma and Redis ones would fail in tests/unit even without the marker. Each behavior they claimed is already covered by mocked unit tests or by the integration suite against real Postgres and Redis, and the gpt-4o token counter test patched a function removed in #10409. The Redis buffer drain test was the one with unique value, so it is rewritten in test_redis_update_buffer.py against the fakeredis-backed RedisCache fixture. It fills every queue a spend flush writes, drains the buffer, and asserts no Redis key is left behind. |
||
|
|
1bc3190a94
|
feat(db): classify errors and recycle poisoned pools like Prisma (#49801)
db::Error now reports a kind that matches PrismaDBExceptionHandler's buckets: unreachable, capacity, deadlock, serialization failure, lock timeout, read-only after failover, stale cached plan and rejected data. is_unavailable gates fail-open and is_requeue_safe says whether a failed write provably never applied Pools::recycle_after is the native counterpart of recreating the Prisma client: after a 25006 (the writer endpoint now points at a read-only node) or "cached plan must not change result type" (a migration changed a table under a prepared statement), every pooled connection opened before the error is dropped on its next acquire, so the pool reconnects and prepares fresh Native failures reached Python as a ValueError, so allow_requests_on_db_unavailable treated a native outage as a permanent fault and a write on a dropped connection was requeued. The bridge now raises NativeDatabaseFailure with the kind and the Rust answers for is_unavailable and is_requeue_safe, and the two Python gates return them |
||
|
|
bd27f4e34c
|
feat(db): emit DB service events from native operations (#49810)
crates/db gains observe(), which records one event per round trip with the call type, table and operation a Prisma call reports. NativeDatabase forwards those events to a Python ServiceLoggingEvents that Database.open injects, and it calls the same ServiceTypes.DB success and failure hooks as db_span on the proxy's event loop. A native health_check, which pings one pooled connection without running SQL (call type health_check), is the first observed operation The native pool is lazy, so a pool that could not connect stayed silent until the first ported query failed on a request path. Database.open now pings it once in the background and logs a warning on failure, without failing or delaying boot The crate AGENTS.md records the Prometheus and OTel names a Rust host sink must match |
||
|
|
f6c9c70398
|
feat(db): add an optional native reader pool with explicit routing (#49804)
When DATABASE_URL_READ_REPLICA is set, NativeDatabase opens a reader pool resolved from that URL, with its own token rotation. Pools exposes writer(), read() and read_after_write(), matching RoutingPrismaWrapper and WriterPinnedClient. read() prefers the replica and read_after_write() the writer. Each falls back to the other side when its pool cannot connect and skips the failed side for 30 seconds, so reads keep working on the replica while the writer is down The reader pool takes its share out of the reader's connection_limit the same way the writer does, with the same hard-coded RUST_RESERVED_CONNECTIONS. While that is 0 the replica URL is built exactly as before and no native reader pool opens. A replica connection_limit that is not a positive integer is left for Prisma to judge, as it always was |
||
|
|
cc8ae16bb4
|
feat(db): add the SQLx pool foundation with the writer pool (#49798)
NativeDatabase.connect opens a lazy SQLx writer pool from an Endpoint that Python resolves from the final DATABASE_URL, read the way Prisma reads it: Unix socket host, sslmode / sslaccept / sslcert with Prisma's verification rules (prefer + strict asks the server once whether it offers TLS), sslidentity / sslpassword handed over as PEM, schema, options, pgbouncer mode, pool_timeout (0 waits forever) and socket_timeout. Options SQLx cannot send are logged instead of failing boot ConnectionBudget splits each worker's connection_limit between Prisma and the native pool, so the budget an operator sized never grows. The Rust share is the hard-coded RUST_RESERVED_CONNECTIONS and no setting names Rust, SQLx or Prisma. It is 0 until a ported query needs the pool, so Prisma's URL is exactly what it was, no native pool opens and nothing connects through SQLx. A connection_limit that is not a positive integer is never split and goes to Prisma as before. Once the share is above 0 Prisma keeps at least one connection, and pools are tagged litellm-prisma and litellm-native in pg_stat_activity. Python keeps minting RDS IAM and Entra tokens and hands each one to the pool statement_cache_size sets the native statement cache (Prisma's default 100) and pgbouncer=true turns it off. sqlx-postgres 0.9.0 still names every statement when the cache is off, so Statements runs queries unnamed in that case, like Prisma, until a release carries transact-rs/sqlx#4337. An application_name the operator set is used on both pools The SQL query files are pinned to LF so Windows checkouts hash them like .sqlx, and the crate lists .sqlx in its package include so the core sdist builds offline crates/db/AGENTS.md tracks parity with the Python Prisma stack The bridge now links litellm-db, so the editable install compiles its queries. The Rust DB workflow builds that install from the checked-in .sqlx metadata, since it runs before the migrations that the online check needs |
||
|
|
007d1c1b5e
|
feat(proxy): support budget_limits windows on internal users (#51763)
* feat(proxy): support internal user budget windows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make user budget migration repeatable and sync API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound lock wait for user budget migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit unset JSON windows when creating users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): retain personal budget windows for proxy admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): give user budget windows their own counters and keep them on the user
User window counters move to spend:user_window:<id>:<duration> in Python and Rust, so a user whose id ends in 🪟<duration> can no longer fill another user's window. /user/new stores the windows on the user only, clearing them stores null like keys, edits keep each window's stored reset, and the all-users bulk update goes through the per-user path
* fix(proxy): keep windows on team auto keys and write all users' windows in one transaction
* fix(proxy): clear spend counters on bulk spend reset and read bulk windows under the lock
An all-users bulk update that resets spend now clears each user's spend counters, so users are not left blocked by the old counter. The bulk path reads each user's stored windows after it takes the users lock, so a concurrent edit cannot be overwritten with stale resets.
JWT users created from default_internal_user_params get a reset_at on each default window, and the user window error names the spend and the limit like key and team errors do
* fix(proxy): give member-add users the default budget windows and bulk-update the users read under the lock
Team and org member add now copy default_internal_user_params.budget_limits onto the users they create, each window with its own reset_at, through the same helper the JWT upsert uses
The all-users bulk update now writes, evicts and reports the users it read after taking the users lock, so a user created before the lock gets the new limits
* test(proxy): check default window resets against the reset helper instead of the wall clock
* fix(proxy): return parsed windows from the all-users bulk update and chunk its window writes
* fix(proxy): fail an all-users bulk update whole when its transaction fails
The per-user fallback after a failed all-users transaction wrote the newest
user and then crashed on a missing user_email attribute, so the call answered
500 with one user changed. Racing window bulk updates past the 5s transaction
limit reached it. Answer 500 with no user changed instead
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
|
88873a70d4
|
fix(proxy): retain unwritten rollups after database disconnects (#51706)
* fix(proxy): retain unwritten rollups after database disconnects * fix(proxy): initialize auto-router turns per client --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
a166574ba4
|
fix(sdk): stop requiring redis to import the Messages streaming path (#51685)
* fix(sdk): stop requiring redis to import the Messages streaming path redis_update_buffer.py and dynamic_rate_limiter_v3.py imported RedisError at module level. Both sit on the import chain that the SDK /v1/messages stream loads, so an SDK install without redis lost every streaming success callback. Import RedisError only where a Redis call can actually raise it. * style(tests): space the optional redis import fixture per ruff format --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
46716fad56
|
fix(proxy): alert on customer org, customer team, and customer member budgets (#51579)
* fix(proxy): alert on customer org, customer team, and customer member budgets Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): scope customer alert dispatch and Redis tracing Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(alerts): escape customer identities in Slack budgets Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): dispatch customer budget events through custom loggers Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): read committed customer budget spend from writer Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): retain customer budget claims for the full period Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
|
ca4c6c0a6d
|
feat(proxy): customer budget rollup, reset on the budget row's window, named templates, team overrides, multi-window limits and pause_allow_increase (#51582)
* fix(proxy): reset customer org, team and member spend on their budget row's window The customer tables carried their own budget_reset_at next to a budget_id whose row already has one. Drop the column from the three tables and let the budget table cascade zero the customer rows it governs, with a member following its unexpired override when that override has a duration and its template otherwise. Customer budget writes stamp budget_reset_at from budget_duration the way /budget/new does, and the management responses report the governing budget's window under the same budget_reset_at field. * feat(budgets): support concurrent customer budget windows * fix(budgets): resolve compatibility and validation checks * feat(proxy): name budget rows, give customer teams an override and members a parent org * fix(budgets): preserve admitted windows during accounting * chore(schema): allow dropping the unreleased customer budget_reset_at columns * fix(budgets): apply standard migration lock timeout * feat(proxy): customer members belong to many teams, charged to the one the request names * test(migrations): allow the unreleased customer budget_reset_at drop * style(ui): prettier the customer dashboard files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(reset-job): name the batch bound on the customer budget IN filters Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(budgets): preserve mixed limits and resolve CI regressions * fix(budgets): reuse shared spend verification for customer windows * refactor(ui): isolate customer budget group creation Keep only the customer-creation UI on current main. This local patch still depends on the BudgetTemplate.budget_limits API and generated types from the #51510 rebuild. * fix(proxy): restore Field import and bound the member team lookup after the merge Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(proxy): send parent_team_ids in the budget window customer tests Co-authored-by: Tin <tin@berri.ai> Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): keep rejecting another organization's template after the budget_template_ids rename Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): reset customer teams and members in pages against each row's own cap, keep a template's clock on resubmitted durations Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * feat(proxy): pause_allow_increase cap policy with customer budget increase requests reviewed like MCP and skill submissions Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(proxy): cover customer budget increase requests and the cap_policy field Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * chore(ui): regenerate API types for cap_policy and customer budget increase requests Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): hang the increase request back-reference on the customer member model Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): declare increase request list filters with Annotated Query Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): scope budget increase requests to the key's organization, require a future expiry, one pending request per member, and never revive an expired override on approval Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): write customer budgets without windows as Prisma Json null, run the increase request table through the hierarchy transaction Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): review increase requests exactly once, let an approval replace template windows, pin member budget_id to the org's templates, open the request route to virtual keys Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Tin <tin@berri.ai> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
|
6eda17aadb
|
fix(proxy): enforce settled customer budgets across workers (#51568)
* fix(proxy): enforce settled customer budgets across workers Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): sync customer budget reset API types Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(proxy): reseed customer spend through shared counters Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
|
d30b2b9b26
|
feat(proxy): enforce customer org, team and member budgets and limits on the request path (#51489)
* feat(proxy): enforce customer org, team and member budgets and limits on the request path Resolve x-litellm-customer-member-id at auth, walk member -> team -> org, enforce max_budget, allowed_models, tpm/rpm and per-model limits at every level in the v3 limiter, honor the hierarchy allowlists in alternate-model steering, and settle spend to the member, team and org rows. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): customer hierarchy survives virtual key auth without a legacy end user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(auth): keys pinned to a customer organization may only act for that organization's members Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): keep the customer hierarchy spend maps within the line limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): check the key's customer org scope after the key is resolved Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): declare the hierarchy cache key family and keep type counts flat Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): resolve the customer hierarchy in the shared authorization gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): drop a member's cached hierarchy through the declared cache target Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(auth): keep the hierarchy reset within the line limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): type the requested models as a sequence Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(spend): add daily customer usage and budget resets (#51493) * feat(spend): add daily customer usage and budget resets * chore(spend): document customer reset query bounds * fix(spend): chart customer queue metrics and update recovery test * fix(proxy): read stamped customer ids from either metadata bucket, alias-aware model check, broadcast member evictions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "feat(spend): add daily customer usage and budget resets (#51493)" This reverts commit |
||
|
|
adc4c01aa9
|
chore: move scripts and runtime sources under litellm (#51006)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
6efbaf539c
|
chore: move deployment sources under litellm/deploy (#50981)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
701639ea22
|
feat(alerts): detect daily changes in recorded usage (#50666)
* feat(alerts): detect daily changes in usage for gateway, models, teams, users and keys
* fix(alerts): drop unused suppressions
* fix(alerts): satisfy strict lint for model usage timing
* fix(alerts): skip only resources that time out or hold invalid data
* fix(alerts): log an unfinished daily sweep at info level
* fix(alerts): keep the deployment-wide run every five minutes
* fix(alerts): record alert and sweep timestamps in UTC
* fix(alerts): repair detector tests and tracing
* refactor(alerts): build the key index in a migration instead of after startup
* refactor(alerts): build the key index after startup again
A concurrent index build inside the boot migration can exceed the 600 second migrate deploy deadline on a large LiteLLM_DailyUserSpend and stop the proxy from starting. This reverts
|
||
|
|
d40dedf23b
|
feat(proxy): run spend log cleanup from a DB-backed scheduled job table (#50131)
* feat(proxy): run spend log cleanup from a DB-backed scheduled job table Spend log retention ran from an in-process APScheduler timer that reset on every restart and needed Redis for leader election, so pods that restart more often than the interval never cleaned up at all. Add LiteLLM_ScheduledJobRun, a generic table with one row per job run. Every pod polls it, plans the next run idempotently and claims a due run under a transaction-scoped advisory lock, so exactly one pod runs each occurrence. Runs carry a lease, a status and a log. Intervals count from the last finished run and survive restarts. Schedule edits cancel and replace the pending run. The retention and size-check jobs share a lock group so they never overlap, and the Redis pod lock is gone. * fix(proxy): plan an earlier cron edit from now, and cover the job runner Moving the cleanup cron earlier queued tomorrow's run because the next occurrence was planned after the pending run being cancelled. Plan from finished runs only. Register LiteLLM_ScheduledJobRun as a span table, keep the advisory lock query inside the tick's span, move the non-string cron startup test to the schedule parser, and add unit tests for the runner and job wiring. * test(proxy): annotate new spend log cleanup test locals as Final * test(proxy): type the parameters of the new spend log cleanup tests * test(proxy): leave an hour of margin when moving a cron earlier so an hour boundary cannot flake it * fix(proxy): reject a cron that never fires instead of failing every tick A crontab like 0 0 31 2 * parses but has no next fire time, so next_run_at raised on every tick and the job was never scheduled again. cron_schedule now returns InvalidSchedule for it, which logs and keeps the pending run like any other bad config * refactor(proxy): drop the informational log column from LiteLLM_ScheduledJobRun Nothing read the column back. Run outcomes stay in status, and the reasons already go to the proxy logs, so JobResult becomes a SUCCEEDED or FAILED literal and the cleanup returns its RunOutcome instead of a summary report |
||
|
|
c2afb07f93
|
fix(spend): keep free response reads and cancels off the creation spend row (#51110)
* fix(spend): track self-heal for test_with_id_security_off_reads_and_cancels_add_no_rows_and_no_replayed_turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): preserve response turn rows when id security is disabled Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
19c48218ae
|
test: deflake proxy metrics and shutdown drain (#51078)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
076ded32be
|
feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066)
* feat(auto_router): report a session's whole spend on /auto_router/session Claude Code sends its background calls straight to Haiku under the same session id, so those requests never reached the auto-router session rollup and the status line under-reported the session. Every billed spend row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total without a shared Redis * fix(auto_router): name the unsupported provider when a turn routes off Anthropic A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model, failed strict native validation and was logged as observation_unavailable, which reads like a broken capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their existing reasons, and no estimate is added, so savings for these turns stay unknown * docs(auto_router): explain when session savings stay unestimated saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected for that window * fix(auto_router): say why a system-only cache prefix has no savings estimate Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the empty body still count it. A live check against claude-opus-5-5 showed a differential count with a placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated and the savings gap for Claude Code is not fixed here * fix(auto_router): declare the session total's Redis key family and test its expiry The session total read and wrote Redis without a service target, so its calls showed up as bare redis spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an active session keeps its total and an idle one restarts * docs(auto_router): note which turns are still estimated inside the history window Turns that took the identical baseline path, and turns whose whole cached prefix the session had already written, are estimated even inside the first cache TTL window, so saved_spend now says so --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
894552205b
|
feat(spend): report guardrail spend separately in daily activity (#51010)
* feat(spend): report guardrail spend separately in daily activity Model and key spend in /user/daily/activity/aggregated include the cost of guardrails like Bedrock guardrails, and nothing in the response says how much. A customer that bills tenants per key and per model could not get model cost alone, so tenants that pay AWS for their own guardrails were charged for them twice Daily spend rows now carry a guardrail_spend column next to spend, and every breakdown in the daily activity responses exposes it, so model cost is spend - guardrail_spend. spend keeps its meaning, so budgets still count guardrail cost as before The value comes from the request's cost_breakdown.guardrail_cost, capped at the row's spend because an allowance discount or a repeat job poll can charge less than the guardrail cost the breakdown records. The delivery ledger freezes it into the accounting payload next to accounting_savings, since the stored payload does not keep cost_breakdown, and the legacy daily queue writes the same value The migration only adds the column with a 0 default, so rows written before it read 0 * fix(spend): count guardrail spend on rows billed without a cost breakdown A request blocked before the LLM call is billed for its guardrails straight from guardrail_information, so it carries no cost_breakdown and its guardrail_spend read 0. Both writers now price guardrail_spend from guardrail_information with guardrail_information_cost, the same function success and failure rows use to add guardrail cost to spend Also fills guardrail_spend in the client usage view built from request logs, qualifies the field description for rows recorded before it existed, and adds the lock timeout the migration checks require * fix(ui): avoid a nested ternary when reading guardrail entries from request logs --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
52cd9ef0e4
|
fix(proxy): let the OS assign Prisma engine ports (#50850)
* fix(proxy): let the OS assign Prisma engine ports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): clear Prisma connection state after failed startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
febbe38326
|
fix(anthropic)!: do not surface server_tool_use as client tool_calls (#48385)
* fix(anthropic): do not surface server_tool_use as client tool_calls Anthropic server-executed tools (web search, tool search, code execution) were returned in chat completions tool_calls with srvtoolu_ ids, so OpenAI-protocol agent clients treated them as pending work and re-invoked the model until their recursion limit. Move them to provider_specific_fields.server_tool_calls on both the non-streaming and streaming paths. History replay, the /v1/responses bridge and the /v1/messages streaming guardrail read that field, so multi-turn reconstruction, web_search_call output items and the guardrail's fail-closed check on server tool blocks keep working. * fix(responses): return Claude web_fetch calls as web_search_call open_page items * fix(responses): keep Claude server web calls without a result as hosted items A non-streamed Claude turn that calls web_search or web_fetch next to a client tool can return the server call before its result. Build an in_progress web_search_call for it, so the Responses bridge no longer returns it as a client function_call * fix(anthropic): keep pending server calls native and hide server calls from client tool paths Replaying a server call that has no result yet now sends it back as server_tool_use, so Anthropic accepts the next turn. The tool permission guardrail checks server calls again and rewrite mode removes a denied one. The /v1/messages adapter keeps Claude's server blocks, and the Responses bridge never returns a server call as a client function_call. * fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls, so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own server_tool_use instead of running again * fix(anthropic): let guardrails and loggers read Claude's server tool calls again Guardrails, tool policies, tracing loggers and the Responses guardrail translation read only client tool calls after server calls moved to provider_specific_fields. They now read server calls first, then client calls, through one shared helper, and the Responses guardrail treats Claude's web_search_call items as web_search or web_fetch calls. Non-streaming splits server calls by block type, as streaming does. * fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy, streamed or not. Redaction with message logging off hides server call arguments and hosted web call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls. * fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use. * fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy. * test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions * test(guardrails): type the fake Pillar client like its Protocol * fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or web_search still needs that call and its thinking replayed, or Claude loses the fetch result * fix(guardrails): scan replayed Claude server calls and hosted web calls Request guardrails now see and rewrite server_tool_calls replayed in chat messages. Cato and Purview scan Claude web_search and web_fetch calls on Responses output, and a function tool call with an extra custom key is no longer dropped from Cisco, Purview, Repello and PANW scans. * fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks * fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows * test: annotate the new server tool call test locals with Final and type their parameters * test: type the new guardrail test helpers with litellm's tool call types instead of Any * test: narrow the remaining new test helper parameters from bare dict and object * fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls * fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera * fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500 --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
b1b236826a
|
feat(redis): serve the proxy's common Redis call with Rust stores (#50170)
* feat(redis): route the router, rate limiters and auth through the native store families * style(redis): satisfy the python lint gates Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(router): give the registry's default usage store a memory-only handle RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(redis): build stores from native handles and wait for router tasks Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a12a6cd154
|
feat(accounting)!: make the SpendLogs ledger the default accounting path (#49872)
* fix(accounting): keep one undecodable delivery from blocking the ledger A request whose metadata carried a 401-digit tokens_saved stored that number in its delivery payload. The query engine could not decode it when the worker claimed the batch, so every later delivery on the proxy stayed pending The worker now claims the payload as text and decodes each row in Python. A row that still cannot be decoded is parked alone while the rest of the batch applies. The payload builder also clamps integers to the JSON-safe range, so caller-supplied counts can no longer reach the engine at that size Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): take spend locks in one order so the ledger cannot deadlock A ledger delivery updates many spend tables in one transaction, so it can deadlock with any writer that locks the same rows in another order. It took the team advisory lock after the user and team rows, while team roster writers take it first The delivery now takes every team advisory lock before any row, then writes tables in one canonical order with ids sorted. It also runs under the rollup lock_timeout, applies 500 deliveries per batch by default, and spreads retries with jitter The writers that could still collide with it now follow the same order. The budget reset cascade, model allowlist rename, global spend reset, bulk user update, org delete and the delete-by-list paths lock their rows in key order first, and access group endpoints update teams and keys in one sorted pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): prove a ledger delivery waits instead of deadlocking Real Postgres tests hold the first lock of a team roster change, an access group update and a budget reset cascade, start a delivery, and let the other writer finish. The delivery must wait and then charge once. Before the lock order fix each case ends in a Postgres deadlock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accounting)!: make the SpendLogs ledger the default accounting path use_spend_logs_ledger now defaults to true, so every request is charged through LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and Redis spend queues, which stay in place. Tests that cover those legacy writers pin the setting off, and the rest check the queued charge the ledger stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): let the mocked client open a transaction for key and team deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): take team locks first when a model rename rewrites allowlists A rename locked team rows before user rows while a roster change holds the team advisory lock and a user row, so the two could deadlock. The rename now takes the advisory lock for every team that lists the model, sorted, before any row lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): drain a backlog in one worker run, one batch per transaction One run applied a single fresh batch, so at 500 per batch charges fell behind any load above that rate. The worker now keeps claiming fresh batches while they come back full, for up to five seconds, and each batch still commits on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): queue only the charge when a request's spend log is suppressed With logging off for a request, the ledger still queued the full row, so a failed flush or a shutdown could leave its prompt and response in Redis. The queue now holds only the fields the charge needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(proxy): format the rename team listing query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): remove expired delivery receipts as fast as the worker applies them The worker now applies several batches per run but still removed one cleanup batch of applied receipts, so the table could outgrow its retention window. Cleanup now repeats while each pass removes a full batch, for up to five seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): retry a model rename when a team starts listing the model mid-lock The team advisory locks come from a snapshot, so a team that gained the old name before its row lock was locked without its advisory lock. That attempt now commits without writing and the next one locks the fresh set of teams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
906fa7ed28
|
feat(accounting): add opt-in SpendLogs ledger (#49856)
* feat(accounting): add opt-in SpendLogs ledger delivery * fix(accounting): preserve missing organization daily spend parity * fix(accounting): preserve ledger receipts when replaying migration * fix(accounting): preserve callback windows and tune ledger batches * fix(accounting): preserve window charges and bound ledger statements * test(accounting): verify budget callback registration order * test(accounting): verify captured charges after repeated startup * feat(accounting): store durable payloads independently of spend logs * fix(accounting): preserve first delivery and trace ledger writes * fix(accounting): keep SDK imports independent of proxy tracing * fix(accounting): honor log retention during batch takeover * test(accounting): cover retained batch alerts and invalid settings * test(accounting): verify downgrade command against pending work * fix(accounting): drain retries without starving fresh deliveries * test(accounting): cover delivery projection, metrics and window key reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): read delivery metrics without the shared registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): charge a leftover $0 batch row when its claim fails A failed batch claim buffers the retrieve's row and reports it charged. At flush the row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back and the charge never reached the spend tables. The flush now takes that row over in the same transaction and keeps the delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): trace the batch takeover at flush as a SpendLogs update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client The export client is never created, so main always writes SpendLogs to the proxy's own database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was set, which dropped those request logs. It now treats a flush as an export only when the client exists, and batch claims write local rows as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep every request's charge when its log row is shared, evicted or rejected - Key non-batch deliveries on a per-request delivery_id so a reused client call id is still charged once per request, while buffer replays stay deduplicated - Persist the redacted api_key instead of an opaque bearer token - Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it - Strip NUL from payloads and skip non-text tags so projection never parks a paid request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging a request whose tag is too long for the daily tag index A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store only guardrail billing fields in the accounting payload The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): move spend payload stamping out of update_database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): restore the modules the import-layering guard evicts The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): keep update_database's payload stamping on its own local Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): type guardrail entries and use a public key redaction helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging requests whose end-user id is too long for an index An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): type the spend log queue snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): leave out any projected row Postgres cannot index or store Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep the guardrail fields compression savings are counted from Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): cap projected counters so daily totals cannot overflow Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): let a buffered batch charge take over its zero-cost receipt A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read a rescued batch row's spend from its accounting payload Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read budget windows through the delivery transaction Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): retry a transient failure while keeping a dropped row's charge The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): give delivery projection payloads their charged entities project() now reads entities only from accounting_entities, so the hand-built delivery payloads carry them like real stored payloads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): own ledger rows by their payload and key batches on the batch id Drop the accounting_path stamp and its SpendLogs column. A buffered row that carries an accounting payload is ledger owned, one without it is legacy, and spend counters get a plain windows_from_ledger flag. Window seeding tells applied ledger rows apart by their delivery row. A ledger batch is plain transaction A keyed by the batch id, so a repeated completed retrieve conflicts and charges nothing. A $0 retrieve records no delivery, a batch a legacy retrieve already charged is not charged again, and the legacy batch path is unchanged from main apart from refusing a batch the ledger already owns. Batch takeover and its delta bookkeeping are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): drop a projection test duplicated while rebasing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop previous_payload now that deliveries are never replaced Batch takeover was the only writer of previous_payload, so the worker applies each delivery's payload as stored and the column leaves the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop the ledger downgrade checklist script The use_spend_logs_ledger flag is the rollback path, so a separate downgrade gate is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): name the window seed reads so their Prisma span resolves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): remove the unused accounting event types and protocols types.py, protocols.py and measurements.py were left from the old event scaffolding. Nothing outside the package imports them, so they go with their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): park only deliveries whose own data fails A delivery that keeps failing on a database outage, lock timeout or deadlock stays pending and retries with capped backoff, so a recovered database applies it. Only a data error or an invalid payload parks the isolated row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a spend log row the insert statement rejects The raw SpendLogs insert validates columns and JSON before Postgres sees the row, so a row that fails there is now bisected out and dropped alone while its charge is kept, instead of failing and replaying the whole flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): carry a job's model window on every completed poll in ledger mode The ledger keeps the poll whose delivery claims the job, which need not be the poll that won the limiter's in-memory charge. Every completed poll now carries the job's window, so the delivery that lands always has it, while the live model budget is still charged once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): leave model-budget windows on the existing window writer Model-budget windows are written by the model budget limiter rather than derived from a request's spend, so they stay on main's window queue like live budget counters do. This drops window capture from the accounting payload, the deferred limiter registration and the job-window capture, and the ledger keeps owning key and team budget windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep a delivery pending when the database is unreachable Prisma reports a can't reach database server outage as a DataError, so the worker now checks for an unavailable database before treating a data error as the delivery's own fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate only rows the raw insert cannot encode Catching every ValueError also swallowed unexpected insert errors that main re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column or non-JSON value, and only that is isolated as a rejected row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store zero savings when caller data makes them uncomputable A huge caller-supplied tokens_saved overflowed the savings math inside build_accounting_payload, which dropped the request's charge in ledger mode. Savings that cannot be computed are now recorded as zero and the charge lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b080c59e78
|
fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows (#50132)
* fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows Postgres refuses every new connection with SQLSTATE 57P03 while it starts up, shuts down or replays WAL, and prisma surfaces that as a base DataError with no code. The spend-log writer took it for a poison row, bisected the batch down to single rows and dropped each one. Spend logs now retry and requeue on 57P03 and on server-terminated sessions (57P01, 57P02, 57P05), tool and model usage rollups requeue when the write provably never reached the server, the tool registry requeues its rows, and auth answers 503 instead of 401 while the auth database restarts. Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com> * fix(proxy): requeue tool discovery rows only when the write never reached Postgres, keep the auth fallback to 57P03 A terminated session (57P01, 57P02, 57P05) may have committed the tool upsert, and the upsert adds call_count on conflict, so replaying it counts the calls twice. The non-Redis tool discovery flush now uses the same never-sent rule as the rollups, which also stops a permanently rejected row (54000) from coming back every tick. One terminated session does not prove the database is down, so it no longer opens the allow_requests_on_db_unavailable fallback, it still answers 503 * fix(proxy): a full server (53300, P2024) also never took the write, so counters and tool rows are sent again Postgres refuses the connection before any statement runs when it is out of slots, and a prisma pool timeout never reaches the server either, so those failures are as safe to replay as 57P03 and P1001 --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com> |
||
|
|
8fcb016dc6
|
test: make three flaky unit tests deterministic (#50114)
* test: make four flaky unit tests deterministic The skill archive fixtures stamped the current time into zip bytes used as parametrize ids, so xdist workers collected different ids and errored the whole shard. They now use a fixed timestamp. The usage-based v2 override test pinned no clock, so a minute boundary between rpm checks reset the counter. It now pins get_utc_datetime the way #49505 does. The stalled-burst DB lookup test asserted a 2s wall-clock bound. It now asserts every lookup was waiting at once, which fails if lookups are serialized. The long tool_search placement test asserted a 1s wall-clock bound. It keeps the placement assertion and drops the bound, as #49505 did for the json accumulator * test(skills): drop the archive fixture timestamp change now that #50118 removed its test |
||
|
|
8ab0321816
|
fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed (#49824)
* fix(proxy): charge a video's provider-reported cost once, on the first poll that sees it completed
A custom video provider can report a video's cost on the status poll that
sees it completed. Every poll of a video shared the video id as its spend
log request_id, so the paid poll's row was dropped as a duplicate and every
later poll of the finished video charged the cost again.
Video status rows now carry their own `_video_cost` request_id, and a poll
that reports a cost goes through the same settle-once claim batches use, so
the first poll that sees the video completed charges it and writes the row,
and every later poll charges nothing.
* fix(proxy): charge a video's provider-reported cost once on model, router and session budgets too
A video status poll gets the _video_cost row suffix only when it reports a cost, so a $0 poll keeps the bare video id and adds no row of its own, and the spend log writer claims a row by that suffix.
The batch charge-once marker now covers videos, keyed on the video id under {spend_key}:video:{id}, with the batch marker key unchanged. Model and router budgets use it, and per-session budgets now charge a repolled batch or video once and ignore a cost that is not a finite number
* fix(proxy): charge a finished video's own cost once, keyed by the id the client polled
Guardrail cost on a running poll no longer takes the charge-once claim, only a completed poll charges, the session marker TTL refreshes like the others, and two deployments whose provider ids match each keep their own charge.
* fix(proxy): charge repeat poll guardrails, key video charges by canonical id and survive redaction
* fix(proxy): keep video status polls in-process for the collector and give guardrail-only job polls an unclaimable row id
* fix(proxy): keep repeat batch polls free and group guardrail-only batch rows under their batch's trace
* fix(proxy): settle a finished job whose own cost is below its poll's guardrail cost under the job's charge row
* test(proxy): count charged_poll_cost as covered through the finished job guardrail poll tests
* fix(proxy): settle a finished batch that cost nothing itself on its first priced retrieve
* fix(proxy): charge no budget for a batch retrieve before its cost is final, and read the retrieved batch from before redaction
* fix(proxy): charge sessions a poll's guardrail-inclusive cost, count a job cost row's guardrail run, and keep batch_cost_is_final importable from litellm.batches
* fix(proxy): claim a job and charge the session budget in one Redis call
The session limiter claimed the job marker and then raised spend in a second Redis call. A failure between the two left the job claimed while its charge sat only in local memory, so later polls skipped it. One script now does both, and a refused script claims and charges locally
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
|
02b8da2f16
|
refactor: replace accounting event scaffolding with SpendLogs projection (#49843)
* refactor(accounting): replace event scaffolding with SpendLogs projection * test: preserve spend timestamps without removed capture callback * fix: guard accounting event removal across schema update paths * refactor(accounting): retain schema and prepare durable payloads * fix(accounting): preserve dates in serialized payloads * fix(accounting): retain schema push compatibility * fix(accounting): retain span registration for the legacy schema * test(accounting): remove obsolete capture acknowledgement test * fix(accounting): date every daily projection row the way the model rollup does A space-separated startTime kept the whole timestamp as the daily key, while the model usage rollup took the calendar date. Every daily row now uses the same date helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(ui): regenerate dashboard API types after rebasing on main Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): project only from the stored accounting payload project() now reads entities only from the payload's accounting_entities, so the raw SpendLogs fallback branches are gone. build_accounting_payload() builds the metadata once instead of dumping it to JSON and parsing it back. Guardrail usage dates use a tolerant UTC parse, so a malformed startTime drops only the guardrail rows instead of failing the whole projection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2d0c720425
|
fix(proxy): cancel the Admin UI's raw spend-log reads in Postgres after 30s (#49965)
* fix(proxy): cancel the Admin UI's raw spend-log reads in Postgres after 30s Prisma's HTTP client gives up on a statement after 30s, so the Logs, Caching, Cost Optimization and spend report pages already fail at 30s, but Postgres kept running the scan. Every raw LiteLLM_SpendLogs read behind those pages now runs under SET LOCAL statement_timeout = 30000 through one shared helper, which /team/spend/by_user also uses instead of its own copy * fix(proxy): keep the spend report's 503 and time out the session totals read The grouped Logs page's session totals query now runs under the same statement timeout, /global/spend/report re-raises the helper's 503 instead of turning it into a 400, and the hand-written prisma fakes in the endpoint tests expose read_db so every guarded route is exercised * fix(proxy): keep the Logs page when Postgres cancels its session totals read A cancelled session totals read degrades like any other failed enrichment read instead of failing the whole page. The span-name test treats the shared statement-timeout helper as a forwarder, and the spend report test no longer depends on a license in the environment * test(proxy): /global/spend/report keeps the 503 hint when Postgres cancels the read --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
391c8013c9
|
fix(proxy): cancel /team/spend/by_user in Postgres after 30s (#49922)
* fix(proxy): let the Admin UI cancel /team/spend/by_user at 60s with an opt-in statement timeout GET /team/spend/by_user takes an optional timeout_seconds (1 to 600). When set, the raw LiteLLM_SpendLogs aggregation runs under SET LOCAL statement_timeout and Postgres cancels it, and the route returns 503 instead of letting the query run for hours. Callers that omit the param are unchanged. The Admin UI sends timeout_seconds=60 and the Spend Per User Within Team card shows the server's error instead of an empty table * fix(proxy): cancel /team/spend/by_user in Postgres after 30s Prisma's HTTP client already gives up on the statement at 30s, so the query now runs under SET LOCAL statement_timeout = 30000 for every caller and the opt-in timeout_seconds param is gone. The Admin UI card shows the request error instead of the empty-state text. * fix(proxy): keep /team/spend/by_user on the read replica inside its timeout transaction --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
be4f3d36ce
|
feat(accounting): capture request usage through the durable meter (#49133)
* feat(accounting): capture request usage through the durable meter * fix(accounting): preserve native Google usage compatibility * fix(accounting): preserve prepared usage observation on failure * fix(accounting): exclude sync acknowledgement policy rejections from capture * test(accounting): verify capture races and failure isolation * fix(accounting): preserve request support with sync acknowledgements * fix(accounting): normalize usage metadata before queue admission * fix(accounting): preserve prior attempt links on successful fallback * fix(accounting): preserve attempt evidence through retries and failures * fix(accounting): depend on prior receipts without resubmitting usage * fix(accounting): preserve pricing inputs and resolved capture attribution * fix(accounting): persist batch usage within capture bounds * test(accounting): cover batch dependency failure and backpressure * fix(accounting): isolate batch persistence and preserve trusted attribution * fix(accounting): share byte admission across batch producers and writes * fix(accounting): reserve batch snapshot reads before admission * fix(accounting): compact repeated batch observations without losing requests --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
02a6a96dec
|
feat(accounting): add bounded transactional usage persistence (#49190)
* feat(accounting): add bounded transactional usage persistence * fix(accounting): preserve usage flush and shutdown on Python 3.10 * test(accounting): cover batch rejection and partial shutdown recovery * fix(accounting): batch durable usage on a five-second interval * fix(accounting): drain queued full batches without timer delays * fix(accounting): isolate invalid producer submissions within usage batches * fix(accounting): await commit prerequisites within bounded usage admission --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
|
2d79d5dba7
|
refactor(proxy): give database shutdown one lifespan owner (#48506)
* refactor(proxy): give database shutdown one lifespan owner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): log database cleanup errors and bound shutdown waits Database.aclose now logs a failure in either cleanup step instead of raising out of the lifespan exit, which matches how main handled stop errors and keeps setup_prisma_client from raising out of cleanup after a failed connect Shutdown waits for an in-flight reconnect for a fixed five seconds instead of the reconnect timeout plus grace, and joins engine wait threads with a fixed limit instead of indefinitely. The threads are daemon threads, so a thread still alive after the limit cannot keep the process from exiting Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
df47ec0171
|
fix(spend): keep a NaN or infinite cost from wedging daily spend writes (#48829)
* fix(spend): keep a NaN or infinite cost from wedging daily spend writes A non-finite provider-reported cost now counts as not reported, so the token price bills the request. Any other non-finite cost is logged and recorded as 0, and spend log JSON writes NaN and Infinity as null so Postgres accepts the row. * fix(spend): price non-finite provider costs on every route and drain parked NaN batches Perplexity, OpenRouter pass-through, and Responses streaming now treat a NaN or infinite reported cost as unreported, so the token price bills the request. A daily spend batch parked in the Redis buffer by an older version with a NaN spend now drains as 0 instead of failing every flush. The cost header never carries NaN or Infinity, and a set holding a non-finite float no longer breaks spend log JSON * fix(spend): record non-finite savings as 0 and warn when a parked NaN batch drains An overflowing price on a compressed request made compression savings infinite, which still wedged the daily spend batch. Every savings column now records a non-finite value as 0 with a warning. The Redis drain warns when it reads a NaN an older version parked, since the spend summed into it is lost from daily spend. The writer test now runs the real payload builder and spend log insert * fix(spend): keep non-finite costs out of headers, responses, and admin JSON Writing NaN as null is now opt-in for spend log columns, the Redis buffer, and streamed frames, so admin writes and logging integrations keep their output. The cost calculator records an overflowing total as 0 with a warning, cost headers leave out non-finite values, a non-streaming NaN usage.cost is dropped instead of failing the response, and a parked NaN spend log row reads back as 0. * fix(spend): keep caller-set call ids from splitting the non-finite cost warning The overflow warning strips newlines from litellm_call_id, and neither non-finite cost warning prints the raw value, which said nothing beyond "non-finite" * fix(spend): drop non-finite Responses and OpenRouter costs and the breakdown of an unpriceable request * fix(types): drop a usage cost breakdown that holds a non-finite number * fix(streaming): keep a dropped non-finite final usage cost from falling back to an earlier chunk's cost * fix(spend): keep the dropped-cost flag out of reach of provider usage keys A provider usage key named dropped_non_finite_cost shadowed the Usage method on the instance, so the stream builder raised and dropped the request's spend. The builder now reads the flag through the class, and Usage sets the flag after copying provider keys, so a _dropped_non_finite_cost key cannot set it either. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
b8283d8b61
|
feat(usage): show subscription-covered providers and account fees separately (#44772)
* feat(usage): show subscription-covered providers and account fees separately * fix(usage): write only the named fee columns and sync the root schema The fee update now writes just the columns the request names, so two admins changing different fields at once both land, and the dashboard edit form sends only what changed. The root schema.prisma carries the new table like the two copies, and a billing period anchored in the calendar's last month ends on December 31, 9999 instead of failing. * fix(usage): register the subscription table span and keep provider logic under llms * fix(subscription_accounts): register the fee routes as management routes and reject an empty account id * fix(usage): run the new tests in CI and close subscription card review gaps Adds the subscription test file to its unit shard, sets the migration lock timeout, rejects infinite fees, keeps the card from rewriting login files, lists each deployment once without aliases, shows the provider badge on entity views, and asks before removing a fee * fix(usage): resolve named-credential ChatGPT accounts and keep admin viewers read-only on the fee card * fix(usage): count a subscription fee only for accounts behind a current deployment Usage listed a fee row for an account no deployment uses while another ChatGPT deployment existed and dropped it when none did. It now lists only accounts behind a current deployment in both cases. Adds a real proxy and Postgres integration test for the fee API and for ChatGPT Responses rows on a deployment's own login, and names the card's multi-component test with the integration suffix. * fix(usage): keep a stored subscription fee listed after its account's deployments are removed The usage route lists every account behind a subscription deployment plus every account carrying a stored fee, and reads the fees on every proxy, so a fee stays visible, counted, and editable for past ranges after its last deployment goes. The integration test now polls every day the calls spanned, so a UTC midnight crossing no longer fails it. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
a0a8ca97d9
|
fix(cost): finalize synchronous response cost once for header, spend log and rollups (#48993)
* fix(cost): finalize synchronous response cost before accounting callbacks * fix(cost): keep serving and billing when cost finalization fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): read the finalized cost only off real logging objects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): follow the renamed success dispatch and deferred sync callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): keep cache hits free and their headers finalized when logging is deferred Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): preserve completed synchronous charges after hook failures * fix(cost): retain completed usage across cancellation and log suppression * fix(cost): reconcile completed failures and retain cached usage metadata * fix(cost): retain finalized request tags in failure budget updates * fix(cost): honor disabled spend tracking for completed failures * fix(cost): preserve pass-through accounting across cancellation --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
ead221995e
|
refactor(router): move router callers onto the entry points and invert the proxy imports (#48847)
Code outside the router now reaches it only through the entry points (registry, settings, health, select, apply, observe, register_plugin, retire) or the endpoint methods; the outside-member allowlist shrinks from 139 to 45. litellm/router.py no longer imports litellm.proxy: it asks the installed RouterHost (litellm/router_host.py) for vertex pass-through credentials, member auto-router authorization and auto-router prompt compression, and the proxy installs its host on import. Call sites where a model name can be missing keep the old answer instead of reaching an entry point with None. |
||
|
|
4bc157e281
|
feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations (#48961)
* feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations * feat(proxy): roll back each migration of the dry run on its own and baseline-check databases without a ledger * test(proxy): cover the dry run CLI and PrismaManager wrapper without a database * fix(proxy): refuse the no-ledger dry run for a custom migration folder like boot does, and probe the rolled-back indexes directly * ci(codecov): stop carrying forward the roi-postgres and redis-compat flags Their carried-forward sessions were measured against older proxy_cli.py revisions (633 and 689 line maps against 710 today) and mark docstring lines of the merged file as missed, which failed codecov/patch on this PR at 70% while every unit shard reports the lines hit --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
c489aa7117
|
feat(proxy)!: add suppress_startup_output to turn off startup prints (#47708)
* fix(proxy): keep every startup line JSON when json_logs is on Boot-time messages the proxy printed straight to stdout (banner, feedback box, model list, cache and callback notices, policy table, migration notices, CLI launch notes) and the first records of uvicorn- or granian-spawned workers came out as plain text while json_logs was on, breaking log pipelines that parse every line as JSON. - print_startup_message() prints unchanged in plain mode and, with json_logs on, emits one INFO record on the proxy logger (ANSI stripped) that skips the logger and handler level checks like the print it replaces; show_banner() routes to it in JSON mode only - the CLI exports JSON_LOGS=true when the config sets json_logs so spawned workers log JSON from their first record - the --enforce_prisma_migration_check deprecation notice prints after the config is read so a config-only json_logs covers it Resolves LIT-1624 * fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize() Since #46803 the `lite` shell calls show_banner() against an uninitialized litellm package, and reading litellm.json_logs inside it initialized the whole SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR removed on small VMs). show_banner() is back to its main version; the proxy's initialize() now emits the JSON banner record itself and calls show_banner() only in plain mode. The banner unit tests move to the initialize() call site. * fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger uvicorn applies --log_config after proxy_cli has switched JSON logging on and before initialize() switches it on again through load_config(). A logging config that names "LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where initialize() emits the banner and the feedback box, so at one worker both went nowhere; two-worker children import litellm after uvicorn's logging setup and were not affected. print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON handler uses, when the proxy logger has no handler of its own. Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the banner and the feedback box as plaintext, |
||
|
|
a2ef18b257
|
test(unit): keep unix socket paths in unit tests under the macOS AF_UNIX limit (#45040)
* test(proxy): keep the spend sidecar socket path under the macOS AF_UNIX limit * test(unit): share the short socket fixtures across unit tests Move socket_dir and socket_path to tests/unit/conftest.py so the datadog redis test's absent socket fails for the right reason on a Mac, rename the pgbouncer helper's directory argument, and record the rule in tests/unit/AGENTS.md * test(unit): fall back to /tmp when TMPDIR leaves no room for a socket path socket_dir now uses /tmp when TMPDIR plus the longest socket suffix would pass macOS's sun_path limit, and its teardown ignores errors so a failed test reports only its own failure * test(unit): keep socket fixtures inside tmp_path behind a short link socket_dir now links a short temp directory to tmp_path, so artifacts keep pytest's retention and a killed run leaks only an empty link. The fallback to /tmp counts the temp directory in bytes, and socket_tempdir points tempfile at it for the pooler tests whose socket directory production picks --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
|
d30c702c5c
|
feat(accounting): add durable event ledger writer (#48883)
* feat(accounting): add durable event ledger writer * fix(accounting): validate delegated transaction managers locally * docs(accounting): clarify ledger integration contracts * fix(accounting): return ledger conflicts as values and keep cleanup on the kind index Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(accounting): import assert_never from typing_extensions for Python 3.10 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(accounting): annotate ledger integration test locals as Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(accounting): finish ledger test conventions --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
567383c503
|
feat(proxy)!: model family and access group entries in model_max_budget (#47722)
* feat(ui): let the per-model budget field take a model pattern or access group * feat(proxy): let model_max_budget entries name a model family or access group, shared across replicas * fix(proxy): open each shared model budget window once for every replica, and match patterns in linear time Without Redis, the first replica to charge an entry in a new window now opens it in the window spend table and the others join it, so no flush replaces spend another replica already added. Budget patterns match by splitting on `*` instead of a regular expression, a refused request names the entry that refused it, and the key form no longer offers a typed pattern another row already budgets. * fix(proxy): read the shared budget window from the writer, and keep concurrent first charges on one replica Opening a shared window now reads the row from the writer, since a lagging read replica could hand a replica its own start and let a flush replace the others' spend. A charge that waited on the open no longer resets a local window another charge on the same replica opened meanwhile. Window tests use fixed timestamps. * fix(proxy): stop a non proxy admin from lifting a team model cap with a looser group or pattern entry Group and pattern entries now take precedence over other entries, so a team update by anyone but a proxy admin must keep each new or changed entry within every existing cap it can take precedence over. A pattern can match a provider-prefixed or wildcard-routed name, so any two patterns count as overlapping, and a model or group entry counts as outranking every pattern. * fix(proxy): narrow model budget pattern overlap, keep team caps under key patterns, and accept only typed patterns in the UI * fix(proxy): keep model family budgets through database blips, team models and key entries A failed window open no longer restarts the window at this replica's clock, so it cannot wipe shared spend. Spend charged meanwhile stays local and joins the shared window once the database answers A key entry named after a model replaces only the team entry of that name, so team pattern and group caps still apply. Team-scoped models are gated and charged under their public name, and group lookups see team deployments The team update guard compares caps by window length and only ranks patterns that can match the same name, including Bedrock vendor patterns behind a bedrock wildcard route * fix(proxy): refuse team entries that take precedence over a capped model_max_budget entry A non-proxy-admin could add a carve-out (an exact model, a narrower pattern or a group) at or under the family cap, and its own counter then multiplied the team's spend on that family. Any new entry that takes precedence over a capped entry now needs a proxy admin, and group members include the team's own public model names so team deployments cannot hide a group overlap. * test(router): cover team_public_model_names_of * docs(proxy): name pattern and access group keys in model_max_budget docstrings * fix(proxy): name the entry holding the cap in the team model budget raise refusal * fix(proxy): check precedence when a team edit makes an unenforced model_max_budget entry take effect * fix(proxy): let an org admin give a legacy model_max_budget row a budget_duration at the same or a lower cap * fix(proxy): keep a legacy model_max_budget row's stored cap when an org admin gives it a budget_duration |
||
|
|
a53ea5bd77
|
fix(proxy): keep every startup line JSON when json_logs is on (#47035)
* fix(proxy): keep every startup line JSON when json_logs is on Boot-time messages the proxy printed straight to stdout (banner, feedback box, model list, cache and callback notices, policy table, migration notices, CLI launch notes) and the first records of uvicorn- or granian-spawned workers came out as plain text while json_logs was on, breaking log pipelines that parse every line as JSON. - print_startup_message() prints unchanged in plain mode and, with json_logs on, emits one INFO record on the proxy logger (ANSI stripped) that skips the logger and handler level checks like the print it replaces; show_banner() routes to it in JSON mode only - the CLI exports JSON_LOGS=true when the config sets json_logs so spawned workers log JSON from their first record - the --enforce_prisma_migration_check deprecation notice prints after the config is read so a config-only json_logs covers it Resolves LIT-1624 * fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize() Since #46803 the `lite` shell calls show_banner() against an uninitialized litellm package, and reading litellm.json_logs inside it initialized the whole SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR removed on small VMs). show_banner() is back to its main version; the proxy's initialize() now emits the JSON banner record itself and calls show_banner() only in plain mode. The banner unit tests move to the initialize() call site. * fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger uvicorn applies --log_config after proxy_cli has switched JSON logging on and before initialize() switches it on again through load_config(). A logging config that names "LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where initialize() emits the banner and the feedback box, so at one worker both went nowhere; two-worker children import litellm after uvicorn's logging setup and were not affected. print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON handler uses, when the proxy logger has no handler of its own. Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the banner and the feedback box as plaintext, |
||
|
|
0a1dbc0fb4
|
refactor(mcp): drop the experimental label from the MCP server and client paths (#48513)
* refactor(mcp): drop the experimental label from the MCP server and client paths Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
24a7bf6af7
|
fix(proxy): reconcile team member total_spend with current cycle spend (#48260)
* fix(proxy): reconcile team member total_spend with current cycle spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): read /team/list as a list and drain the scripted provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(repositories): compose the membership reset writer and drop SQL text asserts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: type the cascade repair test and flatten memberships without a stacked comprehension Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: keep the reset job test file's existing typing untouched Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
e6a32bbeeb
|
fix(proxy): buffer tool discoveries in Redis and write LiteLLM_ToolTable in one statement per drain, on its own lock (#48089)
* fix(proxy): buffer tool discoveries in redis and let the leader write LiteLLM_ToolTable in one statement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): declare the redis family on the tool discovery buffer and drop the blind except and cast Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): satisfy LIT010 and LIT014 in the tool discovery buffer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): mark the aggregation fold Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): requeue drained tool rows to redis when the write provably did not apply Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): drain tool discoveries on their own lock and scheduler job, not the spend commit * chore(proxy): suppress the untyped apscheduler add_job on the tool discovery job * refactor(proxy): move the LiteLLM_ToolTable bulk upsert into its own module next to the other bulk upserts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): name the LiteLLM_ToolTable bulk write span so the prisma span check resolves it * refactor(proxy): move the tool discovery drain out of DBSpendUpdateWriter into tool_registry_writer * refactor(proxy): share db_write_failure_is_requeue_safe from exception_handler so the spend writer stays untouched --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
e159005545
|
fix(spend): await the per-request rollup enqueues and report their failures (#48131)
* fix(spend): await the per-request rollup enqueues and report their failures The daily and entity spend rollups ran in an unreferenced asyncio task that nothing awaited or drained, so a shutdown after the spend log was enqueued could drop every rollup for that request, and a malformed metadata payload dropped the request from all daily tables at debug level. The rollups now run inline in update_database, each helper failure is logged through spend_log_error with the request id, and unreadable metadata only costs the row its metadata-derived fields Resolves LIT-6027 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): type the rollup failure helpers for the lint gates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): track the rollup enqueue task so shutdown drains it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): shutdown drain waits for an in-flight rollup enqueue Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend): register the rollup task without blocking the cost callback Awaiting the rollups inline made a blocked queue put stall increment_spend_counters, so budget checks could run on stale counters. The task is now only registered via spawn_logging_task; drain_logging_tasks still waits for it on shutdown Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
|
1c6d55c7e8
|
fix(proxy): show the newest health check per model group on the model hub (#47552) |