mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-11 08:38:24 +00:00
* fix(accounting): keep one undecodable delivery from blocking the ledger A request whose metadata carried a 401-digit tokens_saved stored that number in its delivery payload. The query engine could not decode it when the worker claimed the batch, so every later delivery on the proxy stayed pending The worker now claims the payload as text and decodes each row in Python. A row that still cannot be decoded is parked alone while the rest of the batch applies. The payload builder also clamps integers to the JSON-safe range, so caller-supplied counts can no longer reach the engine at that size Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): take spend locks in one order so the ledger cannot deadlock A ledger delivery updates many spend tables in one transaction, so it can deadlock with any writer that locks the same rows in another order. It took the team advisory lock after the user and team rows, while team roster writers take it first The delivery now takes every team advisory lock before any row, then writes tables in one canonical order with ids sorted. It also runs under the rollup lock_timeout, applies 500 deliveries per batch by default, and spreads retries with jitter The writers that could still collide with it now follow the same order. The budget reset cascade, model allowlist rename, global spend reset, bulk user update, org delete and the delete-by-list paths lock their rows in key order first, and access group endpoints update teams and keys in one sorted pass Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): prove a ledger delivery waits instead of deadlocking Real Postgres tests hold the first lock of a team roster change, an access group update and a budget reset cascade, start a delivery, and let the other writer finish. The delivery must wait and then charge once. Before the lock order fix each case ends in a Postgres deadlock Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(accounting)!: make the SpendLogs ledger the default accounting path use_spend_logs_ledger now defaults to true, so every request is charged through LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and Redis spend queues, which stay in place. Tests that cover those legacy writers pin the setting off, and the rest check the queued charge the ledger stores Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): let the mocked client open a transaction for key and team deletes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): take team locks first when a model rename rewrites allowlists A rename locked team rows before user rows while a roster change holds the team advisory lock and a user row, so the two could deadlock. The rename now takes the advisory lock for every team that lists the model, sorted, before any row lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): drain a backlog in one worker run, one batch per transaction One run applied a single fresh batch, so at 500 per batch charges fell behind any load above that rate. The worker now keeps claiming fresh batches while they come back full, for up to five seconds, and each batch still commits on its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): queue only the charge when a request's spend log is suppressed With logging off for a request, the ledger still queued the full row, so a failed flush or a shutdown could leave its prompt and response in Redis. The queue now holds only the fields the charge needs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * style(proxy): format the rename team listing query Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): remove expired delivery receipts as fast as the worker applies them The worker now applies several batches per run but still removed one cleanup batch of applied receipts, so the table could outgrow its retention window. Cleanup now repeats while each pass removes a full batch, for up to five seconds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): retry a model rename when a team starts listing the model mid-lock The team advisory locks come from a snapshot, so a team that gained the old name before its row lock was locked without its advisory lock. That attempt now commits without writing and the next one locks the fresh set of teams. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
543 lines
26 KiB
Python
543 lines
26 KiB
Python
import asyncio
|
|
import os
|
|
from collections.abc import AsyncIterator, Iterator, Mapping, Sequence
|
|
from pathlib import Path
|
|
from typing import Final
|
|
from urllib.parse import urlsplit, urlunsplit
|
|
|
|
import psycopg
|
|
import pytest
|
|
import pytest_asyncio
|
|
from prisma import Prisma
|
|
|
|
from litellm.repositories.accounting_delivery_repository import (
|
|
AccountingDeliveryRepository,
|
|
DeliveryProjection,
|
|
LedgerTransaction,
|
|
)
|
|
from litellm.repositories.spend_log_insert import spend_log_insert_statement
|
|
from tests.integration._support.database import scratch_database
|
|
|
|
_MIGRATIONS: Final = Path(__file__).resolve().parents[3] / "litellm-proxy-extras/litellm_proxy_extras/migrations"
|
|
|
|
|
|
@pytest.fixture
|
|
def delivery_database(request: pytest.FixtureRequest) -> Iterator[psycopg.Connection[tuple[object, ...]]]:
|
|
with scratch_database("spendlogs_delivery") as url, psycopg.connect(url, autocommit=True) as connection:
|
|
if getattr(request, "param", "public") == "ledger_tenant":
|
|
connection.execute("CREATE SCHEMA ledger_tenant")
|
|
connection.execute("SET search_path TO ledger_tenant")
|
|
connection.execute((_MIGRATIONS / "20261027000000_add_accounting_ledger_tables/migration.sql").read_text())
|
|
connection.execute("""CREATE TABLE "LiteLLM_SpendLogs" (
|
|
request_id text PRIMARY KEY, call_type text NOT NULL, spend float8 NOT NULL DEFAULT 0,
|
|
status text,
|
|
metadata jsonb DEFAULT '{}', "startTime" timestamp(3) NOT NULL,
|
|
created_at timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
|
updated_at timestamp(3) NOT NULL DEFAULT CURRENT_TIMESTAMP
|
|
)""")
|
|
connection.execute((_MIGRATIONS / "20261103010000_spendlogs_accounting_delivery/migration.sql").read_text())
|
|
yield connection
|
|
|
|
|
|
def _insert(
|
|
connection: psycopg.Connection[tuple[object, ...]], rows: Sequence[Mapping[str, object]], *, ledger: bool = True
|
|
) -> None:
|
|
from psycopg.types.json import Jsonb
|
|
|
|
statement: Final = spend_log_insert_statement(rows)
|
|
with connection.transaction():
|
|
inserted: Final = {
|
|
row[0] for row in connection.execute(statement[0].replace("$1", "%s"), (statement[1],)).fetchall()
|
|
}
|
|
for row in rows:
|
|
if ledger and row["request_id"] in inserted:
|
|
connection.execute(
|
|
'INSERT INTO "LiteLLM_SpendLogDelivery" (delivery_id, request_id, accounting_payload, is_batch) '
|
|
"VALUES (%s, %s, %s, %s)",
|
|
(row["request_id"], row["request_id"], Jsonb(dict(row)), row.get("call_type") == "aretrieve_batch"),
|
|
)
|
|
|
|
|
|
def _with_payload(row: Mapping[str, object]) -> Mapping[str, object]:
|
|
return {**row, "accounting_payload": dict(row)}
|
|
|
|
|
|
def _row(request_id: str) -> Mapping[str, object]:
|
|
return {
|
|
"request_id": request_id,
|
|
"call_type": "acompletion",
|
|
"spend": 0.5,
|
|
"startTime": "2026-11-02T12:00:00+00:00",
|
|
"metadata": '{"accounting_savings":{"compression":0.2}}',
|
|
}
|
|
|
|
|
|
@pytest.mark.parametrize("call_type", ["acompletion", "aretrieve_batch"])
|
|
@pytest.mark.parametrize("mode", ["local", "url", "disabled"])
|
|
@pytest.mark.parametrize("partitioned", [False, True])
|
|
@pytest.mark.asyncio
|
|
async def test_delivery_first_acceptance_deduplicates_all_storage_modes(
|
|
delivery_database, delivery_client, mode, partitioned, call_type
|
|
):
|
|
from types import SimpleNamespace
|
|
|
|
from litellm.repositories.table_repositories import SpendLogsRepository
|
|
|
|
if partitioned:
|
|
delivery_database.execute('ALTER TABLE "LiteLLM_SpendLogs" RENAME TO spend_template')
|
|
delivery_database.execute(
|
|
'CREATE TABLE "LiteLLM_SpendLogs" (LIKE spend_template INCLUDING DEFAULTS INCLUDING CONSTRAINTS, PRIMARY KEY(request_id, "startTime")) PARTITION BY RANGE ("startTime")'
|
|
)
|
|
delivery_database.execute('CREATE TABLE "LiteLLM_SpendLogs_p20261102" PARTITION OF "LiteLLM_SpendLogs" DEFAULT')
|
|
delivery_database.execute("DROP TABLE spend_template")
|
|
repository = SpendLogsRepository(SimpleNamespace(db=delivery_client, writer_db=delivery_client))
|
|
row = {**_with_payload({**_row("shared"), "call_type": call_type}), "accounting_log_disabled": mode == "disabled"}
|
|
results = await asyncio.gather(
|
|
repository.insert_with_deliveries((row,), write_spend_logs=mode != "url"),
|
|
repository.insert_with_deliveries(
|
|
({**row, "startTime": "2026-11-03T12:00:00"},), write_spend_logs=mode != "url"
|
|
),
|
|
)
|
|
assert sorted(results) == [0, 1]
|
|
assert delivery_database.execute('SELECT COUNT(*) FROM "LiteLLM_SpendLogs"').fetchone() == (int(mode == "local"),)
|
|
assert delivery_database.execute(
|
|
"SELECT accounting_payload->>'spend', state::text FROM \"LiteLLM_SpendLogDelivery\""
|
|
).fetchall() == [("0.5", "pending")]
|
|
if partitioned:
|
|
delivery_database.execute('DROP TABLE "LiteLLM_SpendLogs_p20261102"')
|
|
else:
|
|
delivery_database.execute('DELETE FROM "LiteLLM_SpendLogs"')
|
|
|
|
async def apply(transaction, deliveries):
|
|
await transaction.execute_raw("UPDATE totals SET spend = spend + $1::float8", float(deliveries[0].row["spend"]))
|
|
|
|
worker = AccountingDeliveryRepository(delivery_client)
|
|
assert (await worker.apply_pending(apply)).applied == 1
|
|
assert (await worker.apply_pending(apply)).applied == 0
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.5,)
|
|
|
|
|
|
@pytest.mark.parametrize("mode", ["local", "url", "disabled"])
|
|
@pytest.mark.asyncio
|
|
async def test_duplicate_rows_in_one_flush_keep_first_accounting_payload(delivery_database, delivery_client, mode):
|
|
from types import SimpleNamespace
|
|
|
|
from litellm.repositories.table_repositories import SpendLogsRepository
|
|
|
|
repository = SpendLogsRepository(SimpleNamespace(db=delivery_client, writer_db=delivery_client))
|
|
first = {**_with_payload(_row("repeated")), "accounting_log_disabled": mode == "disabled"}
|
|
later = {
|
|
**_with_payload({**_row("repeated"), "spend": 9.0, "startTime": "2026-11-03T12:00:00+00:00"}),
|
|
"accounting_log_disabled": mode == "disabled",
|
|
}
|
|
assert await repository.insert_with_deliveries((first, later), write_spend_logs=mode != "url") == 1
|
|
assert delivery_database.execute('SELECT accounting_payload FROM "LiteLLM_SpendLogDelivery"').fetchone() == (
|
|
first["accounting_payload"],
|
|
)
|
|
assert delivery_database.execute('SELECT spend FROM "LiteLLM_SpendLogs"').fetchall() == (
|
|
[(first["spend"],)] if mode == "local" else []
|
|
)
|
|
|
|
|
|
@pytest.mark.parametrize(("legacy_spend", "accepted"), [(0.5, 0), (0.0, 1)])
|
|
@pytest.mark.asyncio
|
|
async def test_batch_charged_before_the_ledger_is_not_charged_again(
|
|
delivery_database, delivery_client, legacy_spend, accepted
|
|
):
|
|
from types import SimpleNamespace
|
|
|
|
from litellm.repositories.table_repositories import SpendLogsRepository
|
|
|
|
batch: Final = {**_row("batch_legacy"), "call_type": "aretrieve_batch", "status": "success"}
|
|
_insert(delivery_database, ({**batch, "spend": legacy_spend},), ledger=False)
|
|
repository = SpendLogsRepository(SimpleNamespace(db=delivery_client, writer_db=delivery_client))
|
|
assert await repository.insert_with_deliveries((_with_payload({**batch, "spend": 1.0}),)) == accepted
|
|
assert delivery_database.execute('SELECT COUNT(*) FROM "LiteLLM_SpendLogDelivery"').fetchone() == (accepted,)
|
|
assert delivery_database.execute('SELECT spend FROM "LiteLLM_SpendLogs"').fetchall() == [(legacy_spend,)]
|
|
|
|
|
|
@pytest.mark.parametrize("table", ["LiteLLM_SpendLogs", "LiteLLM_SpendLogDelivery"])
|
|
@pytest.mark.asyncio
|
|
async def test_transaction_a_rolls_back_payload_and_log_on_failure(delivery_database, delivery_client, table):
|
|
from types import SimpleNamespace
|
|
|
|
from prisma.errors import PrismaError
|
|
from psycopg import sql
|
|
|
|
from litellm.repositories.table_repositories import SpendLogsRepository
|
|
|
|
delivery_database.execute(
|
|
sql.SQL("ALTER TABLE {} ADD CONSTRAINT reject_poison CHECK (request_id <> 'poison')").format(
|
|
sql.Identifier(table)
|
|
)
|
|
)
|
|
repository = SpendLogsRepository(SimpleNamespace(db=delivery_client, writer_db=delivery_client))
|
|
with pytest.raises(PrismaError):
|
|
await repository.insert_with_deliveries(tuple(_with_payload(_row(name)) for name in ("good", "poison")))
|
|
assert delivery_database.execute('SELECT COUNT(*) FROM "LiteLLM_SpendLogDelivery"').fetchone() == (0,)
|
|
assert delivery_database.execute('SELECT COUNT(*) FROM "LiteLLM_SpendLogs"').fetchone() == (0,)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_worker_rolls_back_destinations_and_parks_only_isolated_failure(
|
|
delivery_database: psycopg.Connection[tuple[object, ...]],
|
|
) -> None:
|
|
url: Final = urlunsplit(urlsplit(os.environ["DATABASE_URL"])._replace(path=f"/{delivery_database.info.dbname}"))
|
|
client: Final = Prisma(datasource={"url": url})
|
|
delivery_database.execute("CREATE TABLE totals (spend float8 NOT NULL)")
|
|
delivery_database.execute("INSERT INTO totals VALUES (0)")
|
|
_insert(delivery_database, (_row("bad"), _row("good")))
|
|
|
|
async def apply(transaction: LedgerTransaction, rows: tuple[DeliveryProjection, ...]) -> None:
|
|
await transaction.execute_raw(
|
|
"UPDATE totals SET spend = spend + $1::float8", sum(float(str(row.row["spend"])) for row in rows)
|
|
)
|
|
if any(row.row["request_id"] == "bad" for row in rows):
|
|
raise ValueError("Invalid projected destination")
|
|
|
|
await client.connect()
|
|
try:
|
|
repository: Final = AccountingDeliveryRepository(client)
|
|
failed: Final = await repository.apply_pending(apply, max_attempts=2)
|
|
assert failed.failed == 2
|
|
assert failed.parked == ()
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.0,)
|
|
assert delivery_database.execute(
|
|
'SELECT DISTINCT state::text, attempts FROM "LiteLLM_SpendLogDelivery"'
|
|
).fetchall() == [("pending", 1)]
|
|
delivery_database.execute('UPDATE "LiteLLM_SpendLogDelivery" SET next_attempt_at = CURRENT_TIMESTAMP')
|
|
isolated: Final = await repository.apply_pending(apply, max_attempts=2)
|
|
assert isolated.parked == ("bad",)
|
|
applied: Final = await repository.apply_pending(apply, max_attempts=2)
|
|
assert applied.applied == 1
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.5,)
|
|
assert delivery_database.execute(
|
|
'SELECT request_id, state::text FROM "LiteLLM_SpendLogDelivery" ORDER BY request_id'
|
|
).fetchall() == [("bad", "parked"), ("good", "applied")]
|
|
assert (await repository.apply_pending(apply)).applied == 0
|
|
finally:
|
|
await client.disconnect()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_a_locked_database_never_parks_a_valid_delivery(delivery_database, delivery_client):
|
|
_insert(delivery_database, (_row("valid"),))
|
|
|
|
async def apply(transaction, deliveries):
|
|
await transaction.execute_raw("SET LOCAL lock_timeout = '50ms'")
|
|
await transaction.execute_raw(
|
|
"UPDATE totals SET spend = spend + $1::float8", sum(float(row.row["spend"]) for row in deliveries)
|
|
)
|
|
|
|
repository = AccountingDeliveryRepository(delivery_client)
|
|
with psycopg.connect(
|
|
urlunsplit(urlsplit(os.environ["DATABASE_URL"])._replace(path=f"/{delivery_database.info.dbname}"))
|
|
) as locker:
|
|
locker.execute("LOCK TABLE totals IN ACCESS EXCLUSIVE MODE")
|
|
for _ in range(4):
|
|
result = await repository.apply_pending(apply, max_attempts=2)
|
|
assert result.failed == 1 and result.parked == ()
|
|
delivery_database.execute('UPDATE "LiteLLM_SpendLogDelivery" SET next_attempt_at = CURRENT_TIMESTAMP')
|
|
assert delivery_database.execute('SELECT state::text, attempts FROM "LiteLLM_SpendLogDelivery"').fetchall() == [
|
|
("pending", 4)
|
|
]
|
|
assert (await repository.apply_pending(apply, max_attempts=2)).applied == 1
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.5,)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_an_unreachable_database_reported_as_a_data_error_never_parks_a_delivery(
|
|
delivery_database, delivery_client
|
|
):
|
|
from prisma.errors import DataError
|
|
|
|
_insert(delivery_database, (_row("valid"),))
|
|
|
|
async def unreachable(transaction, deliveries):
|
|
raise DataError(data={"user_facing_error": {"message": "Can't reach database server at `db`:`5432`"}})
|
|
|
|
repository = AccountingDeliveryRepository(delivery_client)
|
|
for _ in range(3):
|
|
assert (await repository.apply_pending(unreachable, max_attempts=2)).parked == ()
|
|
delivery_database.execute('UPDATE "LiteLLM_SpendLogDelivery" SET next_attempt_at = CURRENT_TIMESTAMP')
|
|
assert delivery_database.execute('SELECT state::text, attempts FROM "LiteLLM_SpendLogDelivery"').fetchall() == [
|
|
("pending", 3)
|
|
]
|
|
|
|
|
|
@pytest_asyncio.fixture(loop_scope="function")
|
|
async def delivery_client(delivery_database: psycopg.Connection[tuple[object, ...]]) -> AsyncIterator[Prisma]:
|
|
url: Final = urlunsplit(urlsplit(os.environ["DATABASE_URL"])._replace(path=f"/{delivery_database.info.dbname}"))
|
|
client: Final = Prisma(datasource={"url": url})
|
|
delivery_database.execute("CREATE TABLE totals (spend float8 NOT NULL)")
|
|
delivery_database.execute("INSERT INTO totals VALUES (0)")
|
|
await client.connect()
|
|
try:
|
|
yield client
|
|
finally:
|
|
await client.disconnect()
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_two_workers_claim_disjoint_rows_and_apply_once(
|
|
delivery_database: psycopg.Connection[tuple[object, ...]],
|
|
delivery_client: Prisma,
|
|
) -> None:
|
|
_insert(delivery_database, (_row("first"), _row("second")))
|
|
entered: Final[asyncio.Queue[str]] = asyncio.Queue()
|
|
release: Final = asyncio.Event()
|
|
|
|
async def apply(transaction: LedgerTransaction, rows: tuple[DeliveryProjection, ...]) -> None:
|
|
assert len(rows) == 1
|
|
await entered.put(str(rows[0].row["request_id"]))
|
|
await release.wait()
|
|
await transaction.execute_raw("UPDATE totals SET spend = spend + $1::float8", float(str(rows[0].row["spend"])))
|
|
|
|
repository: Final = AccountingDeliveryRepository(delivery_client)
|
|
first: Final = asyncio.create_task(repository.apply_pending(apply, batch_size=1))
|
|
first_id: Final = await asyncio.wait_for(entered.get(), timeout=10)
|
|
second: Final = asyncio.create_task(repository.apply_pending(apply, batch_size=1))
|
|
try:
|
|
second_id: Final = await asyncio.wait_for(entered.get(), timeout=10)
|
|
assert first_id != second_id
|
|
finally:
|
|
release.set()
|
|
results: Final = await asyncio.gather(first, second)
|
|
assert sum(result.applied for result in results) == 2
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (1.0,)
|
|
assert delivery_database.execute(
|
|
"SELECT COUNT(*) FROM \"LiteLLM_SpendLogDelivery\" WHERE state = 'applied'"
|
|
).fetchone() == (2,)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_cancelled_worker_rolls_back_and_restart_applies_once(
|
|
delivery_database: psycopg.Connection[tuple[object, ...]],
|
|
delivery_client: Prisma,
|
|
) -> None:
|
|
_insert(delivery_database, (_row("restart"),))
|
|
wrote: Final = asyncio.Event()
|
|
hold: Final = asyncio.Event()
|
|
|
|
async def interrupted(transaction: LedgerTransaction, rows: tuple[DeliveryProjection, ...]) -> None:
|
|
await transaction.execute_raw("UPDATE totals SET spend = spend + $1::float8", float(str(rows[0].row["spend"])))
|
|
wrote.set()
|
|
await hold.wait()
|
|
|
|
repository: Final = AccountingDeliveryRepository(delivery_client)
|
|
task: Final = asyncio.create_task(repository.apply_pending(interrupted))
|
|
await asyncio.wait_for(wrote.wait(), timeout=10)
|
|
task.cancel()
|
|
with pytest.raises(asyncio.CancelledError):
|
|
await task
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.0,)
|
|
assert delivery_database.execute('SELECT state::text, attempts FROM "LiteLLM_SpendLogDelivery"').fetchone() == (
|
|
"pending",
|
|
0,
|
|
)
|
|
|
|
async def complete(transaction: LedgerTransaction, rows: tuple[DeliveryProjection, ...]) -> None:
|
|
await transaction.execute_raw("UPDATE totals SET spend = spend + $1::float8", float(str(rows[0].row["spend"])))
|
|
|
|
assert (await repository.apply_pending(complete)).applied == 1
|
|
assert (await repository.apply_pending(complete)).applied == 0
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.5,)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_worker_applies_accounting_after_spend_logs_are_deleted(
|
|
delivery_database: psycopg.Connection[tuple[object, ...]], delivery_client: Prisma
|
|
) -> None:
|
|
_insert(delivery_database, (_row("removed"),))
|
|
delivery_database.execute('DELETE FROM "LiteLLM_SpendLogs"')
|
|
|
|
async def apply(transaction: LedgerTransaction, rows: tuple[DeliveryProjection, ...]) -> None:
|
|
await transaction.execute_raw("UPDATE totals SET spend = spend + $1::float8", float(str(rows[0].row["spend"])))
|
|
|
|
repository: Final = AccountingDeliveryRepository(delivery_client)
|
|
assert (await repository.apply_pending(apply)).applied == 1
|
|
assert (await repository.apply_pending(apply)).applied == 0
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.5,)
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_receipt_cleanup_keeps_pending_and_parked_rows_and_reports_lag(
|
|
delivery_database: psycopg.Connection[tuple[object, ...]], delivery_client: Prisma
|
|
) -> None:
|
|
_insert(delivery_database, tuple(_row(state) for state in ("pending", "parked", "applied")))
|
|
delivery_database.execute(
|
|
'UPDATE "LiteLLM_SpendLogDelivery" '
|
|
'SET state = request_id::"AccountingDeliveryState", '
|
|
"created_at = CURRENT_TIMESTAMP - INTERVAL '40 days', "
|
|
"applied_at = CURRENT_TIMESTAMP - INTERVAL '40 days'"
|
|
)
|
|
repository = AccountingDeliveryRepository(delivery_client)
|
|
assert await repository.cleanup_applied() == 1
|
|
metrics = await repository.metrics()
|
|
assert metrics.pending_count == 1
|
|
assert metrics.parked_count == 1
|
|
assert metrics.oldest_pending_seconds >= 40 * 86400
|
|
assert delivery_database.execute(
|
|
'SELECT request_id FROM "LiteLLM_SpendLogDelivery" ORDER BY request_id'
|
|
).fetchall() == [("parked",), ("pending",)]
|
|
|
|
|
|
@pytest.mark.parametrize("delivery_database", ["public", "ledger_tenant"], indirect=True)
|
|
@pytest.mark.parametrize("state", [None, "pending", "parked", "applied"])
|
|
def test_delivery_migration_replay_preserves_ledger_receipts(delivery_database, state):
|
|
if state is not None:
|
|
_insert(delivery_database, (_row("replay"),))
|
|
delivery_database.execute(
|
|
'UPDATE "LiteLLM_SpendLogDelivery" SET state=%s, attempts=3, last_error=%s',
|
|
(state, "retained failure"),
|
|
)
|
|
before = delivery_database.execute('SELECT * FROM "LiteLLM_SpendLogDelivery"').fetchall()
|
|
migration = (_MIGRATIONS / "20261103010000_spendlogs_accounting_delivery/migration.sql").read_text()
|
|
delivery_database.execute(migration)
|
|
delivery_database.execute(migration)
|
|
assert delivery_database.execute('SELECT * FROM "LiteLLM_SpendLogDelivery"').fetchall() == before
|
|
assert delivery_database.execute('SELECT count(*) FROM "LiteLLM_SpendLogs"').fetchone() == (int(state is not None),)
|
|
|
|
|
|
@pytest.mark.parametrize("state", ["pending", "parked"])
|
|
def test_delivery_migration_preserves_unused_old_receipts(state):
|
|
with scratch_database("spendlogs_old_delivery") as url, psycopg.connect(url, autocommit=True) as connection:
|
|
connection.execute((_MIGRATIONS / "20261027000000_add_accounting_ledger_tables/migration.sql").read_text())
|
|
connection.execute('CREATE TABLE "LiteLLM_SpendLogs" (request_id text PRIMARY KEY)')
|
|
connection.execute(
|
|
'INSERT INTO "LiteLLM_AccountingDelivery" (event_id, target, version, state) VALUES (%s, %s, 1, %s)',
|
|
("old-event", "LiteLLM_UserTable", state),
|
|
)
|
|
connection.execute((_MIGRATIONS / "20261103010000_spendlogs_accounting_delivery/migration.sql").read_text())
|
|
assert connection.execute('SELECT event_id, state::text FROM "LiteLLM_AccountingDelivery"').fetchall() == [
|
|
("old-event", state)
|
|
]
|
|
assert connection.execute('SELECT COUNT(*) FROM "LiteLLM_SpendLogDelivery"').fetchone() == (0,)
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
("payload", "huge_state"),
|
|
[
|
|
(
|
|
"jsonb_set(accounting_payload, '{metadata}', ('{\"tokens_saved\": ' || repeat('9', 401) || '}')::jsonb)",
|
|
"applied",
|
|
),
|
|
(
|
|
"jsonb_set(accounting_payload, '{metadata}', ('{\"tokens_saved\": ' || repeat('9', 5000) || '}')::jsonb)",
|
|
"parked",
|
|
),
|
|
("'[]'::jsonb", "parked"),
|
|
],
|
|
ids=["beyond_float", "beyond_python_int_parse", "not_an_object"],
|
|
)
|
|
@pytest.mark.asyncio
|
|
async def test_a_payload_the_engine_cannot_decode_never_blocks_other_deliveries(
|
|
delivery_database, delivery_client, payload, huge_state
|
|
):
|
|
_insert(delivery_database, (_row("huge"), _row("good")))
|
|
delivery_database.execute(
|
|
f'UPDATE "LiteLLM_SpendLogDelivery" SET accounting_payload = {payload} WHERE request_id = $$huge$$'
|
|
)
|
|
|
|
async def apply(transaction, deliveries):
|
|
await transaction.execute_raw(
|
|
"UPDATE totals SET spend = spend + $1::float8", sum(float(row.row["spend"]) for row in deliveries)
|
|
)
|
|
|
|
repository = AccountingDeliveryRepository(delivery_client)
|
|
results = [await repository.apply_pending(apply, max_attempts=1) for _ in range(3)]
|
|
_insert(delivery_database, (_row("later"),))
|
|
results.append(await repository.apply_pending(apply, max_attempts=1))
|
|
|
|
applied: Final = 3 if huge_state == "applied" else 2
|
|
assert sum(result.applied for result in results) == applied
|
|
assert delivery_database.execute("SELECT spend FROM totals").fetchone() == (0.5 * applied,)
|
|
assert delivery_database.execute(
|
|
'SELECT request_id, state::text FROM "LiteLLM_SpendLogDelivery" ORDER BY request_id'
|
|
).fetchall() == [("good", "applied"), ("huge", huge_state), ("later", "applied")]
|
|
|
|
|
|
@pytest.mark.asyncio
|
|
async def test_retention_keeps_batch_receipts_indefinitely(delivery_database, delivery_client):
|
|
_insert(delivery_database, (_row("normal"), {**_row("batch"), "call_type": "aretrieve_batch"}))
|
|
delivery_database.execute(
|
|
"UPDATE \"LiteLLM_SpendLogDelivery\" SET state='applied', applied_at=CURRENT_TIMESTAMP - INTERVAL '400 days'"
|
|
)
|
|
repository = AccountingDeliveryRepository(delivery_client)
|
|
assert await repository.cleanup_applied() == 1
|
|
assert await repository.cleanup_applied() == 0
|
|
assert delivery_database.execute('SELECT request_id FROM "LiteLLM_SpendLogDelivery"').fetchall() == [("batch",)]
|
|
metrics = await repository.metrics()
|
|
assert metrics.batch_count == 1
|
|
assert metrics.table_bytes > 0
|
|
|
|
|
|
@pytest.mark.parametrize("disabled", [False, True])
|
|
@pytest.mark.asyncio
|
|
async def test_ledger_batch_retrieves_charge_once_in_every_logging_mode(
|
|
delivery_database, delivery_client, monkeypatch, disabled
|
|
):
|
|
from types import SimpleNamespace
|
|
|
|
from litellm.proxy import proxy_server
|
|
from litellm.proxy.db.db_spend_update_writer import DBSpendUpdateWriter
|
|
from litellm.proxy.utils import jsonify_object
|
|
|
|
completed = {
|
|
**_row("batch_completed"),
|
|
"call_type": "aretrieve_batch",
|
|
"status": "success",
|
|
"spend": 0.5,
|
|
"metadata": '{"status":"success","private":"request detail"}',
|
|
"accounting_payload": {
|
|
"request_id": "batch_completed",
|
|
"call_type": "aretrieve_batch",
|
|
"status": "success",
|
|
"spend": 0.5,
|
|
"metadata": {"accounting_savings": {"compression": 0.2}},
|
|
},
|
|
}
|
|
monkeypatch.setattr(proxy_server, "general_settings", {"spend_logs_metadata_fields": {"include": ["status"]}})
|
|
client = SimpleNamespace(db=delivery_client, writer_db=delivery_client, jsonify_object=jsonify_object)
|
|
writer = DBSpendUpdateWriter()
|
|
first = await writer._claim_settle_once_cost_row(
|
|
payload=completed, prisma_client=client, disable_spend_logs=disabled
|
|
)
|
|
again = await writer._claim_settle_once_cost_row(
|
|
payload={**completed, "startTime": "2026-11-05T12:00:00+00:00"},
|
|
prisma_client=client,
|
|
disable_spend_logs=disabled,
|
|
)
|
|
assert (first, again) == (True, False)
|
|
assert delivery_database.execute('SELECT spend, metadata FROM "LiteLLM_SpendLogs"').fetchall() == (
|
|
[] if disabled else [(0.5, {"status": "success"})]
|
|
)
|
|
assert delivery_database.execute(
|
|
'SELECT delivery_id, accounting_payload FROM "LiteLLM_SpendLogDelivery"'
|
|
).fetchall() == [("batch_completed", completed["accounting_payload"])]
|
|
|
|
|
|
@pytest.mark.parametrize("setting", ["retention_days", "batch_size"])
|
|
@pytest.mark.asyncio
|
|
async def test_invalid_cleanup_settings_preserve_receipts(delivery_database, delivery_client, setting):
|
|
_insert(delivery_database, (_row("protected"),))
|
|
with pytest.raises(ValueError, match="positive"):
|
|
await AccountingDeliveryRepository(delivery_client).cleanup_applied(**{setting: 0})
|
|
assert delivery_database.execute('SELECT request_id, state::text FROM "LiteLLM_SpendLogDelivery"').fetchall() == [
|
|
("protected", "pending")
|
|
]
|
|
|
|
|
|
@pytest.mark.parametrize("setting", ["max_attempts", "batch_size"])
|
|
@pytest.mark.asyncio
|
|
async def test_invalid_worker_settings_preserve_pending_work(delivery_database, delivery_client, setting):
|
|
from unittest.mock import AsyncMock
|
|
|
|
_insert(delivery_database, (_row("protected"),))
|
|
apply = AsyncMock()
|
|
with pytest.raises(ValueError, match="positive"):
|
|
await AccountingDeliveryRepository(delivery_client).apply_pending(apply, **{setting: 0})
|
|
apply.assert_not_awaited()
|
|
assert delivery_database.execute(
|
|
'SELECT request_id, state::text, attempts FROM "LiteLLM_SpendLogDelivery"'
|
|
).fetchall() == [("protected", "pending", 0)]
|