* feat(budgets): replace the python budget hold and settle engine with a rust engine admitted at auth
* fix(budgets): keep native_call_id out of the budget reservation like main
* fix(proxy): keep the team member budget cap and reset for members given default team member models
* test(proxy): mark the new test's locals Final
* fix(proxy): give models-only members the team's reset window and leave the cap to the team
* fix(proxy): move members on the team's reset window when the team's window changes
* fix(proxy): move members before the team window write, skip an unchanged window, cover the first member budget
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A typed read of the same in-memory payload returns a shallow copy of the model the previous read validated instead of running model_validate again. Writes, deletes and flushes drop the memo, and the Redis and shared store payloads are unchanged.
* fix(router): free a failed rust router call's request without waiting for a full gc
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): release retained retry errors after successful calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hold the logging object weakly in the detached stream failure hook
The closure armed by _arm_detached_stream_failure_hook is stored on the
logging object and captured the logging object, the request data that
contains it, and the processing object that holds that data. Every
streamed /v1/messages request therefore ended in a reference cycle and
stayed in memory until a full cyclic GC pass instead of being freed by
reference counting when the stream finished.
Capture the logging object through a weakref, a copy of the request data
without the litellm_logging_obj key, and the route type as a value. The
hook adds the live logging object back to the request data when it fires.
All three callers reach the hook through the logging object, so it is
always alive at that point.
Add a regression test that streams a request to completion, drops every
reference, and checks with the cyclic collector disabled that the logging
object is gone.
Fixes#51277
* test(proxy): cover finished streamed /v1/messages calls being freed without a gc pass
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): annotate the finished-stream gc test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): split the freed-calls assert message under the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(admin-mcp): restore the budget route operation ids the connector pins
* test(admin-mcp): skip the operation id check below Python 3.12 like its siblings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dynamic_rate_limiter): let customer orgs and keys borrow idle model capacity and split class shares
Replace the per-team borrowing and class share booleans with model_limit_borrowing_config.borrow_idle_capacity
and priority_class_share_config.split_equally_among_active, which list the entity types they apply to.
Use one borrow cap field, model_limit_max_borrow_multiple, on team metadata, key metadata and the customer
organization budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): size holds with the tightest share, fail closed on websocket turns, guard lookalike team ids and the borrow cap
One token estimate now covers every share a request holds, using the tightest TPM share's output cap.
A team id spelled like a key or customer organization holder falls back to the class pool.
model_limit_max_borrow_multiple in key or team metadata is changed by proxy admins only.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): keep a team's key on the team's class share slice and reset websocket share state per turn
A team listed for the class share split now holds the slice ahead of customer headers, which the caller chooses.
A websocket turn clears the shares of borrowers that do not apply on its model, so it is not settled on an earlier model's share.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dynamic_rate_limiter): read the borrow cap from model_limit_borrowing_config.max_borrow_multiple instead of a budget table column
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): a fallback target a borrower has no limit on no longer charges its share on the failed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): keep a deployment's per-second video price over the catalog row's flat price
* fix(cost): drop the superseded flat video price from the registration kept for reloads
* test(cost): derive the per-second rate from the catalog row so the regression holds at any vendor price
* fix(cost): keep the flat video price when only a resolution tier is priced per second
* test(integration): assert the body sent to fal in the per-second price test
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nate Armstrong <narmstrong@berri.ai>
* fix(router): stop the rust router keeping every finished request in memory until a full gc
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove finished /v1/messages and chat calls free their log record without a gc pass
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read the memory endpoint with typed helpers instead of a type ignore
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): bill a video request without a resolution at the row's default rate
* test(fal_ai): drop the assertion that two fal rates differ
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): put extra_body fields inside modelInput for async-invoke embeddings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep clientRequestToken and tags at the top level of async-invoke requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): cover clientRequestToken staying top-level in the async-invoke wire test
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(perplexity): bill cached prompt tokens at the cache read rate when the response has no cost
* fix(perplexity): bill cached tokens at the off-peak cache read rate when the model has no standard one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): send versioned seedream and flux-pro image models to the endpoint they name
* fix(fal_ai): reject dot segments in the model path and tighten the versioned endpoint tests
* test(fal_ai): match the dot segment error message
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run the /fal_ai passthrough price gate on the path sent upstream
The gate checked the raw request path for /requests/ and priced the raw path, so a path that
collapses to a submit URL upstream (a /requests/.. dot segment, or an encoded ? or # tail)
skipped the gate and reached fal on the proxy's key
* test(proxy): use an endpoint with no catalog row in the fal passthrough gate test
* fix(proxy): price the fal passthrough gate on the encoded upstream path
* fix(proxy): strip the fal queue base prefix in the form it is sent upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep an exhausted key on model listing and token counting routes
The header-only preflight added in #51428 ran the rate limit check on every LLM API route, so a key that had used up its rpm limit got 429 on GET /v1/models, GET /models and POST /utils/token_counter, routes whose handlers never run the limiter. The preflight now skips the limit check on GET requests, which carry no body to avoid reading, and on unbilled routes.
* fix(proxy): still refuse a GET that declares a body before reading it
A GET can carry a body, so the limit check is only skipped for a GET that declares none. A metered route such as /v1/batches keeps its 429 before a declared body is buffered.
* test(proxy): fail the preflight test on any outcome other than admitted or 429
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): join default teams on the team member budget instead of a private one
* fix(proxy): keep an explicit per-user team budget and drop only the default team one
* fix(proxy): drop a default team's per-user cap only when the team has a member budget
* fix(proxy): chunk the default team lookup
* fix(proxy): type the team metadata read
* fix(proxy): validate the team metadata instead of casting it
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(auth): read token model fields from the class instead of deprecated __fields__
* test(s3): assert 403 retry backoff on the logged wait instead of wall-clock gaps
run_async_value polls the future once on the caller's thread with the GIL released and the runtime entered. A future that completes returns an already settled asyncio future on the caller's loop, and only a pending one is handed to future_into_py.
cleanup_batch waited up to 120s for a managed batch that was still
cancelling to reach a terminal state. Across 55 saved litellm-e2e builds it
reached one in time once, so test_batch_lifecycle[openai-unified] spent 126s
and TestBatchSecondHop 140s on every run, and both then left the input file
in place anyway. A managed batch still cancelling is now left alone, the same
as a raw provider batch, and the input file cleanup still reports the
leftover.
TestBatchTerminalState listed the marker batches every 10s for up to 300s.
Each e2e stack starts with an empty database, so the list only ever holds
this run's own marker, which never completed inside the window in any of
those builds, and the test took 303s for its documented vacuous pass. It now
lists once.
The litellm-proxy manifest still said 1.109.0 after litellm moved to 1.110.0 and 1.111.0, because version bumps only rewrote the root pyproject.toml. The Docker image builds that manifest unchanged, so ghcr.io/berriai/litellm:v1.111.0-dev.2 reports litellm-proxy 1.109.0, a version that was never published.
Set the manifest and uv.lock to 1.111.0, add the manifest to the commitizen version_files so cz bump rewrites it, and add a test that fails when the two versions or the lock entry disagree.
* fix(spend): support PostgreSQL 12 delivery migrations
The SpendLogDelivery migration defaulted delivery_id to gen_random_uuid(), which
PostgreSQL 12 lacks without pgcrypto, so migrate deploy failed there. The writer
always supplies delivery_id, so the column no longer needs a database default.
Databases that already applied the original migration drop the default in a
follow-up migration without rewriting rows.
* test(spend): apply the ledger enum migration before the delivery migration tests
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
test_batch_lifecycle[azure-provider_fallback] has failed in most scheduled
e2e runs since the suite moved from 8 to 12 xdist workers. Batch create
returned 404 Resource not found from Azure after a successful upload.
A batch call that names a provider and no model takes its credentials from
the first deployment of that provider the key may use. The test used an
all-models key, so on the shared proxy that was whichever Azure deployment
got registered first. With 12 workers the realtime suite starts in the same
instant as the batches suite, and its session-long azure-realtime deployment
carries api_version 2025-08-28. When it registers before azure-batch, the
batch create goes out with that api_version and Azure answers 404. The
upload still works because the file-create route overrides the borrowed
credentials with files_settings. The file delete on cleanup does not, so the
model_param and provider_fallback cleanups were also sent with the wrong
api_version and their 404 was read as already deleted.
The lifecycle key now belongs to a team that may only use the batch
deployment under test, so provider-only calls resolve that deployment in any
registration order.
* test(unit): reset module state only when a test changed it
The proxy conftest deep-copied every snapshotted litellm attribute before each
test, and the copy of litellm.model_cost alone was a median 0.12 s for each of
56,031 proxy tests on the runner. The reset now copies an attribute back only
when it no longer equals the snapshot.
The enterprise and MCP server conftests reloaded litellm before every test,
3.27 s and 0.21 s per test. They now use the same reset.
* test(unit): remove six flaky tests
They failed on main without a code change behind them between 2026-11-05 and
2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM
count, three Prometheus metrics server port tests and the OpenCode list cost
billing test.
* ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests
proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2
times faster with 8 workers on a 16-core runner in a benchmark run.
windows-sdk and proxy-build-from-pip still run on pushes to main and manual
runs. proxy-build-from-pip installs a published release and never runs the
pull request's code.
* test: move 109 tests that start a process or open a socket into tests/integration
tests/unit is in-process by its own rules, and these tests spawn a Python
interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit
shard time per run, 396 s of it in the MCP client transport tests. Each one now
lives in a *_real_io.py file under the integration directory for its domain and
runs with that directory's group.
The tests themselves are unchanged apart from one repository-root path in the
CLI test.
* test(unit): copy back only the dict entries a test changed
A test module that registers models at import leaves litellm.model_cost four
entries larger than the snapshot before every test in its directory, so the
reset still deep-copied the whole price map for each of them. It now rebuilds a
changed dict from the live entries that still match and copies only the rest.
* ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate
A skipped job passing the gate is what test_unit_passed_gate guards against,
so the gate goes back to requiring success from every job it needs and
windows-sdk and proxy-build-from-pip stop being part of it. They still run
and report on pushes to main and manual runs.
db::Error now reports a kind that matches PrismaDBExceptionHandler's buckets:
unreachable, capacity, deadlock, serialization failure, lock timeout, read-only
after failover, stale cached plan and rejected data. is_unavailable gates
fail-open and is_requeue_safe says whether a failed write provably never
applied
Pools::recycle_after is the native counterpart of recreating the Prisma
client: after a 25006 (the writer endpoint now points at a read-only node) or
"cached plan must not change result type" (a migration changed a table under
a prepared statement), every pooled connection opened before the error is
dropped on its next acquire, so the pool reconnects and prepares fresh
Native failures reached Python as a ValueError, so allow_requests_on_db_unavailable
treated a native outage as a permanent fault and a write on a dropped connection
was requeued. The bridge now raises NativeDatabaseFailure with the kind and the
Rust answers for is_unavailable and is_requeue_safe, and the two Python gates
return them
crates/db gains observe(), which records one event per round trip with the
call type, table and operation a Prisma call reports. NativeDatabase forwards
those events to a Python ServiceLoggingEvents that Database.open injects, and
it calls the same ServiceTypes.DB success and failure hooks as db_span on the
proxy's event loop. A native health_check, which pings one pooled connection
without running SQL (call type health_check), is the first observed operation
The native pool is lazy, so a pool that could not connect stayed silent until
the first ported query failed on a request path. Database.open now pings it
once in the background and logs a warning on failure, without failing or
delaying boot
The crate AGENTS.md records the Prometheus and OTel names a Rust host sink must match
When DATABASE_URL_READ_REPLICA is set, NativeDatabase opens a reader pool
resolved from that URL, with its own token rotation. Pools exposes writer(),
read() and read_after_write(), matching RoutingPrismaWrapper and
WriterPinnedClient. read() prefers the replica and read_after_write() the
writer. Each falls back to the other side when its pool cannot connect and
skips the failed side for 30 seconds, so reads keep working on the replica
while the writer is down
The reader pool takes its share out of the reader's connection_limit the same
way the writer does, with the same hard-coded RUST_RESERVED_CONNECTIONS. While
that is 0 the replica URL is built exactly as before and no native reader pool
opens. A replica connection_limit that is not a positive integer is left for
Prisma to judge, as it always was
NativeDatabase.connect opens a lazy SQLx writer pool from an Endpoint that
Python resolves from the final DATABASE_URL, read the way Prisma reads it:
Unix socket host, sslmode / sslaccept / sslcert with Prisma's verification
rules (prefer + strict asks the server once whether it offers TLS),
sslidentity / sslpassword handed over as PEM, schema, options, pgbouncer mode,
pool_timeout (0 waits forever) and socket_timeout. Options SQLx cannot send
are logged instead of failing boot
ConnectionBudget splits each worker's connection_limit between Prisma and the
native pool, so the budget an operator sized never grows. The Rust share is the
hard-coded RUST_RESERVED_CONNECTIONS and no setting names Rust, SQLx or Prisma.
It is 0 until a ported query needs the pool, so Prisma's URL is exactly what it
was, no native pool opens and nothing connects through SQLx. A connection_limit
that is not a positive integer is never split and goes to Prisma as before.
Once the share is above 0 Prisma keeps at least one connection, and pools are
tagged litellm-prisma and litellm-native in pg_stat_activity. Python keeps
minting RDS IAM and Entra tokens and hands each one to the pool
statement_cache_size sets the native statement cache (Prisma's default 100) and
pgbouncer=true turns it off. sqlx-postgres 0.9.0 still names every statement when
the cache is off, so Statements runs queries unnamed in that case, like Prisma,
until a release carries transact-rs/sqlx#4337. An application_name the operator
set is used on both pools
The SQL query files are pinned to LF so Windows checkouts hash them like .sqlx,
and the crate lists .sqlx in its package include so the core sdist builds offline
crates/db/AGENTS.md tracks parity with the Python Prisma stack
The bridge now links litellm-db, so the editable install compiles its queries.
The Rust DB workflow builds that install from the checked-in .sqlx metadata,
since it runs before the migrations that the online check needs