Commit graph

2297 commits

Author SHA1 Message Date
mateo-berri
e130ab8ca8 Merge remote-tracking branch 'origin/main' into litellm_messages_drop_unsigned_thinking
# Conflicts:
#	litellm/rust/crates/inference-messages/src/handler.rs
2026-11-15 19:53:13 -08:00
Nate Armstrong
8d200a4f98
feat(messages): serve bedrock invoke messages on rust only (#51330) 2026-11-15 16:58:31 -08:00
Kerry Lu
d0bf848d91
fix(dynamic_rate_limiter): charge tokens used while the model was idle against a team's class share (#51952) 2026-11-15 16:49:47 -08:00
Nate Armstrong
2aa57d7e04
feat(budgets): replace the python budget hold and settle engine with a rust engine admitted at auth (#51535)
* feat(budgets): replace the python budget hold and settle engine with a rust engine admitted at auth

* fix(budgets): keep native_call_id out of the budget reservation like main
2026-11-15 16:03:51 -08:00
moe-berri
175993910a
fix(proxy): batch customer organization cascade deletion (#51969) 2026-11-16 08:03:45 +08:00
Kerry Lu
ceb458c929
fix(proxy): reset members given default team member models on the team's budget window (#51960)
* fix(proxy): keep the team member budget cap and reset for members given default team member models

* test(proxy): mark the new test's locals Final

* fix(proxy): give models-only members the team's reset window and leave the cap to the team

* fix(proxy): move members on the team's reset window when the team's window changes

* fix(proxy): move members before the team window write, skip an unchanged window, cover the first member budget

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 16:00:44 -08:00
Nate Armstrong
eb8a2d224e
perf(auth): reuse validated models on UserApiKeyCache memory hits (#51912)
A typed read of the same in-memory payload returns a shallow copy of the model the previous read validated instead of running model_validate again. Writes, deletes and flushes drop the memo, and the Redis and shared store payloads are unchanged.
2026-11-15 15:39:44 -08:00
Mateo Wang
8c97c243fe
fix(router): free a failed rust router call's request without waiting for a full gc (#51963)
* fix(router): free a failed rust router call's request without waiting for a full gc

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(router): release retained retry errors after successful calls

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 17:04:38 -05:00
Mateo Wang
1e3a960a0e
refactor(vertex_ai): remove unreachable VertexAIFilesHandler.afile_content (#51966)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 13:54:05 -08:00
moe-berri
bddb97132b
feat(customers): sort organizations, teams and members by budget limit (#51924) 2026-11-16 05:22:10 +08:00
Kerry Lu
0dee2a8a0d
fix(responses): retry a stream that breaks before output in its own model group (#51953) 2026-11-15 12:19:58 -08:00
Rhythm Arya
e6b673e6e0
fix(proxy): hold the logging object weakly in the detached stream failure hook (#51318)
* fix(proxy): hold the logging object weakly in the detached stream failure hook

The closure armed by _arm_detached_stream_failure_hook is stored on the
logging object and captured the logging object, the request data that
contains it, and the processing object that holds that data. Every
streamed /v1/messages request therefore ended in a reference cycle and
stayed in memory until a full cyclic GC pass instead of being freed by
reference counting when the stream finished.

Capture the logging object through a weakref, a copy of the request data
without the litellm_logging_obj key, and the route type as a value. The
hook adds the live logging object back to the request data when it fires.
All three callers reach the hook through the logging object, so it is
always alive at that point.

Add a regression test that streams a request to completion, drops every
reference, and checks with the cyclic collector disabled that the logging
object is gone.

Fixes #51277

* test(proxy): cover finished streamed /v1/messages calls being freed without a gc pass

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): annotate the finished-stream gc test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): split the freed-calls assert message under the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 15:13:24 -05:00
Kerry Lu
b168ca7ae4
fix(admin-mcp): restore the budget route operation ids the connector pins (#51959)
* fix(admin-mcp): restore the budget route operation ids the connector pins

* test(admin-mcp): skip the operation id check below Python 3.12 like its siblings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 20:11:02 +00:00
Kerry Lu
765af9ab71
fix(count_tokens): return 400 for malformed Anthropic server-tool blocks (#51957)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 20:10:12 +00:00
Kerry Lu
60716572a4
feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705)
* feat(dynamic_rate_limiter): let customer orgs and keys borrow idle model capacity and split class shares

Replace the per-team borrowing and class share booleans with model_limit_borrowing_config.borrow_idle_capacity
and priority_class_share_config.split_equally_among_active, which list the entity types they apply to.
Use one borrow cap field, model_limit_max_borrow_multiple, on team metadata, key metadata and the customer
organization budget.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): size holds with the tightest share, fail closed on websocket turns, guard lookalike team ids and the borrow cap

One token estimate now covers every share a request holds, using the tightest TPM share's output cap.
A team id spelled like a key or customer organization holder falls back to the class pool.
model_limit_max_borrow_multiple in key or team metadata is changed by proxy admins only.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): keep a team's key on the team's class share slice and reset websocket share state per turn

A team listed for the class share split now holds the slice ahead of customer headers, which the caller chooses.
A websocket turn clears the shares of borrowers that do not apply on its model, so it is not settled on an earlier model's share.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(dynamic_rate_limiter): read the borrow cap from model_limit_borrowing_config.max_borrow_multiple instead of a budget table column

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): a fallback target a borrower has no limit on no longer charges its share on the failed model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 12:05:58 -08:00
Kerry Lu
8abcd13a9d
fix(cost): keep a deployment's per-second video price over the catalog row's flat price (#51688)
* fix(cost): keep a deployment's per-second video price over the catalog row's flat price

* fix(cost): drop the superseded flat video price from the registration kept for reloads

* test(cost): derive the per-second rate from the catalog row so the regression holds at any vendor price

* fix(cost): keep the flat video price when only a resolution tier is priced per second

* test(integration): assert the body sent to fal in the per-second price test

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nate Armstrong <narmstrong@berri.ai>
2026-11-15 12:04:41 -08:00
moe-berri
28c892497c
fix(budgets): match window starts to stored millisecond precision (#51921) 2026-11-16 04:02:44 +08:00
moe-berri
92115c26aa
fix(customers): keep uncapped customer spend across scheduled budget resets (#51918) 2026-11-16 04:02:27 +08:00
Mateo Wang
d49706f55d
fix(router): stop the rust router keeping every finished request in memory until a full gc (#51888)
* fix(router): stop the rust router keeping every finished request in memory until a full gc

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): prove finished /v1/messages and chat calls free their log record without a gc pass

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): read the memory endpoint with typed helpers instead of a type ignore

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 11:31:07 -08:00
Kerry Lu
61a1648572
fix(fal_ai): bill a video request without a resolution at the row's default rate (#51665)
* fix(fal_ai): bill a video request without a resolution at the row's default rate

* test(fal_ai): drop the assertion that two fal rates differ

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:51 +08:00
Kerry Lu
45c05c5bea
fix(bedrock): put extra_body fields inside modelInput for async-invoke embeddings (#51676)
* fix(bedrock): put extra_body fields inside modelInput for async-invoke embeddings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): keep clientRequestToken and tags at the top level of async-invoke requests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(bedrock): cover clientRequestToken staying top-level in the async-invoke wire test

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:25 +08:00
Kerry Lu
2d487abfc1
fix(vertex_ai): send extra_body fields as predict parameters for BGE embedding models (#51662)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:21 +08:00
Kerry Lu
19d8e57485
fix(perplexity): bill cached prompt tokens at the cache read rate when the response has no cost (#51669)
* fix(perplexity): bill cached prompt tokens at the cache read rate when the response has no cost

* fix(perplexity): bill cached tokens at the off-peak cache read rate when the model has no standard one

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:13 +08:00
Kerry Lu
ab7af57878
fix(replicate): answer malformed video seconds and size with a 400 or a dropped size instead of a 500 (#51670)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:29:08 +08:00
Kerry Lu
af6056eca1
fix(fal_ai): send versioned seedream and flux-pro image models to the endpoint they name (#51668)
* fix(fal_ai): send versioned seedream and flux-pro image models to the endpoint they name

* fix(fal_ai): reject dot segments in the model path and tighten the versioned endpoint tests

* test(fal_ai): match the dot segment error message

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:28:53 +08:00
Kerry Lu
2899c97601
fix(proxy): run the /fal_ai passthrough price gate on the path sent upstream (#51671)
* fix(proxy): run the /fal_ai passthrough price gate on the path sent upstream

The gate checked the raw request path for /requests/ and priced the raw path, so a path that
collapses to a submit URL upstream (a /requests/.. dot segment, or an encoded ? or # tail)
skipped the gate and reached fal on the proxy's key

* test(proxy): use an endpoint with no catalog row in the fal passthrough gate test

* fix(proxy): price the fal passthrough gate on the encoded upstream path

* fix(proxy): strip the fal queue base prefix in the form it is sent upstream

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:28:31 +08:00
Kerry Lu
063b63de6c
fix(proxy): let an exhausted key keep listing models and counting tokens (#51675)
* fix(proxy): keep an exhausted key on model listing and token counting routes

The header-only preflight added in #51428 ran the rate limit check on every LLM API route, so a key that had used up its rpm limit got 429 on GET /v1/models, GET /models and POST /utils/token_counter, routes whose handlers never run the limiter. The preflight now skips the limit check on GET requests, which carry no body to avoid reading, and on unbilled routes.

* fix(proxy): still refuse a GET that declares a body before reading it

A GET can carry a body, so the limit check is only skipped for a GET that declares none. A metered route such as /v1/batches keeps its 429 before a declared body is buffered.

* test(proxy): fail the preflight test on any outcome other than admitted or 429

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:28:26 +08:00
Kerry Lu
086fc1da3f
fix(proxy): join default teams on the team member budget when the team has one (#51728)
* fix(proxy): join default teams on the team member budget instead of a private one

* fix(proxy): keep an explicit per-user team budget and drop only the default team one

* fix(proxy): drop a default team's per-user cap only when the team has a member budget

* fix(proxy): chunk the default team lookup

* fix(proxy): type the team metadata read

* fix(proxy): validate the team metadata instead of casting it

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:27:57 +08:00
Nate Armstrong
09ce4f50ed
perf(auth): read token model fields from the class instead of deprecated __fields__ (#51901)
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / enterprise-managed-files (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / assert-shard-coverage (push) Waiting to run
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* perf(auth): read token model fields from the class instead of deprecated __fields__

* test(s3): assert 403 retry backoff on the logged wait instead of wall-clock gaps
2026-11-15 07:07:49 +00:00
Nate Armstrong
9e3a82ff03
perf(caching): format dual cache debug logs lazily (#51902)
* perf(caching): format dual cache debug logs lazily

* fix(caching): type dual cache kwargs so lazy log args stay inside the pyright ceiling
2026-11-15 06:50:04 +00:00
Nate Armstrong
98c86f6e7c
perf(streaming): extract usage once per chunk when assembling stream usage (#51903) 2026-11-14 22:31:39 -08:00
Nate Armstrong
d76a7d76c4
perf(host-python): settle ready async values inline instead of spawning on tokio (#51906)
run_async_value polls the future once on the caller's thread with the GIL released and the runtime entered. A future that completes returns an already settled asyncio future on the caller's loop, and only a pending one is handed to future_into_py.
2026-11-15 06:27:55 +00:00
Nate Armstrong
3bcd77954e
refactor(s3-log): move the rust-backed s3 logger from s3_v3 to s3_v2 (#51893) 2026-11-14 22:24:23 -08:00
devin-ai-integration[bot]
b90ac69841
test(integration): record a stack when an owned proxy crashes (#51922)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 06:15:35 +00:00
kenneth-berri
26807a9925
feat(customers): search customer members by ID or alias in the Members tab (#51896) 2026-11-15 04:47:25 +00:00
moe-berri
00313c9e5b
feat(customers): move existing members between teams (#51913) 2026-11-15 12:22:02 +08:00
ishaan-j-berri
61483d8f04
fix(ui): rename Customer Portal to Customer Management (#51891)
* fix(ui): rename Customer Portal to Customer Management

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(ui): polish Customer Management product copy

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* style(ui): format Customer Management test assertions

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-14 21:23:35 -05:00
yujonglee
745886e5ff
fix(packaging): keep the litellm-proxy package version in step with litellm (#51807)
The litellm-proxy manifest still said 1.109.0 after litellm moved to 1.110.0 and 1.111.0, because version bumps only rewrote the root pyproject.toml. The Docker image builds that manifest unchanged, so ghcr.io/berriai/litellm:v1.111.0-dev.2 reports litellm-proxy 1.109.0, a version that was never published.

Set the manifest and uv.lock to 1.111.0, add the manifest to the commitizen version_files so cz bump rewrites it, and add a test that fails when the two versions or the lock entry disagree.
2026-11-14 18:16:47 -08:00
Shivam Rawat
a1ea22e996
feat(otel v2): proxy-level default langfuse_span_scope for key and team destinations (#51726)
* feat(otel v2): proxy-level default langfuse_span_scope for key and team destinations

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): regenerate dashboard API documentation types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): preserve tenant destinations with malformed otel settings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(otel): cover explicit narrow scope and non-Langfuse defaults

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Read only the prefixed Langfuse destination scope environment variable

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-14 15:45:11 -08:00
yuneng-jiang
da55893656
ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864)
* test(unit): reset module state only when a test changed it

The proxy conftest deep-copied every snapshotted litellm attribute before each
test, and the copy of litellm.model_cost alone was a median 0.12 s for each of
56,031 proxy tests on the runner. The reset now copies an attribute back only
when it no longer equals the snapshot.

The enterprise and MCP server conftests reloaded litellm before every test,
3.27 s and 0.21 s per test. They now use the same reset.

* test(unit): remove six flaky tests

They failed on main without a code change behind them between 2026-11-05 and
2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM
count, three Prometheus metrics server port tests and the OpenCode list cost
billing test.

* ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests

proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2
times faster with 8 workers on a 16-core runner in a benchmark run.

windows-sdk and proxy-build-from-pip still run on pushes to main and manual
runs. proxy-build-from-pip installs a published release and never runs the
pull request's code.

* test: move 109 tests that start a process or open a socket into tests/integration

tests/unit is in-process by its own rules, and these tests spawn a Python
interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit
shard time per run, 396 s of it in the MCP client transport tests. Each one now
lives in a *_real_io.py file under the integration directory for its domain and
runs with that directory's group.

The tests themselves are unchanged apart from one repository-root path in the
CLI test.

* test(unit): copy back only the dict entries a test changed

A test module that registers models at import leaves litellm.model_cost four
entries larger than the snapshot before every test in its directory, so the
reset still deep-copied the whole price map for each of them. It now rebuilds a
changed dict from the live entries that still match and copies only the rest.

* ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate

A skipped job passing the gate is what test_unit_passed_gate guards against,
so the gate goes back to requiring success from every job it needs and
windows-sdk and proxy-build-from-pip stop being part of it. They still run
and report on pushes to main and manual runs.
2026-11-14 15:41:30 -08:00
mateo-berri
8c7fb9a474 Merge remote-tracking branch 'origin/main' into litellm_messages_drop_unsigned_thinking
# Conflicts:
#	litellm/rust/crates/llms/src/anthropic/common_utils.rs
#	litellm/rust/crates/llms/src/anthropic/messages/transformation.rs
2026-11-14 15:36:25 -08:00
yujonglee
1bc3190a94
feat(db): classify errors and recycle poisoned pools like Prisma (#49801)
db::Error now reports a kind that matches PrismaDBExceptionHandler's buckets:
unreachable, capacity, deadlock, serialization failure, lock timeout, read-only
after failover, stale cached plan and rejected data. is_unavailable gates
fail-open and is_requeue_safe says whether a failed write provably never
applied

Pools::recycle_after is the native counterpart of recreating the Prisma
client: after a 25006 (the writer endpoint now points at a read-only node) or
"cached plan must not change result type" (a migration changed a table under
a prepared statement), every pooled connection opened before the error is
dropped on its next acquire, so the pool reconnects and prepares fresh

Native failures reached Python as a ValueError, so allow_requests_on_db_unavailable
treated a native outage as a permanent fault and a write on a dropped connection
was requeued. The bridge now raises NativeDatabaseFailure with the kind and the
Rust answers for is_unavailable and is_requeue_safe, and the two Python gates
return them
2026-11-14 15:28:11 -08:00
yujonglee
bd27f4e34c
feat(db): emit DB service events from native operations (#49810)
crates/db gains observe(), which records one event per round trip with the
call type, table and operation a Prisma call reports. NativeDatabase forwards
those events to a Python ServiceLoggingEvents that Database.open injects, and
it calls the same ServiceTypes.DB success and failure hooks as db_span on the
proxy's event loop. A native health_check, which pings one pooled connection
without running SQL (call type health_check), is the first observed operation

The native pool is lazy, so a pool that could not connect stayed silent until
the first ported query failed on a request path. Database.open now pings it
once in the background and logs a warning on failure, without failing or
delaying boot

The crate AGENTS.md records the Prometheus and OTel names a Rust host sink must match
2026-11-14 15:28:10 -08:00
yujonglee
f6c9c70398
feat(db): add an optional native reader pool with explicit routing (#49804)
When DATABASE_URL_READ_REPLICA is set, NativeDatabase opens a reader pool
resolved from that URL, with its own token rotation. Pools exposes writer(),
read() and read_after_write(), matching RoutingPrismaWrapper and
WriterPinnedClient. read() prefers the replica and read_after_write() the
writer. Each falls back to the other side when its pool cannot connect and
skips the failed side for 30 seconds, so reads keep working on the replica
while the writer is down

The reader pool takes its share out of the reader's connection_limit the same
way the writer does, with the same hard-coded RUST_RESERVED_CONNECTIONS. While
that is 0 the replica URL is built exactly as before and no native reader pool
opens. A replica connection_limit that is not a positive integer is left for
Prisma to judge, as it always was
2026-11-14 15:28:10 -08:00
yujonglee
cc8ae16bb4
feat(db): add the SQLx pool foundation with the writer pool (#49798)
NativeDatabase.connect opens a lazy SQLx writer pool from an Endpoint that
Python resolves from the final DATABASE_URL, read the way Prisma reads it:
Unix socket host, sslmode / sslaccept / sslcert with Prisma's verification
rules (prefer + strict asks the server once whether it offers TLS),
sslidentity / sslpassword handed over as PEM, schema, options, pgbouncer mode,
pool_timeout (0 waits forever) and socket_timeout. Options SQLx cannot send
are logged instead of failing boot

ConnectionBudget splits each worker's connection_limit between Prisma and the
native pool, so the budget an operator sized never grows. The Rust share is the
hard-coded RUST_RESERVED_CONNECTIONS and no setting names Rust, SQLx or Prisma.
It is 0 until a ported query needs the pool, so Prisma's URL is exactly what it
was, no native pool opens and nothing connects through SQLx. A connection_limit
that is not a positive integer is never split and goes to Prisma as before.
Once the share is above 0 Prisma keeps at least one connection, and pools are
tagged litellm-prisma and litellm-native in pg_stat_activity. Python keeps
minting RDS IAM and Entra tokens and hands each one to the pool

statement_cache_size sets the native statement cache (Prisma's default 100) and
pgbouncer=true turns it off. sqlx-postgres 0.9.0 still names every statement when
the cache is off, so Statements runs queries unnamed in that case, like Prisma,
until a release carries transact-rs/sqlx#4337. An application_name the operator
set is used on both pools

The SQL query files are pinned to LF so Windows checkouts hash them like .sqlx,
and the crate lists .sqlx in its package include so the core sdist builds offline

crates/db/AGENTS.md tracks parity with the Python Prisma stack

The bridge now links litellm-db, so the editable install compiles its queries.
The Rust DB workflow builds that install from the checked-in .sqlx metadata,
since it runs before the migrations that the online check needs
2026-11-14 15:28:10 -08:00
Nate Armstrong
8e92894ff2
feat(s3-log): replace the python s3_v2 logger with a rust callback (#51539)
The s3_v2 S3Logger is ported to a Rust callback, callbacks-s3-log, behind
the CustomLoggerV2 contract, with the bridge constructor NativeS3Log and the
Python glue in litellm/integrations/s3_v3.py. It is now the only
implementation: the Python logger is deleted, s3_v2.py re-exports the
Rust-backed S3Logger, and LITELLM_NATIVE_S3_V2 is removed. The callback keeps
its name, list position and settings, so no config changes.

The original s3_v2 tests are replicated against the Rust callback as
<original>_rust.py, and Python's outputs for the differential suite are
recorded in tests/unit/integrations/s3_v2_golden/python_s3_v2.json.
2026-11-14 23:03:58 +00:00
Abdellatif Anaflous
74fd46a59f
fix(vertex_ai): resolve correct default max_tokens for versioned claude ids (#40376)
* fix(vertex_ai): resolve correct default max_tokens for versioned claude ids

two defects stacked on the vertex claude path so a request without
max_tokens was silently capped at 4096 output tokens

the fill in transform_request called AnthropicConfig.get_config directly
so get_max_tokens_for_model on VertexAIAnthropicConfig never ran, and the
bare versioned id like claude-haiku-4-5@20251001 is not a cost map key,
so the fill fell back to the 4096 default

now transform_request calls self.get_config and VertexAIAnthropicConfig
resolves the vertex_ai prefixed key first, then falls back to the base
lookup

also the two vertex_ai claude haiku 4 5 map entries said 8192 max output
while every sibling claude-haiku-4-5 key says 64000 and google documents
64000, the reporter verified 64000 live on vertex with a 200 and 15290
completion tokens, both entries and the runtime backup map now say 64000

Fixes #40363

* chore(anthropic): drop redundant mutable-ok suppressions

the type discipline gate flags these two comments as LIT013, both suppress nothing because neither line has an LIT001 violation. no behavior change

* fix(vertex_ai): look up the vertex max tokens without the partially unknown cost map

the membership check on litellm.model_cost is a reportUnknownMemberType the basedpyright delta gate counts, and it adds exactly one over the merge-base. get_max_tokens already raises for a model that is not mapped, so catch that and fall back to the parent, same as the parent does for its own lookup. the focused vertex anthropic tests still pass

* fix(vertex_ai): guard the model cost map lookup with a pyright ignore

the try/except form added a blind except that the strict ruff gate counts as BLE001, one over the merge-base. keep the membership check and suppress only the reportUnknownMemberType on the dynamically typed registry, the same pattern the complexity router already uses. ruff strict is identical to the base, ruff format and ruff check clean, focused vertex anthropic tests 42 passed

* fix(vertex_ai): read the unversioned Claude row before the Vertex row for a missing max_tokens

A Vertex @version id now reads the cost-map row of its id without the version before the vertex_ai row, so claude-sonnet-4-6@default keeps the 128000 default that Vertex accepts while versioned Haiku, Opus, and Sonnet 4.5 ids still get 64000.

* fix(anthropic): keep an explicit DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS ahead of the unversioned and provider rows

An operator who set DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS, for example to 4096 as the v1.109.0 notes
suggest, got that value for every Claude id with no own cost-map row. The new unversioned and
provider-row steps ranked above it, so vertex_ai/claude-haiku-4-5@20251001 went from 4096 to 64000
and claude-sonnet-4-6@default from 4096 to 128000 for them. An explicit value now comes right after
the model's own row, so only deployments on the built-in fallback change

* fix(anthropic): keep DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS as the last fallback after the unversioned and provider rows

Reverts the previous commit. With DEFAULT_ANTHROPIC_CHAT_MAX_TOKENS set to any value, versioned
Vertex Claude ids ignored their model's row again: at 4096 a Haiku 4.5 reply still stopped at 4096
tokens, the symptom the linked issue reports, and at 128000 Vertex still answered 400. The env var is
documented as the default for models missing from the cost map, and a versioned id whose model has a
row is not missing, so it now gets the same cap as its unversioned id

* fix(anthropic): read the unversioned max_tokens row directly and test the lookup on other routes

The unversioned lookup went through get_max_tokens, which prints the provider banner a second time
when the id has no row, so SDK calls on ids like claude-3-5-haiku@20241022 printed it twice. Reading
the row directly gives the same max_tokens for every cost-map Claude id on every route. New tests
drive the provider and unversioned rows on the Anthropic, Azure AI and Claude Platform routes

* ci: restart the required checks after their held runs failed at startup

---------

Co-authored-by: hktitof <hktitof@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 17:36:51 -05:00
maciej-berri
8366e15be0
fix(proxy): close streams before rebuilding the served output for logging (#51487)
* fix(proxy): close streams before rebuilding the served output for logging

The streaming iterator hook awaited _record_served_stream_output before it
returned, so every streamed response stayed open after its last event while
the proxy re-parsed all served chunks. That held the connection about 11 ms
plus 25 ms per 1,000 output tokens, and SDK clients wait for the close.

The parked deferred logging is now claimed at stream end and run in a rooted
background task after the served output is recorded, so logging still sees
the served text and the client sees the stream end right away.

* fix(proxy): cap background served-output rebuilds and fall back to waiting past it

Each background rebuild holds its served chunks until it runs, and the stream's
concurrency slot is already released. Past 64 pending rebuilds a stream waits
for its own rebuild as it did before, so overload keeps its backpressure and
no spend logging is dropped.
2026-11-14 22:32:48 +00:00
quietjules
bc3113fff2
fix(vertex-ai): send union-root tool schemas as parametersJsonSchema (#51835)
* fix(vertex-ai): send union-root tool schemas as parametersJsonSchema

A function tool whose parameters root is a union (anyOf, oneOf or allOf at
the root, a type list, or a root $ref to one of those) cannot be expressed in
the Vertex and Gemini parameters field, so those tools answered 400. They now
go out as parametersJsonSchema with the caller's JSON Schema unchanged, on both
the tools and the legacy functions paths. Every other tool keeps the converted
parameters it had before. The local token counter also reads
parametersJsonSchema, so these tools still count

* fix(vertex-ai): resolve chained root refs and tuple schemas for parametersJsonSchema

A root $ref that points at another $ref now resolves to its end, so a chain
that ends in a union also goes out as parametersJsonSchema. A schema built
with tuples or sets is turned into its JSON form first, so it no longer falls
back to the parameters field. A type of ["object"] stays on the parameters
field, which already accepts it. Only tools whose root could need
parametersJsonSchema pay for the JSON conversion

* fix(vertex-ai): resolve long root $ref chains without recursion

* fix(vertex-ai): keep parameters for Vertex batch rows and Vertex countTokens

Vertex batch merges every row's keys into one shape, so a parametersJsonSchema tool in one row breaks every parameters row in the file. Vertex countTokens accepts parametersJsonSchema but leaves it out of the count. Both keep sending parameters, and the Gemini API count route keeps parametersJsonSchema, which it counts in full.

* fix(vertex-ai): resolve a root $ref chain in linear time

Walk $defs by name once with cycle detection, so a long missing ref or a
long-named ref cycle beside many defs no longer costs quadratic CPU

* fix(vertex-ai): send a root $ref tool as parametersJsonSchema when a union sits beside the ref

Check the root and every ref on its chain for anyOf, oneOf, allOf, or a non-object type, since a rule next to a $ref was dropped when only the chain's end was checked

* fix(vertex-ai): keep a bare root union tool on parameters when the call can be forced to it

* fix(vertex-ai): keep a bare root union tool on parameters when extra_body forces the call

* fix(vertex-ai): keep integer enums on a tool that extra_body forces back to parameters

* fix(vertex-ai): keep a union beside a looping root $ref on parameters and read the forcing mode in any case

A root union next to a $ref chain that loops back on itself went out as parametersJsonSchema, and Google
answers 400 "circular ref loop" for that, where base sent parameters and got 200. A looping chain now
stays on parameters whatever sits next to the $ref

An extra_body toolConfig with mode "any" forces the call on the Gemini API, but only "ANY" and "VALIDATED"
were read as forcing, so a bare root union tool moved to parametersJsonSchema and the forced call answered
400. The mode is now compared in upper case

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 17:29:37 -05:00
moe-berri
d8e8621b7b
perf(logging): scan opt-in spend-log tokens natively (#51770) 2026-11-15 06:09:02 +08:00