Commit graph

56658 commits

Author SHA1 Message Date
mateo-berri
d91bc3fb8e test(messages): use rstest for the new bedrock invoke thinking tests
Some checks are pending
Cost map pins recheck / rerun-cost-map-guard (push) Waiting to run
LiteLLM Rust DB / db-queries (push) Waiting to run
LiteLLM Rust / rust-test (push) Waiting to run
LiteLLM Rust / rust-lint (push) Waiting to run
LiteLLM Rust / rust-wheel (push) Waiting to run
Terraform Modules / fmt, validate, test (aws) (push) Waiting to run
Terraform Modules / fmt, validate, test (gcp) (push) Waiting to run
2026-11-15 21:15:28 -08:00
mateo-berri
46a843b030 fix(messages): drop unsigned thinking on bedrock invoke's rust messages route 2026-11-15 20:01:50 -08:00
mateo-berri
e130ab8ca8 Merge remote-tracking branch 'origin/main' into litellm_messages_drop_unsigned_thinking
# Conflicts:
#	litellm/rust/crates/inference-messages/src/handler.rs
2026-11-15 19:53:13 -08:00
Nate Armstrong
8d200a4f98
feat(messages): serve bedrock invoke messages on rust only (#51330) 2026-11-15 16:58:31 -08:00
Kerry Lu
d0bf848d91
fix(dynamic_rate_limiter): charge tokens used while the model was idle against a team's class share (#51952) 2026-11-15 16:49:47 -08:00
Nate Armstrong
2aa57d7e04
feat(budgets): replace the python budget hold and settle engine with a rust engine admitted at auth (#51535)
* feat(budgets): replace the python budget hold and settle engine with a rust engine admitted at auth

* fix(budgets): keep native_call_id out of the budget reservation like main
2026-11-15 16:03:51 -08:00
moe-berri
175993910a
fix(proxy): batch customer organization cascade deletion (#51969) 2026-11-16 08:03:45 +08:00
Kerry Lu
ceb458c929
fix(proxy): reset members given default team member models on the team's budget window (#51960)
* fix(proxy): keep the team member budget cap and reset for members given default team member models

* test(proxy): mark the new test's locals Final

* fix(proxy): give models-only members the team's reset window and leave the cap to the team

* fix(proxy): move members on the team's reset window when the team's window changes

* fix(proxy): move members before the team window write, skip an unchanged window, cover the first member budget

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 16:00:44 -08:00
Nate Armstrong
eb8a2d224e
perf(auth): reuse validated models on UserApiKeyCache memory hits (#51912)
A typed read of the same in-memory payload returns a shallow copy of the model the previous read validated instead of running model_validate again. Writes, deletes and flushes drop the memo, and the Redis and shared store payloads are unchanged.
2026-11-15 15:39:44 -08:00
Mateo Wang
8c97c243fe
fix(router): free a failed rust router call's request without waiting for a full gc (#51963)
* fix(router): free a failed rust router call's request without waiting for a full gc

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(router): release retained retry errors after successful calls

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 17:04:38 -05:00
Mateo Wang
1e3a960a0e
refactor(vertex_ai): remove unreachable VertexAIFilesHandler.afile_content (#51966)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 13:54:05 -08:00
Mateo Wang
1824550225
chore(lint): flag conditionals with more than 3 boolean clauses via ruff PLR0916 (#51964)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 16:53:38 -05:00
moe-berri
bddb97132b
feat(customers): sort organizations, teams and members by budget limit (#51924) 2026-11-16 05:22:10 +08:00
moe-berri
923162a4e7
fix(db): index customer hierarchy filters (#51961) 2026-11-16 04:28:34 +08:00
Kerry Lu
0dee2a8a0d
fix(responses): retry a stream that breaks before output in its own model group (#51953) 2026-11-15 12:19:58 -08:00
Rhythm Arya
e6b673e6e0
fix(proxy): hold the logging object weakly in the detached stream failure hook (#51318)
* fix(proxy): hold the logging object weakly in the detached stream failure hook

The closure armed by _arm_detached_stream_failure_hook is stored on the
logging object and captured the logging object, the request data that
contains it, and the processing object that holds that data. Every
streamed /v1/messages request therefore ended in a reference cycle and
stayed in memory until a full cyclic GC pass instead of being freed by
reference counting when the stream finished.

Capture the logging object through a weakref, a copy of the request data
without the litellm_logging_obj key, and the route type as a value. The
hook adds the live logging object back to the request data when it fires.
All three callers reach the hook through the logging object, so it is
always alive at that point.

Add a regression test that streams a request to completion, drops every
reference, and checks with the cyclic collector disabled that the logging
object is gone.

Fixes #51277

* test(proxy): cover finished streamed /v1/messages calls being freed without a gc pass

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): annotate the finished-stream gc test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): split the freed-calls assert message under the line limit

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 15:13:24 -05:00
Kerry Lu
b168ca7ae4
fix(admin-mcp): restore the budget route operation ids the connector pins (#51959)
* fix(admin-mcp): restore the budget route operation ids the connector pins

* test(admin-mcp): skip the operation id check below Python 3.12 like its siblings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 20:11:02 +00:00
Kerry Lu
765af9ab71
fix(count_tokens): return 400 for malformed Anthropic server-tool blocks (#51957)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 20:10:12 +00:00
Kerry Lu
60716572a4
feat(dynamic_rate_limiter): customer organizations and keys borrow idle model capacity and split class shares (#51705)
* feat(dynamic_rate_limiter): let customer orgs and keys borrow idle model capacity and split class shares

Replace the per-team borrowing and class share booleans with model_limit_borrowing_config.borrow_idle_capacity
and priority_class_share_config.split_equally_among_active, which list the entity types they apply to.
Use one borrow cap field, model_limit_max_borrow_multiple, on team metadata, key metadata and the customer
organization budget.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): size holds with the tightest share, fail closed on websocket turns, guard lookalike team ids and the borrow cap

One token estimate now covers every share a request holds, using the tightest TPM share's output cap.
A team id spelled like a key or customer organization holder falls back to the class pool.
model_limit_max_borrow_multiple in key or team metadata is changed by proxy admins only.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): keep a team's key on the team's class share slice and reset websocket share state per turn

A team listed for the class share split now holds the slice ahead of customer headers, which the caller chooses.
A websocket turn clears the shares of borrowers that do not apply on its model, so it is not settled on an earlier model's share.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(dynamic_rate_limiter): read the borrow cap from model_limit_borrowing_config.max_borrow_multiple instead of a budget table column

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(dynamic_rate_limiter): a fallback target a borrower has no limit on no longer charges its share on the failed model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 12:05:58 -08:00
Kerry Lu
8abcd13a9d
fix(cost): keep a deployment's per-second video price over the catalog row's flat price (#51688)
* fix(cost): keep a deployment's per-second video price over the catalog row's flat price

* fix(cost): drop the superseded flat video price from the registration kept for reloads

* test(cost): derive the per-second rate from the catalog row so the regression holds at any vendor price

* fix(cost): keep the flat video price when only a resolution tier is priced per second

* test(integration): assert the body sent to fal in the per-second price test

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nate Armstrong <narmstrong@berri.ai>
2026-11-15 12:04:41 -08:00
moe-berri
93c3074ecb
fix(ui): default customer usage range to month to date (#51923) 2026-11-16 04:03:02 +08:00
moe-berri
28c892497c
fix(budgets): match window starts to stored millisecond precision (#51921) 2026-11-16 04:02:44 +08:00
moe-berri
92115c26aa
fix(customers): keep uncapped customer spend across scheduled budget resets (#51918) 2026-11-16 04:02:27 +08:00
Mateo Wang
d49706f55d
fix(router): stop the rust router keeping every finished request in memory until a full gc (#51888)
* fix(router): stop the rust router keeping every finished request in memory until a full gc

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): prove finished /v1/messages and chat calls free their log record without a gc pass

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): read the memory endpoint with typed helpers instead of a type ignore

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 11:31:07 -08:00
Kerry Lu
61a1648572
fix(fal_ai): bill a video request without a resolution at the row's default rate (#51665)
* fix(fal_ai): bill a video request without a resolution at the row's default rate

* test(fal_ai): drop the assertion that two fal rates differ

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:51 +08:00
Kerry Lu
45c05c5bea
fix(bedrock): put extra_body fields inside modelInput for async-invoke embeddings (#51676)
* fix(bedrock): put extra_body fields inside modelInput for async-invoke embeddings

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(bedrock): keep clientRequestToken and tags at the top level of async-invoke requests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(bedrock): cover clientRequestToken staying top-level in the async-invoke wire test

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:25 +08:00
Kerry Lu
2d487abfc1
fix(vertex_ai): send extra_body fields as predict parameters for BGE embedding models (#51662)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:21 +08:00
Kerry Lu
19d8e57485
fix(perplexity): bill cached prompt tokens at the cache read rate when the response has no cost (#51669)
* fix(perplexity): bill cached prompt tokens at the cache read rate when the response has no cost

* fix(perplexity): bill cached tokens at the off-peak cache read rate when the model has no standard one

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:30:13 +08:00
Kerry Lu
ab7af57878
fix(replicate): answer malformed video seconds and size with a 400 or a dropped size instead of a 500 (#51670)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:29:08 +08:00
Kerry Lu
af6056eca1
fix(fal_ai): send versioned seedream and flux-pro image models to the endpoint they name (#51668)
* fix(fal_ai): send versioned seedream and flux-pro image models to the endpoint they name

* fix(fal_ai): reject dot segments in the model path and tighten the versioned endpoint tests

* test(fal_ai): match the dot segment error message

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:28:53 +08:00
Kerry Lu
2899c97601
fix(proxy): run the /fal_ai passthrough price gate on the path sent upstream (#51671)
* fix(proxy): run the /fal_ai passthrough price gate on the path sent upstream

The gate checked the raw request path for /requests/ and priced the raw path, so a path that
collapses to a submit URL upstream (a /requests/.. dot segment, or an encoded ? or # tail)
skipped the gate and reached fal on the proxy's key

* test(proxy): use an endpoint with no catalog row in the fal passthrough gate test

* fix(proxy): price the fal passthrough gate on the encoded upstream path

* fix(proxy): strip the fal queue base prefix in the form it is sent upstream

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:28:31 +08:00
Kerry Lu
063b63de6c
fix(proxy): let an exhausted key keep listing models and counting tokens (#51675)
* fix(proxy): keep an exhausted key on model listing and token counting routes

The header-only preflight added in #51428 ran the rate limit check on every LLM API route, so a key that had used up its rpm limit got 429 on GET /v1/models, GET /models and POST /utils/token_counter, routes whose handlers never run the limiter. The preflight now skips the limit check on GET requests, which carry no body to avoid reading, and on unbilled routes.

* fix(proxy): still refuse a GET that declares a body before reading it

A GET can carry a body, so the limit check is only skipped for a GET that declares none. A metered route such as /v1/batches keeps its 429 before a declared body is buffered.

* test(proxy): fail the preflight test on any outcome other than admitted or 429

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:28:26 +08:00
Kerry Lu
086fc1da3f
fix(proxy): join default teams on the team member budget when the team has one (#51728)
* fix(proxy): join default teams on the team member budget instead of a private one

* fix(proxy): keep an explicit per-user team budget and drop only the default team one

* fix(proxy): drop a default team's per-user cap only when the team has a member budget

* fix(proxy): chunk the default team lookup

* fix(proxy): type the team metadata read

* fix(proxy): validate the team metadata instead of casting it

---------

Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-16 03:27:57 +08:00
Nate Armstrong
d63224db15
refactor(s3-log): move python-compatible helpers into python-compat (#51895) 2026-11-15 11:19:03 -08:00
Nate Armstrong
09ce4f50ed
perf(auth): read token model fields from the class instead of deprecated __fields__ (#51901)
Some checks failed
Cost map pins recheck / rerun-cost-map-guard (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
unit / root (push) Blocked by required conditions
unit / proxy-utils (push) Blocked by required conditions
unit / enterprise-managed-files (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / rust-bridge (push) Waiting to run
unit / assert-shard-coverage (push) Waiting to run
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.0.0 compatibility (push) Blocked by required conditions
unit / OpenAI 3.25.0 compatibility (push) Blocked by required conditions
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* perf(auth): read token model fields from the class instead of deprecated __fields__

* test(s3): assert 403 retry backoff on the logged wait instead of wall-clock gaps
2026-11-15 07:07:49 +00:00
Nate Armstrong
9e3a82ff03
perf(caching): format dual cache debug logs lazily (#51902)
* perf(caching): format dual cache debug logs lazily

* fix(caching): type dual cache kwargs so lazy log args stay inside the pyright ceiling
2026-11-15 06:50:04 +00:00
Nate Armstrong
98c86f6e7c
perf(streaming): extract usage once per chunk when assembling stream usage (#51903) 2026-11-14 22:31:39 -08:00
Nate Armstrong
d76a7d76c4
perf(host-python): settle ready async values inline instead of spawning on tokio (#51906)
run_async_value polls the future once on the caller's thread with the GIL released and the runtime entered. A future that completes returns an already settled asyncio future on the caller's loop, and only a pending one is handed to future_into_py.
2026-11-15 06:27:55 +00:00
Nate Armstrong
3bcd77954e
refactor(s3-log): move the rust-backed s3 logger from s3_v3 to s3_v2 (#51893) 2026-11-14 22:24:23 -08:00
devin-ai-integration[bot]
b90ac69841
test(integration): record a stack when an owned proxy crashes (#51922)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 06:15:35 +00:00
yuneng-jiang
91fc6c2071
test(e2e): drop two batch waits that never finish in time (#51882)
cleanup_batch waited up to 120s for a managed batch that was still
cancelling to reach a terminal state. Across 55 saved litellm-e2e builds it
reached one in time once, so test_batch_lifecycle[openai-unified] spent 126s
and TestBatchSecondHop 140s on every run, and both then left the input file
in place anyway. A managed batch still cancelling is now left alone, the same
as a raw provider batch, and the input file cleanup still reports the
leftover.

TestBatchTerminalState listed the marker batches every 10s for up to 300s.
Each e2e stack starts with an empty database, so the list only ever holds
this run's own marker, which never completed inside the window in any of
those builds, and the test took 303s for its documented vacuous pass. It now
lists once.
2026-11-14 21:41:43 -08:00
kenneth-berri
26807a9925
feat(customers): search customer members by ID or alias in the Members tab (#51896) 2026-11-15 04:47:25 +00:00
moe-berri
00313c9e5b
feat(customers): move existing members between teams (#51913) 2026-11-15 12:22:02 +08:00
tin-berri
c3b929d203
feat(ui): reuse usage spend design for customer teams and members (#51724) 2026-11-15 11:45:29 +08:00
moe-berri
bc78cbe9a2
fix(ui): clarify customer usage periods and preserve date ranges (#51899)
* fix(ui): clarify customer usage periods and preserve date ranges

* fix(ui): preserve fixed customer dates with stale preset labels
2026-11-14 19:09:35 -08:00
devin-ai-integration[bot]
aeb9ed07aa
test(database): expect no native pool in proxy lifespan cleanup (#51911)
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-11-15 03:09:22 +00:00
ishaan-j-berri
61483d8f04
fix(ui): rename Customer Portal to Customer Management (#51891)
* fix(ui): rename Customer Portal to Customer Management

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* fix(ui): polish Customer Management product copy

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

* style(ui): format Customer Management test assertions

Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com>
2026-11-14 21:23:35 -05:00
yujonglee
745886e5ff
fix(packaging): keep the litellm-proxy package version in step with litellm (#51807)
The litellm-proxy manifest still said 1.109.0 after litellm moved to 1.110.0 and 1.111.0, because version bumps only rewrote the root pyproject.toml. The Docker image builds that manifest unchanged, so ghcr.io/berriai/litellm:v1.111.0-dev.2 reports litellm-proxy 1.109.0, a version that was never published.

Set the manifest and uv.lock to 1.111.0, add the manifest to the commitizen version_files so cz bump rewrites it, and add a test that fails when the two versions or the lock entry disagree.
2026-11-14 18:16:47 -08:00
joshua-berri
e38187324a
fix(spend): support PostgreSQL 12 delivery migrations (#51868)
* fix(spend): support PostgreSQL 12 delivery migrations

The SpendLogDelivery migration defaulted delivery_id to gen_random_uuid(), which
PostgreSQL 12 lacks without pgcrypto, so migrate deploy failed there. The writer
always supplies delivery_id, so the column no longer needs a database default.
Databases that already applied the original migration drop the default in a
follow-up migration without rewriting rows.

* test(spend): apply the ledger enum migration before the delivery migration tests

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-14 17:47:36 -08:00
yuneng-jiang
4a8fa7c60a
test(e2e): run the batch lifecycle with a key that only reaches its own deployment (#51857)
test_batch_lifecycle[azure-provider_fallback] has failed in most scheduled
e2e runs since the suite moved from 8 to 12 xdist workers. Batch create
returned 404 Resource not found from Azure after a successful upload.

A batch call that names a provider and no model takes its credentials from
the first deployment of that provider the key may use. The test used an
all-models key, so on the shared proxy that was whichever Azure deployment
got registered first. With 12 workers the realtime suite starts in the same
instant as the batches suite, and its session-long azure-realtime deployment
carries api_version 2025-08-28. When it registers before azure-batch, the
batch create goes out with that api_version and Azure answers 404. The
upload still works because the file-create route overrides the borrowed
credentials with files_settings. The file delete on cleanup does not, so the
model_param and provider_fallback cleanups were also sent with the wrong
api_version and their 404 was read as already deleted.

The lifecycle key now belongs to a team that may only use the batch
deployment under test, so provider-only calls resolve that deployment in any
registration order.
2026-11-14 16:55:01 -08:00