* fix(proxy): hold the logging object weakly in the detached stream failure hook
The closure armed by _arm_detached_stream_failure_hook is stored on the
logging object and captured the logging object, the request data that
contains it, and the processing object that holds that data. Every
streamed /v1/messages request therefore ended in a reference cycle and
stayed in memory until a full cyclic GC pass instead of being freed by
reference counting when the stream finished.
Capture the logging object through a weakref, a copy of the request data
without the litellm_logging_obj key, and the route type as a value. The
hook adds the live logging object back to the request data when it fires.
All three callers reach the hook through the logging object, so it is
always alive at that point.
Add a regression test that streams a request to completion, drops every
reference, and checks with the cyclic collector disabled that the logging
object is gone.
Fixes#51277
* test(proxy): cover finished streamed /v1/messages calls being freed without a gc pass
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): annotate the finished-stream gc test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): split the freed-calls assert message under the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dynamic_rate_limiter): let customer orgs and keys borrow idle model capacity and split class shares
Replace the per-team borrowing and class share booleans with model_limit_borrowing_config.borrow_idle_capacity
and priority_class_share_config.split_equally_among_active, which list the entity types they apply to.
Use one borrow cap field, model_limit_max_borrow_multiple, on team metadata, key metadata and the customer
organization budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): size holds with the tightest share, fail closed on websocket turns, guard lookalike team ids and the borrow cap
One token estimate now covers every share a request holds, using the tightest TPM share's output cap.
A team id spelled like a key or customer organization holder falls back to the class pool.
model_limit_max_borrow_multiple in key or team metadata is changed by proxy admins only.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): keep a team's key on the team's class share slice and reset websocket share state per turn
A team listed for the class share split now holds the slice ahead of customer headers, which the caller chooses.
A websocket turn clears the shares of borrowers that do not apply on its model, so it is not settled on an earlier model's share.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dynamic_rate_limiter): read the borrow cap from model_limit_borrowing_config.max_borrow_multiple instead of a budget table column
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): a fallback target a borrower has no limit on no longer charges its share on the failed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): keep a deployment's per-second video price over the catalog row's flat price
* fix(cost): drop the superseded flat video price from the registration kept for reloads
* test(cost): derive the per-second rate from the catalog row so the regression holds at any vendor price
* fix(cost): keep the flat video price when only a resolution tier is priced per second
* test(integration): assert the body sent to fal in the per-second price test
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nate Armstrong <narmstrong@berri.ai>
* fix(router): stop the rust router keeping every finished request in memory until a full gc
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove finished /v1/messages and chat calls free their log record without a gc pass
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read the memory endpoint with typed helpers instead of a type ignore
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): bill a video request without a resolution at the row's default rate
* test(fal_ai): drop the assertion that two fal rates differ
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): put extra_body fields inside modelInput for async-invoke embeddings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep clientRequestToken and tags at the top level of async-invoke requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): cover clientRequestToken staying top-level in the async-invoke wire test
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(perplexity): bill cached prompt tokens at the cache read rate when the response has no cost
* fix(perplexity): bill cached tokens at the off-peak cache read rate when the model has no standard one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): send versioned seedream and flux-pro image models to the endpoint they name
* fix(fal_ai): reject dot segments in the model path and tighten the versioned endpoint tests
* test(fal_ai): match the dot segment error message
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run the /fal_ai passthrough price gate on the path sent upstream
The gate checked the raw request path for /requests/ and priced the raw path, so a path that
collapses to a submit URL upstream (a /requests/.. dot segment, or an encoded ? or # tail)
skipped the gate and reached fal on the proxy's key
* test(proxy): use an endpoint with no catalog row in the fal passthrough gate test
* fix(proxy): price the fal passthrough gate on the encoded upstream path
* fix(proxy): strip the fal queue base prefix in the form it is sent upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep an exhausted key on model listing and token counting routes
The header-only preflight added in #51428 ran the rate limit check on every LLM API route, so a key that had used up its rpm limit got 429 on GET /v1/models, GET /models and POST /utils/token_counter, routes whose handlers never run the limiter. The preflight now skips the limit check on GET requests, which carry no body to avoid reading, and on unbilled routes.
* fix(proxy): still refuse a GET that declares a body before reading it
A GET can carry a body, so the limit check is only skipped for a GET that declares none. A metered route such as /v1/batches keeps its 429 before a declared body is buffered.
* test(proxy): fail the preflight test on any outcome other than admitted or 429
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): support PostgreSQL 12 delivery migrations
The SpendLogDelivery migration defaulted delivery_id to gen_random_uuid(), which
PostgreSQL 12 lacks without pgcrypto, so migrate deploy failed there. The writer
always supplies delivery_id, so the column no longer needs a database default.
Databases that already applied the original migration drop the default in a
follow-up migration without rewriting rows.
* test(spend): apply the ledger enum migration before the delivery migration tests
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* test(unit): reset module state only when a test changed it
The proxy conftest deep-copied every snapshotted litellm attribute before each
test, and the copy of litellm.model_cost alone was a median 0.12 s for each of
56,031 proxy tests on the runner. The reset now copies an attribute back only
when it no longer equals the snapshot.
The enterprise and MCP server conftests reloaded litellm before every test,
3.27 s and 0.21 s per test. They now use the same reset.
* test(unit): remove six flaky tests
They failed on main without a code change behind them between 2026-11-05 and
2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM
count, three Prometheus metrics server port tests and the OpenCode list cost
billing test.
* ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests
proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2
times faster with 8 workers on a 16-core runner in a benchmark run.
windows-sdk and proxy-build-from-pip still run on pushes to main and manual
runs. proxy-build-from-pip installs a published release and never runs the
pull request's code.
* test: move 109 tests that start a process or open a socket into tests/integration
tests/unit is in-process by its own rules, and these tests spawn a Python
interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit
shard time per run, 396 s of it in the MCP client transport tests. Each one now
lives in a *_real_io.py file under the integration directory for its domain and
runs with that directory's group.
The tests themselves are unchanged apart from one repository-root path in the
CLI test.
* test(unit): copy back only the dict entries a test changed
A test module that registers models at import leaves litellm.model_cost four
entries larger than the snapshot before every test in its directory, so the
reset still deep-copied the whole price map for each of them. It now rebuilds a
changed dict from the live entries that still match and copies only the rest.
* ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate
A skipped job passing the gate is what test_unit_passed_gate guards against,
so the gate goes back to requiring success from every job it needs and
windows-sdk and proxy-build-from-pip stop being part of it. They still run
and report on pushes to main and manual runs.
The s3_v2 S3Logger is ported to a Rust callback, callbacks-s3-log, behind
the CustomLoggerV2 contract, with the bridge constructor NativeS3Log and the
Python glue in litellm/integrations/s3_v3.py. It is now the only
implementation: the Python logger is deleted, s3_v2.py re-exports the
Rust-backed S3Logger, and LITELLM_NATIVE_S3_V2 is removed. The callback keeps
its name, list position and settings, so no config changes.
The original s3_v2 tests are replicated against the Rust callback as
<original>_rust.py, and Python's outputs for the differential suite are
recorded in tests/unit/integrations/s3_v2_golden/python_s3_v2.json.
A Straiker verdict that blocks the prompt now comes back as a reply (HTTP 200,
finish_reason content_filter) on chat completions, text completions, messages and
responses, the way an answer-side block already does, instead of an HTTP 400 that
some clients show as a malformed request. Routes that cannot send a reply, such as
embeddings and /guardrails/apply_guardrail, keep the HTTP 400. A blocked answer on
those routes is now an HTTP 400 too, where it was a 500. A backend failure still
fails closed with an error.
HTTP 408, 429 and 500 from Straiker now follow unreachable_fallback once retries
are spent, and an attempt that times out is no longer retried.
The new stream_mode option picks how a streamed response reaches the client.
"buffered", the default, holds it until Straiker has checked the whole response.
"streaming" sends chunks as they arrive and checks the assembled response at the end.
Each v3 phase now writes its own guardrail log entry with the session, action,
turn id and controls, in place of the generic entry that read "allow" even when
the guardrail failed open.
A claude-cli/ User-Agent is named Claude only for Claude Code entrypoints, so
other clients that share the prefix are no longer misnamed.
* fix(otel): send the Langfuse environment, release and version on OTel v2
* fix(otel): keep the Langfuse environment on the Langfuse exporter only
* fix(otel): annotate the exporter resource attributes as a read-only mapping
* fix(otel): shed the Langfuse release and version first at the attribute limit
* fix(otel): keep base attributes when the Langfuse release and version do not fit
- the model span sheds release and version before the identity fields
- the request root takes release and version only when they fit under its attribute limit
- an exporter routed to a tenant's credentials drops the proxy's Langfuse environment
- a langfuse.environment the proxy's resource already names wins over LANGFUSE_TRACING_ENVIRONMENT
* test(otel): annotate the new attribute-limit and routing tests
* feat(proxy): add per-model project batch enqueued token limits
Enqueued batch token limits only read key and team metadata, so a deployment
that gives each application its own project under a shared team cannot cap one
application's queued batch tokens for a given model. The team limit is shared
by every project in it, and JWT or custom-auth callers get a new key scope with
every token.
Read model_batch_enqueued_token_limit from project metadata, keyed by model like
model_tpm_limit. Each limited model group the batch runs is reserved against
only its own rows' tokens, rows that name no model count toward the routing
model, and the reservation records each scope's charge so the terminal-state
refund releases exactly that. Only proxy admins may set it, on project create
and update as well as on keys and teams, so a team admin cannot lift their own
project's cap.
* fix(proxy): charge the routing model and keep daily caps for project batch limits
Rows can name any model while the batch runs on the routing deployment, for
example LiteLLM-executed batches replace every row's model with the run model,
so charging only row models let a batch skip a capped deployment's per-model
limit. Charge the routing model the whole batch as well as each other limited
row model its own rows' tokens.
An enqueued-token reservation replaced every batch counter, so a project limit
also skipped the key, team and end-user daily token caps. Keep checking those
daily caps after reserving, and refund the reservation if one rejects the batch.
* fix(proxy): keep per-minute checks for batches a project allowance does not cover
A per-model project allowance replaced every per-minute batch check as soon as
any row's model had one. A batch routed to an unlimited model could add a
single tiny row naming a limited model and skip the routing model's RPM/TPM and
project ITPM/OTPM quotas, although every row runs on the routing model. Waive
those checks only when the reservation covers the whole batch, through a key or
team allowance or a project allowance on the routing model; otherwise reserve
the covered models and run the normal checks too.
* test(proxy): cover the project per-model batch cap through custom auth on a live proxy
* fix(proxy): keep key and team batch allowances authoritative and harden project limits
Key and team enqueued allowances again replace the other batch checks. A project
allowance on the routing model keeps only the daily caps. Team admins can only be
blocked from the project key, invalid project maps answer 400, the edit modal keeps
stored limits, aliases share their tightest limit, file-bound batches charge only
the routing model, deployment ids resolve to their model group, and reservation
records keep project scopes apart so older pods still refund key and team scopes.
* fix(proxy): charge row models for model-bound batch uploads
Only a managed-file upload rewrites rows to the deployment model. A model-bound
upload keeps each row's model, so its rows still count toward the project caps of
the models they name.
* fix(proxy): skip per-minute batch limits only when the project caps every model in the batch
A project cap on the routing model alone used to switch off the project's
per-minute input and output limits for the other models a batch file names.
The skip now needs a project cap on every one of those models.
-latest names are charged under the cap of the model they point to, the
limit map only takes JSON integers, and a /project/update that sends only
top-level fields keeps the stored metadata instead of replacing it.
* fix(proxy): lift per-minute limits off managed batches under a project cap and edit the cap in the Admin UI
* fix(ui): keep the project batch cap read-only for team admins and require a whole number above 0
* fix(ui): narrow project metadata directly when keeping the stored batch cap
* fix(proxy): charge dated snapshot rows and header or query routed batches to project caps
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore: move LiteLLM's configs, cookbooks and e2e tests off gpt-4o and gpt-4.1 before their 2027-01-14 shutdown
OpenAI shuts down gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini on 2027-01-14. The example proxy
configs, cookbook guides and e2e suites now call gpt-5.6, gpt-5.6-luna or gpt-5.4-mini, and the
predicted-outputs e2e test is removed because no current OpenAI model accepts prediction
* docs(cookbook): name gpt-5.6 instead of gpt-4o as a tool-capable model in the gollem troubleshooting tips
* chore(ci): delete the observatory proxy config, which nothing has loaded since its workflow was removed
* chore: send max_completion_tokens in batch lines and move README and deploy examples to gpt-5.6
gpt-5.6-luna rejects max_tokens on OpenAI's batch lines, which reach OpenAI unchanged, so the batch JSONL builders send max_completion_tokens. The README quick start, the adaptive router README and the Helm and Terraform example configs move off gpt-4o with the configs
* chore: pin output prices next to input prices in the adaptive router example
The adaptive router reads only input_cost_per_token from config, and a deployment priced by input alone bills output at $0, so the example and its README pin both at the cost map's current prices
* chore: move the configs and e2e tests to gpt-6-luna and gpt-6.1-sol
The cost map names gpt-6-luna as the successor of gpt-4o, gpt-4o-mini, gpt-4.1
and gpt-4.1-mini. Cheap roles move to gpt-6-luna and full-size and vision roles
to gpt-6.1-sol, the pair main's other configs use. The adaptive router example
pins the gpt-6 prices, and the installer quick start and two READMEs drop gpt-4o
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): skip groups with no team so a PATCH or PUT still deactivates the user
A group the IdP pushes before LiteLLM has a team for it made the roster
write fail, which answered 404 before the user row was written, so the
user stayed active and their keys kept serving. #37700 made that failure
fatal.
Groups with no team row are now logged and left out of the roster write,
the persisted teams, and the role resolution. Group ops after a replace
now apply in order, as RFC 7644 section 3.5.2 says.
* fix(scim): drop the redundant unchanged-groups return and cover the warning and PATCH admin role
* fix(scim): strip line breaks from the teamless-group warning
* fix(scim): read the PUT role from every requested group, as POST does
A PUT naming only a group with no team no longer demotes an admin whose admin group has no team. The new tests also get typed fixtures and Final locals
* fix(scim): look up the requested teams on the primary database, as the roster write does
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): run baggage tag mappings and budget checks on /queue/chat/completions
* fix(proxy): check the key's max_budget on /queue/chat/completions
* fix(proxy): run budget, tag, and request policy checks on queued chat calls
* fix(proxy): bill a dropped queued stream and run the trace ID and failure logging checks on queued calls
A queued stream now hands its logging object to the stream generator, so a
client that leaves mid-stream is billed for the reply it already got, and a
stream left before the first model chunk releases its budget reservation.
A team's require_trace_id and the failure logging for rejected calls now
use the same route check as the other inference checks.
* fix(proxy): settle a dropped queued stream's budget hold through the first-chunk step chat uses
* fix(proxy): keep the trace ID, call ID, unpriced fallback filter, and JWT admin team header on queued chat calls
* fix(proxy): send the call ID on a queued call the provider rejects before streaming
* fix(proxy): release a dropped stream's budget hold only when nothing bills it
A guardrail that buffers the stream can pull model chunks the client never
saw. The early-drop release settled the hold at the input cost first, then
the partial bill for those chunks added its full cost on top. The release
now runs after partial billing and only when no bill fired
* fix(proxy): fold the caller's litellm_metadata into a queued call's metadata
A queued call that sent a litellm_metadata bucket, even an empty one, made tag
routing and logging read that bucket, so the key and team tags the handler
merged into metadata no longer narrowed deployments. The queue handler now
folds the bucket into metadata and drops it, as chat does, after stripping the
proxy-owned user_api_key_* and internal control fields from both buckets, so a
caller can no longer set the queued spend row's organization ID either
* fix(proxy): release a queued call's budget hold when the call is cancelled before it answers
* fix(proxy): refund a cancelled queued call's hold in full until the model call claims it
* fix(proxy): keep a cancelled queued call's hold release running when the call is cancelled again
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Anthropic-compatible hosts such as DeepSeek accept Gemini's unsigned thought and need
it passed back on tool turns, so a Claude model name alone no longer triggers the strip
under anthropic/. Python and Rust now also require an anthropic.com or Azure AI Foundry
host there, while fixed-host providers keep the name check.
Bedrock Converse drops the unsigned thought instead of sending it as reply text, and
count_tokens drops it for Claude targets so it accepts the history /v1/messages accepts.
The Bedrock ARN override is removed, so Bedrock uses the shared name check.
* fix(azure): send no api-version on Foundry project /openai/v1 Responses bases
Azure answers 400 "api-version query parameter is not allowed when using /v1 path" for every api-version on a Foundry project's /api/projects/<p>/openai/v1/responses route, so drop it there and keep it everywhere else
* fix(azure): send no api-version on Foundry project v1 chat and WebSocket, bill routed models
Chat completions on a Foundry project /openai/v1 base now use the v1 OpenAI client, so they send no
api-version, which that path rejects. The Responses WebSocket URL builder strips a bare /openai/v1 base
before adding /openai/v1/responses, so it no longer dials a doubled path. An azure/model-router call that
routes to a model priced only under azure_ai/ is billed from that row plus the router fee once
* fix(azure): keep the Azure client off chat-only paths, honor api_version none, drop api-version from v1 base queries
Only chat completions switch to the v1 OpenAI client on a Foundry project /openai/v1 base, so transcription and
azure_text keep the Azure client and no longer fail inside LiteLLM. A deployment with api_version none sends no
api-version on Responses and chat, for gateways that hide the project path. The v1 client now moves an api_base
query into default_query without api-version instead of gluing the route after it
* fix(azure): pick the v1 client for o-series chat, keep api_version none off a doubled /openai/responses path
* fix(azure): send no api-version on embeddings under api_version none, test async and streaming chat on v1 bases
* fix(azure): join openai/v1 once onto a v1 client base that ends in /openai
Chat and embeddings under api_version none, and chat under api_version v1, built /openai/openai/v1 from an api_base ending in /openai, while Responses on the same base already built /openai/v1. The v1 client now joins the route the same overlap-aware way
* fix(azure): keep a percent-encoded gateway path as typed on the v1 client base
Join openai/v1 by string instead of add_path_to_api_base, which decodes %2F, %25, %3F and %23 in the base path. Type the chat test helper's mode as a Literal with assert_never.
* test(azure): import assert_never from typing_extensions so the Azure tests load on Python 3.10
typing.assert_never needs Python 3.11, and the package supports 3.10
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses bridge): carry cache_control as a prompt_cache_breakpoint to deployments that read them
A chat request with Anthropic-style cache_control on a system or user part got its mark renamed to
prompt_cache_breakpoint on the plain chat path, but the chat-to-Responses bridge (taken once a
function tool and reasoning are on, and always for bedrock_mantle GPT) dropped it, so explicit-mode
caching never wrote. The bridge now renames the marks before converting the messages when the
served deployment reads breakpoints, using the same gate the Responses config uses for native
/v1/responses input. Deployments that do not read breakpoints keep today's behavior.
The rename also leaves unmarked content parts untouched instead of validating them, so a part with
non-string keys no longer raises on either path.
* fix(responses): keep leading system messages in client order once one goes to input
A marked first system string became a list and moved to input while the
next unmarked string stayed in instructions, so the model read them in
reverse. The leading run that joins instructions now stops at the first
system message whose content is not a string
* fix(responses): leave Bedrock Mantle and unmarked system order as they were on the bridge
Bedrock Mantle already caches the growing prefix on its own, so turning
cache_control into breakpoints there added write charges. The client-order
rule for leading system messages now applies only when a mark turns a system
string into a list, so unmarked chats send the same request as before.
* test(responses): cover a marked user turn in the unmarked system split test
A cache_control mark on the user message must not switch unmarked leading
system messages to the client-order rule. The split test now runs with the
user turn plain and marked, so dropping the role check from the system
string test fails it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): honor cancel_on_disconnect on the rerank endpoints
* test(proxy): cover a connected rerank succeeding with cancel_on_disconnect on
* test(proxy): cover a connected rerank answer through the cancel watcher in the integration rig
* fix(proxy): read cancel_on_disconnect on rerank the same way the chat path does
* fix(proxy): set up the rerank request logger before routing so a hang-up after dispatch logs the query's tokens like chat
* fix(proxy): keep flag-off rerank logging unchanged and price a cut-off rerank from its query and documents
The rerank endpoint now builds its request logger only under cancel_on_disconnect, so a flag-off rerank logs one unbilled 408 row as on main.
The failure estimate for rerank counts the query and every document sent, prices the deployment the router picked, and bills one query on a model priced per query.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(mistral): add native upstream token counting support
* test(mistral): standardize on mistral-large-4 and add tool calls tests
* fix(mistral): unify credential resolution and guard Azure endpoint boundary
* fix(mistral): resolve api base from provider env vars and route responses count natively
* fix(test): use groq for bridged proxy tests and increase mistral coverage
Switch bridged provider tests in proxy token counter from mistral to groq, with comments noting that PR #50653 added native mistral token counting so groq tests the local estimate fallback. Also add test coverage for mistral error responses, invalid json, and system instruction blocks to satisfy codecov.
* fix(lint): fix unit package layout, ruff strict rules, and type discipline in mistral token counter
* fix(mistral): suppress basedpyright warnings on untyped httpx handler
* fix(mistral): keep chat credential resolution and count with the key chat resolves
The token counter now resolves its Mistral key and base through the same helper chat uses, so
a count goes to the endpoint chat would call. A deployment that would route to Azure, or an
Azure-only env key, falls back to the local estimate instead of sending that key to Mistral.
Chat key and base precedence is unchanged from main.
* fix(mistral): prefix live /v1/models ids with mistral/ so listed ids route
* fix(mistral): count follow-up turns, replayed reasoning, and file_id files the way Mistral chat sends them
Mistral has no native Responses API in litellm, so /v1/responses inference runs through the chat bridge,
which restores previous_response_id history. The count now takes the same bridge instead of forwarding
previous_response_id to Mistral's stateless count endpoint, so the separate Responses count method is gone.
The chat-format count adds a reasoning item where Mistral chat replays reasoning as a think chunk, and an
input_file part for files sent by file_id, so both reach Mistral's count the way inference sends them.
Tests now go through the real handler with respx, and the integration Mistral peer serves the count route.
* fix(mistral): leave empty assistant turns out of the count, as Mistral chat does
Mistral chat drops an assistant turn with no content and no tool calls before sending, and Mistral's count
endpoint rejects one. A follow-up whose stored turn spent its whole output budget on reasoning hit that 400
through previous_response_id, so the count now skips the same turns chat skips.
* fix(mistral): count what chat sends and keep the bundled Mistral model list
A user turn that mixed file_id files with an image or an inline file lost the image
or inline file in the count, since the converted content arrives as a tuple and only
lists were read. Both content readers now take either shape
A /v1/responses count with a json_schema text.format now sends that format to
Mistral, so the count matches what the bridged chat call bills
Counts for Mistral models that are not chat models (embeddings, OCR, transcription)
now fall back to the local estimate instead of returning Mistral's 400
Count calls now use the deployment's timeout when one is set, and a count response
without input_tokens falls back to the local estimate
MistralConfig.get_models goes back to the bundled model list, so turning on
check_provider_endpoint does not change what /v1/models shows for mistral/*
The previous_response_id count test now fakes the spend logs table instead of the
history lookup, so the real lookup and its query run
* fix(mistral): count tool_choice, the magistral reasoning prompt and system messages like chat, and never send the Azure key to Mistral
* fix(mistral): answer 408 on a count timeout and use the shared deployment timeout
Main's native counters answer a count timeout with 408 and a connection
error with 500. The Mistral counter fell back to a local estimate on both,
so a hung Mistral endpoint looked like a successful count. It now raises the
same count errors, and it reads the deployment timeout with the shared
deployment_timeout helper instead of its own copy
* fix(mistral): join count text blocks the way Mistral chat does
* fix(mistral): count file ids in order, drop the empty turns chat drops, and join tool result text like Mistral chat
* fix(mistral): count the thinking chunks Mistral chat replays in assistant history
The OpenAI-format count now takes Mistral chat's own replayed turns, so a Magistral assistant turn whose content
carries a native thinking chunk is counted as the reasoning chat sends, and a reasoning_content beside that chunk is
skipped the way chat skips it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): reuse one Google token across vector store searches
* fix(vertex_ai): load Vertex credentials once when a cold worker gets a burst
* test(vertex_ai): count Google token fetches for RAG Engine searches through the proxy
* test(vertex_ai): type the fixtures and token callback in the concurrent search test
* fix(vertex_ai): lock Google token loads per credential and reload a key Google rejects
A shared RAG Engine config made one store's slow token fetch hold up every
other store, and kept a rotated key file's deleted key until a restart.
First loads and expired refreshes now take a lock per credentials and project,
refreshes take a lock per credentials object, and any RefreshError reloads the
credentials once.
* fix(vertex_ai): share one Google token load or failure per credential and reload rejected keys on async calls
* test(vertex_ai): check async callers sharing an expired credential refresh it once
* fix(vertex_ai): reload Google credentials only when a refresh is rejected for good
A retryable refresh error such as a 503 now raises without reloading the key, so an outage
does not double google-auth's retries. Credentials that cannot be weak-referenced share the
config's refresh lock again, as on main. The burst test for a rejected key moves to VertexBase
and holds the token response until the other callers have started
* fix(vertex_ai): reload Google credentials only when Google answers invalid_grant
google-auth leaves its retryable flag False on metadata server, impersonation, and workload identity outages, and on a 502, so the previous check reloaded the key file on those outages and doubled the token tries. Now only an invalid_grant answer or "Reauthentication is needed" reloads
The sync reload also refills the cache under the key the caller asked with, so a call without vertex_project keeps one cached copy, as the async reload already did
* fix(vertex_ai): pick up a swapped Google key file on the next call
The credential cache key now carries the key file's inode, mtime and size, for a
vertex_credentials path and for GOOGLE_APPLICATION_CREDENTIALS. A key file that is replaced
or rewritten, such as a rotated Kubernetes secret, is loaded on the next call instead of
after Google rejects the old key or the proxy restarts
* fix(vertex_ai): keep the earlier key while a replaced key file does not load
A key file caught halfway through a non-atomic write, or replaced by one that does not load,
now keeps the credentials loaded from its earlier version in use and logs a warning, as the
cache did before key files were re-read. The next complete file is loaded on the next call.
The pub/sub test no longer writes a key file it only reaches through the seeded cache
* fix(vertex_ai): try a replaced key file again when its first token fetch fails
The earlier key now stands in only when the replaced file does not load. A token fetch that fails on the new key raises like any first load, so the next call tries the new key again instead of keeping the old one
* fix(vertex): raise when an async reload keeps an earlier key Google rejected
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(vertex_ai): label Vertex requests with the caller's key, team, org, and project
Adds the opt-in litellm.vertex_request_label_fields setting, the Vertex twin of bedrock_request_metadata_fields. When set, Gemini, embeddings, Imagen and Gemini image generation, rerank, and Claude on Vertex (chat and /v1/messages) send the listed proxy identity fields and spend_logs_metadata as request labels, identity first, with the user_api_key_ prefix reserved so a caller cannot forge or evict them. Identity text is rewritten to fit Google's label rules. Gemini image generation now forwards labels at all.
* feat(vertex_ai): send the opt-in identity labels on multimodal embeddings, the Google GenAI route, and batches
With litellm_settings.vertex_request_label_fields on, the multimodal embedding
predict body, the native generateContent and streamGenerateContent body, and the
BatchPredictionJob now carry the key's identity labels the same way chat does
Caller labels on GenAI are resolved after the extra_body merge, so neither a
top-level labels field nor extra_body.labels can write the reserved prefix. Batch
labels come only from the proxy's litellm_metadata, never the caller's OpenAI
batch metadata. With the setting off every route sends the same body as before
* fix(vertex_ai): keep identity labels when a Gemini caller sends extra_body labels
* fix(vertex_ai): label Interactions, image edits, Mistral, and the /vertex_ai pass-through, and gate Gemini image labels on the setting
* fix(vertex_ai): move body labels into the labels header on Vertex rawPredict pass-through calls
* fix(vertex_ai): send Mistral extra_body labels in the labels header instead of the body
Labels a caller nests in extra_body now merge over top-level labels before the
X-Vertex-AI-Labels header is built, the way the Gemini body merges them, and
leave the outbound body. The token-fetch stubs in the Mistral and multimodal
embedding tests are scoped to their instance so later tests see the real method
* fix(vertex_ai): limit pass-through labels to POST and to publishers that take the labels header
rawPredict to custom endpoints and publishers other than Anthropic and Mistral, and PATCH or PUT bodies, now reach Vertex as the caller sent them. The pass-through drops a caller's labels header where it writes body labels, a single field written as text turns the setting on for that field, and an undecodable Claude labels header falls back to the caller's labels
* fix(vertex): label Veo video, Gemini transcription, and pass-through predictLongRunning requests
* fix(vertex): read the pass-through publisher from the end of the upstream path
A project id named publishers shifted the first-match lookup, so a forged X-Vertex-AI-Labels header reached Claude or Mistral untouched
* fix(vertex): drop a caller labels header on body-labelled Vertex routes while identity labels are on
* test(vertex): script the v1beta1 GenAI target in the labels header integration test
* test(vertex): type the transcription test's access token override
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): answer 400, not 500, for a message without role or content
Anthropic, Azure AI Anthropic and Bedrock Claude counters now reject a
message missing role or content before calling the provider, with the
same 400 invalid_request_error the provider answers, instead of a 500
from the count transform.
* fix(count_tokens): count a chat message without content the way chat inference sends it, and check every Mantle count
* refactor(bedrock): leave message validation to the Bedrock count handlers
Main's Bedrock runtime and Mantle count handlers already answer 400 for a message without role or content before any AWS call, so the counter's own check was a second copy
* fix(count_tokens): name the chat message chat translation cannot send
A contentless system, developer, or tool message, or a message without a role, now answers "Message N must have a 'content' field" or "'role' field" instead of the translator's raw error text, which echoed the whole conversation back
* fix(bedrock): keep converse Claude deployments answering 400 for a chat message without content
Converse inference drops a non-assistant message without content, and for a last turn Claude then refuses the assistant prefill, so the count has no inference request to match. Invoke and Mantle deployments use Anthropic's chat translation and count the message the way their inference sends it
* fix(count_tokens): keep base counting for null-content tool and named turns on Bedrock converse
The converse check now covers only messages with just role and content, the
ones the merge base rejected. A content null message is never blamed for a
translation error, and a body whose messages all drop in translation answers
400 instead of a local estimate
* fix(count_tokens): reject only contentless user turns before counting on Bedrock converse
A system turn with content null counted on the merge base and converse
inference serves it, so it goes through chat translation again
* fix(count_tokens): keep the merge base's converse handling and leave null turns chat translation cannot send as given
* fix(count_tokens): keep a non-string message role from crashing the Anthropic count checks
* test(integration): expect the 400 for a count whose leading system message has no content
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): count thinking, tool_choice, and effort the way inference sends them, and count DeepSeek on its own count route
/v1/messages/count_tokens dropped the body's thinking, tool_choice, and output_config, so a Claude count with
tool_choice any came out lower than the bill. The route now passes those options to the provider counter, which
runs them through the deployment's own /v1/messages transform first, so the count sees what inference sends.
A body inference would reject now answers the same 400 before any provider call.
DeepSeek had no provider counter and fell back to a local estimate that ignores thinking. It now counts on
DeepSeek's Anthropic-compatible count_tokens route with the same base, key, tool sanitizer, and reasoning
mapping inference uses, and falls back to the estimate only when that route is missing.
* perf(proxy): delete count drop paths from one copy of the options
additional_drop_params on the count route now removes every path from a single deep copy of the request options, so a long client path list over a large schema stays linear
* fix(count_tokens): count deployment thinking defaults and match inference on betas, drops, timeouts, and DeepSeek listing
Count routes now merge the deployment's and router's default thinking and output_config under the request's own, the way inference does, so a deployment with thinking disabled counts what it bills
Claude counters send the thinking betas inference adds for thinking.display and block_binding, so those counts no longer answer 400
additional_drop_params that is not a list is skipped and non-string entries are ignored, the way inference skips them, instead of turning the count into a 500
The DeepSeek counter passes the deployment timeout to its count call, and DeepSeek model listing reads the bundled model list instead of recursing through get_valid_models
* fix(count_tokens): count deployment thinking on Responses and let a request drop list replace the deployment's
A Responses count now merges deployment and router defaults under the bridged chat body, so deployment thinking and reasoning_effort reach the count the way they reach chat inference
A request's additional_drop_params now replaces the deployment's list for both count drop passes, as inference does
The Anthropic, Azure AI, and DeepSeek counters read litellm_params through one typed helper instead of Any
* fix(count_tokens): estimate DeepSeek counts when the count route is down, and count deployment thinking on /utils/token_counter and Vertex Claude
A timeout, 429, 5xx, or connect error from DeepSeek's count endpoint now falls back to the local estimate instead of failing the count. A 401 or 422 still answers DeepSeek's own status
/utils/token_counter now counts the thinking and effort a DeepSeek deployment sets, and Vertex Claude counts run the Anthropic messages transform on the count options instead of sending none
* fix(count_tokens): send the Vertex count's thinking betas and count a Responses request's own thinking
A Vertex Claude count with thinking display or block binding now sends the beta headers Vertex inference adds, so Vertex no longer rejects a count for a request inference accepts
A Responses count now takes the request's own top-level thinking over the deployment's, the order inference uses, since the Responses-to-chat body never carried it
* fix(count_tokens): cap DeepSeek count waits, remember missing count routes, and count /utils request thinking
A DeepSeek count now waits at most 10 s even when the deployment sets no timeout, a 422 from the count route falls back to the estimate, and a base whose count route answered 404 or 405 is not called again for an hour. /utils/token_counter now passes the request's thinking, reasoning_effort, and tool_choice to the count
* fix(count_tokens): count on DeepSeek's endpoint only for api.deepseek.com and send Messages count options to Bedrock Mantle
* fix(bedrock): send the thinking beta headers on Mantle counts
A Mantle count with thinking display updates or block_binding now sends the
same anthropic-beta header Mantle inference sends
* fix(count_tokens): drop the region hint on a Mantle count 400 for forwarded count options and unskip the count options wire test
* fix(deepseek): count only the tools DeepSeek chat inference sends
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): keep carried end-user spend when a budget reset is retried
End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.
Fixes#50898
* fix(proxy): leave never-reset tiers unguarded and test each tier's own window
A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.
* test(proxy): give the end-user ledger its own fake prisma client
* fix(proxy): pin a reset time on tiers without one before end-user pages commit
* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced
* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset
* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration
* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window
* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time
A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users
* fix(proxy): name each end-user reset window by the tier's last write
Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due
* fix(proxy): name end-user reset windows by the tier and its advance count
Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.
* fix(proxy): keep carried spend for customer teams and members on a retried reset
Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.
The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.
* test(integration): list every budget column and relation in the budget update field check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A cost map with a top-level value that is not an object made a reload answer 500
after the new table was already published, so one worker billed new prices while
the others kept the old ones. The document is now refused before anything changes,
which also lets boot fall back to the bundled map instead of failing the import.
Also restores behavior the Rust cost map move changed for SDK callers:
- a document that starts with a UTF-8 or UTF-16 byte order mark loads again
- an alias_of row that carries other fields is served as its target
- register_model(url) fetches nothing when LITELLM_OFFLINE_MODE is set
- register_model(None) is a no-op again
- the public names RESERVED_TOP_LEVEL_KEYS, FALLBACK_GENERALIZATIONS_KEY,
RETRYABLE_FETCH_STATUS_CODES and git_blob_id import again, and so does
utils._runtime_registered_model_cost, which downstream test code replaces
litellm_core_utils/AGENTS.md now says which removed names come back.