* test(gemini): cover client-mangled thought signatures in tool call ids
* fix(gemini): ignore a client-mangled thought signature in a tool call id
Only replay the signature embedded in a tool call id when it is still valid base64. Clients that strip or replace +, / and = in ids now get the existing placeholder fallback instead of a Vertex 400 on every later turn. Fixes#52001.
* fix(anthropic): close /v1/messages thinking blocks with a signature and keep bridged output
* fix(anthropic): close rebuilt-stream thinking blocks with a signature_delta
* fix(anthropic): drop unsigned thinking before Claude, stream raw reasoning_text, close rebuilt thinking blocks with signature_delta
A Claude target on /v1/messages now gets unsigned thinking blocks left out of the request, so a switch from a reasoning_content model makes one call and keeps thinking, except when that would leave the final tool turn without its opening thinking block
Raw response.reasoning_text deltas from the Responses bridge stream as thinking, and a stream rebuilt from a full answer or rewritten by a guardrail closes each thinking block with a signature_delta, the real signature or an empty string
* fix(anthropic): stream a whole reasoning item as a closed thinking block
A Responses stream can send a reasoning item whole in output_item.added and output_item.done with no text deltas, as litellm's own stream does for gpt-oss on OpenRouter. The /v1/messages stream now builds that item's thinking block with the same helper the non-streamed response uses, so streamed and non-streamed answers carry the same reasoning
* test(anthropic): mark the new streaming test locals Final
* fix(anthropic): drop unsigned thinking before every Claude call, a pending tool turn included
Anthropic answers 400 to a thinking block with an empty signature but accepts a
tool turn with no thinking, so the strip no longer keeps the block that opens the
pending tool turn. The check now looks for "claude" in the model name, which covers
Claude behind every messages route and the chat adapter, not only four providers
* fix(anthropic): strip unsigned thinking when a deployment's base_model names Claude
* fix(anthropic): drop unsigned thinking for an alias whose base_model names Claude on the native messages path
The chat adapter already read base_model before leaving unsigned thinking out of a Claude request. The native messages transform and the Rust request transform now read it too, so an aliased Claude deployment gets one call instead of a 400 and a retry
* test(anthropic): build the pending tool turn without mutating it in the Rust strip test
The Rust test now builds its expected messages from the same helper instead of editing them afterward, and a doc comment that only restated a helper's name is gone
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend-logs): match key aliases exactly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(spend-logs): drop unused key alias ORM lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate exact key alias API descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-logs): keep key alias filter matching CLI session logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(providers): cover upstream 403 from OpenAI-compatible providers as permission denied and not retried
* fix(exceptions): map upstream 403 to PermissionDeniedError for OpenAI-style providers
* fix(bedrock): split the OpenAI breakpoint and Bedrock cachePoint model flags
supports_prompt_cache_breakpoint meant both "uses the OpenAI prompt_cache_breakpoint
format" and "Bedrock Converse takes cachePoint blocks", so an OpenAI-format opt-out on a
Bedrock Claude deployment silently turned off its prompt caching.
Bedrock now reads only supports_bedrock_converse_cache_point, then supports_prompt_caching
as before. The OpenAI path reads supports_openai_prompt_cache_breakpoint and keeps the old
key as a deprecated alias. The Kimi K3 catalog rows move to the new Bedrock key, and a
Bedrock deployment still on the old key logs a one-time migration warning.
Fixes#52140
* fix(bedrock): expose the split prompt cache flags in get_model_info and word the migration warning for unlisted models
* test(bedrock): mark the system prompt directly in the routed cache point regression
* fix(bedrock): warn about the old breakpoint flag even when a catalog cache point flag decides
* fix(cost-map): keep the old Kimi K3 breakpoint flag for clients that read the remote catalog
* test(bedrock): keep the old breakpoint flag on catalog cache point opt-outs for older clients
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache
Behind general_settings.prompt_cache_alerts, which is off by default, a
logger watches native Anthropic chat completions that send exactly one
explicit five-minute cache breakpoint. A cache write the provider reported
for that prefix proves it. Three later requests that each report zero
cache reads and writes, all started after the proof and within its five
minutes, save one alert for the deployment and configuration through the
existing alert repository, so a dismissed or resolved alert is never
reopened.
Only the provider's raw response counts. A missing or contradictory usage
is unknown, never zero. Streaming requests, failures and unknown usage end
the streak, and so does a caching request whose prefix can't be compared.
A read or a new write resets it. The prefix is a keyed hash, and the
configuration covers the model, endpoint and a keyed digest of the API key,
so nothing from the prompt or credentials is kept.
Streaks live in the proxy's shared Redis, one Lua script per update, and
requests are ordered by their own start and end so replays and overlaps
can't add misses. Without Redis nothing is tracked. A Redis or database
error never affects the request, and evidence lost to Redis ends its
scope's streak before anything else counts.
* fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration
Streamed attempts log an empty API key, so their evidence could not reach
the streak of the assessed configuration. Streams, failures, missing usage
and prefixes that can't be compared now reset every streak on their
deployment. Assessed traffic keeps its credential, model and endpoint
specific configuration.
After an alert is saved, its configuration stays quiet in Redis for a day,
so later misses don't write the same incident again. The quiet window is
only set once the save succeeds, and a failed save can trigger again.
Evidence lost to Redis now resets its deployment from a loop instead of
recursion, and a lost reset older than any proof is dropped. Asyncio
timeouts count as store failures on Python 3.10, and the module no longer
uses StrEnum. The proxy imports and registers the logger only when
prompt_cache_alerts is on, and removes it from the callback lists when
the lifespan ends.
* fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading
Registering prompt cache alerts had moved the asynccontextmanager decorator
from the proxy lifespan onto the new helper, which broke every proxy start.
The decorator is back on _proxy_lifespan, and a lifespan test now checks
that registration runs on entry and its cleanup on exit.
A config reload can leave the shared Redis getter empty for a moment.
Evidence seen then is remembered like evidence lost to a Redis error, so
its deployment is reset before anything else reaches Redis once it is back.
* test(alerts): let a failed completion raise its mapped error before asserting the deployment reset
* fix(alerts): describe prompt cache alerts as advisory and type their boundaries
Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view.
* fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof
A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry.
* fix(alerts): serialize each message's fields once in the prompt cache prefix
Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request.
* test(proxy): keep a leaked worker config out of the prompt cache lifespan test
An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs.
* test(alerts): clean each prompt cache deployment tag in its own comprehension
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(alerts): read and show prompt cache alerts alongside the existing alert types
Saved alerts can now carry a prompt_cache type on a deployment resource,
with evidence made of opaque fingerprints and counters only. One incident
exists per deployment and cache configuration, whatever window or prefix
found it, so a dismissed or resolved one stays the only one.
The performance list, the daily summary, the free-alert allowance and the
history lookup now name the four metric types instead of excluding model
deprecations, so these rows never reach metric charts, digests or the quota.
The All view lists prompt cache alerts from their own query, leaving the
performance query and its pagination unchanged. An alert's page shows the
deployment, model, misses in a row, the proving write, the cache lifetime
and when the misses happened, in place of a metric chart, and keeps the
existing dismiss, resolve and reopen controls.
* fix(alerts): pin prompt cache evidence to what is observed and prove the readers against the database
Prompt cache evidence now only accepts the anthropic provider and the
five-minute TTL. The title says cache usage was not observed rather than
that caching stopped, and the scope copy names the checked requests:
non-streaming Anthropic Chat Completions with one five-minute breakpoint,
after a confirmed write, with never-proven prefixes left unassessed.
Alerts are presented by their evidence shape. Metric labels come from the
evidence unit and the activity link label skips deployments, so the
unrecognized-alert fallback is gone.
The real-database alert tests now round-trip a prompt cache alert, keep it
out of the performance list and the daily summary, keep a dismissed one
as the only incident, and leave it out of the free allowance. They replace
the unit tests that only checked SQL arguments. The UI tests reopen an
incident, page both lists in All from their own cursors, and drop loaded
prompt cache alerts when Performance is chosen.
* chore(ui): regenerate API types for prompt cache alerts
Alerts can now carry the prompt_cache type, the deployment resource and prompt cache evidence, so the generated dashboard types include them.
* fix(alerts): satisfy the dashboard lint and the bounded IN check for prompt cache alerts
The prompt cache list query reuses useAlerts with the prompt_cache type, so
no five-parameter helper is needed. The new prompt cache mock handling in
the Alerts integration test lives in its own function, a presentation test
names its expected details, and AlertDetail is formatted. The performance
list's IN filter over the four fixed metric types records that bound.
* fix(alerts): reject metric alerts on a deployment resource
Only prompt cache alerts use the new deployment resource. Metric alerts have no deployment history source, so a metric alert on a deployment is now invalid. Every combination accepted before this change is still accepted.
* test(alerts): match the plain token count the prompt cache evidence renders
* fix(ui): describe prompt cache alerts as advisory observed evidence
The alert reports requests the proxy observed. Delayed, missing or unlogged requests can fall between them, so the list and detail say they are not necessarily consecutive and do not prove an outage. The serialized consecutive_misses field is unchanged.
* fix(ui): title prompt cache alerts as advisories and attribute the limit to observations
The title no longer reads as a deployment-wide finding, and the limitation says delayed, missing or unlogged observations mean the requests are not necessarily consecutive and do not prove a caching outage.
* docs(alerts): describe consecutive_misses as observed requests that are not necessarily consecutive
* chore(ui): regenerate API types for the advisory prompt cache evidence description
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): resolve admin MCP authorization without listing every key and team
Co-Authored-By: bot_apk <apk@cognition.ai>
* test(proxy): pin the compact admin MCP user info shape and describe it in the query parameter
Co-Authored-By: bot_apk <apk@cognition.ai>
* test(proxy): cover admin MCP user info per role and after a worker kill on a proxy over the connector byte limit
Co-Authored-By: bot_apk <apk@cognition.ai>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
* fix(dynamic_rate_limiter): keep counting and reporting priority requests after a priority pool's tokens run out
* fix(dynamic_rate_limiter): return the tracked priority statuses as a tuple
* fix(dynamic_rate_limiter): read optional rate limit fields without assuming they are set
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The alerts monitoring response now carries model_deprecations_enabled (#51418), and the
licensed integration gateway reports no free alert limit. The migration job now builds the
DailyUserSpend index it owns (#50666). /key/generate rejects guardrail names the proxy does
not know (#51080), so the E2 key is created on the proxy that defines the guardrail. The key
detail tab is kept in the URL (#51104), so the reload spec opens Overview before reading the cap.
* test(auto_router): expect the session total_spend reported since #51066
Every exact /auto_router/session contract in the integration test now
includes total_spend, and the session reads wait until the Redis total is
recorded instead of only for the rollup row.
* test(auto_router): wait for the session total even once the rollup row exists
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(gemini): send reasoning_effort xhigh as thinkingLevel xhigh for gemini-4-hydrogen
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(gemini): allow the one in-place thinkingLevel write now that the field is ReadOnly
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(proxy): hold the logging object weakly in the detached stream failure hook
The closure armed by _arm_detached_stream_failure_hook is stored on the
logging object and captured the logging object, the request data that
contains it, and the processing object that holds that data. Every
streamed /v1/messages request therefore ended in a reference cycle and
stayed in memory until a full cyclic GC pass instead of being freed by
reference counting when the stream finished.
Capture the logging object through a weakref, a copy of the request data
without the litellm_logging_obj key, and the route type as a value. The
hook adds the live logging object back to the request data when it fires.
All three callers reach the hook through the logging object, so it is
always alive at that point.
Add a regression test that streams a request to completion, drops every
reference, and checks with the cyclic collector disabled that the logging
object is gone.
Fixes#51277
* test(proxy): cover finished streamed /v1/messages calls being freed without a gc pass
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): annotate the finished-stream gc test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): split the freed-calls assert message under the line limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dynamic_rate_limiter): let customer orgs and keys borrow idle model capacity and split class shares
Replace the per-team borrowing and class share booleans with model_limit_borrowing_config.borrow_idle_capacity
and priority_class_share_config.split_equally_among_active, which list the entity types they apply to.
Use one borrow cap field, model_limit_max_borrow_multiple, on team metadata, key metadata and the customer
organization budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): size holds with the tightest share, fail closed on websocket turns, guard lookalike team ids and the borrow cap
One token estimate now covers every share a request holds, using the tightest TPM share's output cap.
A team id spelled like a key or customer organization holder falls back to the class pool.
model_limit_max_borrow_multiple in key or team metadata is changed by proxy admins only.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): keep a team's key on the team's class share slice and reset websocket share state per turn
A team listed for the class share split now holds the slice ahead of customer headers, which the caller chooses.
A websocket turn clears the shares of borrowers that do not apply on its model, so it is not settled on an earlier model's share.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dynamic_rate_limiter): read the borrow cap from model_limit_borrowing_config.max_borrow_multiple instead of a budget table column
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dynamic_rate_limiter): a fallback target a borrower has no limit on no longer charges its share on the failed model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): keep a deployment's per-second video price over the catalog row's flat price
* fix(cost): drop the superseded flat video price from the registration kept for reloads
* test(cost): derive the per-second rate from the catalog row so the regression holds at any vendor price
* fix(cost): keep the flat video price when only a resolution tier is priced per second
* test(integration): assert the body sent to fal in the per-second price test
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Nate Armstrong <narmstrong@berri.ai>
* fix(router): stop the rust router keeping every finished request in memory until a full gc
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove finished /v1/messages and chat calls free their log record without a gc pass
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read the memory endpoint with typed helpers instead of a type ignore
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): bill a video request without a resolution at the row's default rate
* test(fal_ai): drop the assertion that two fal rates differ
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): put extra_body fields inside modelInput for async-invoke embeddings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep clientRequestToken and tags at the top level of async-invoke requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): cover clientRequestToken staying top-level in the async-invoke wire test
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(perplexity): bill cached prompt tokens at the cache read rate when the response has no cost
* fix(perplexity): bill cached tokens at the off-peak cache read rate when the model has no standard one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(fal_ai): send versioned seedream and flux-pro image models to the endpoint they name
* fix(fal_ai): reject dot segments in the model path and tighten the versioned endpoint tests
* test(fal_ai): match the dot segment error message
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run the /fal_ai passthrough price gate on the path sent upstream
The gate checked the raw request path for /requests/ and priced the raw path, so a path that
collapses to a submit URL upstream (a /requests/.. dot segment, or an encoded ? or # tail)
skipped the gate and reached fal on the proxy's key
* test(proxy): use an endpoint with no catalog row in the fal passthrough gate test
* fix(proxy): price the fal passthrough gate on the encoded upstream path
* fix(proxy): strip the fal queue base prefix in the form it is sent upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep an exhausted key on model listing and token counting routes
The header-only preflight added in #51428 ran the rate limit check on every LLM API route, so a key that had used up its rpm limit got 429 on GET /v1/models, GET /models and POST /utils/token_counter, routes whose handlers never run the limiter. The preflight now skips the limit check on GET requests, which carry no body to avoid reading, and on unbilled routes.
* fix(proxy): still refuse a GET that declares a body before reading it
A GET can carry a body, so the limit check is only skipped for a GET that declares none. A metered route such as /v1/batches keeps its 429 before a declared body is buffered.
* test(proxy): fail the preflight test on any outcome other than admitted or 429
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): support PostgreSQL 12 delivery migrations
The SpendLogDelivery migration defaulted delivery_id to gen_random_uuid(), which
PostgreSQL 12 lacks without pgcrypto, so migrate deploy failed there. The writer
always supplies delivery_id, so the column no longer needs a database default.
Databases that already applied the original migration drop the default in a
follow-up migration without rewriting rows.
* test(spend): apply the ledger enum migration before the delivery migration tests
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* test(unit): reset module state only when a test changed it
The proxy conftest deep-copied every snapshotted litellm attribute before each
test, and the copy of litellm.model_cost alone was a median 0.12 s for each of
56,031 proxy tests on the runner. The reset now copies an attribute back only
when it no longer equals the snapshot.
The enterprise and MCP server conftests reloaded litellm before every test,
3.27 s and 0.21 s per test. They now use the same reset.
* test(unit): remove six flaky tests
They failed on main without a code change behind them between 2026-11-05 and
2026-11-14: the vector store hook SDK fallback, the concurrent user config TPM
count, three Prometheus metrics server port tests and the OpenCode list cost
billing test.
* ci: run four heavy unit shards on 16-core runners and keep two smoke jobs off pull requests
proxy-feature-endpoints, proxy-endpoints, root and llms-providers ran 1.9 to 2.2
times faster with 8 workers on a 16-core runner in a benchmark run.
windows-sdk and proxy-build-from-pip still run on pushes to main and manual
runs. proxy-build-from-pip installs a published release and never runs the
pull request's code.
* test: move 109 tests that start a process or open a socket into tests/integration
tests/unit is in-process by its own rules, and these tests spawn a Python
interpreter, a shell or a local TCP or HTTP server. They took 921 s of unit
shard time per run, 396 s of it in the MCP client transport tests. Each one now
lives in a *_real_io.py file under the integration directory for its domain and
runs with that directory's group.
The tests themselves are unchanged apart from one repository-root path in the
CLI test.
* test(unit): copy back only the dict entries a test changed
A test module that registers models at import leaves litellm.model_cost four
entries larger than the snapshot before every test in its directory, so the
reset still deep-copied the whole price map for each of them. It now rebuilds a
changed dict from the live entries that still match and copies only the rest.
* ci(smoke): keep the two jobs that skip on pull requests out of the smoke gate
A skipped job passing the gate is what test_unit_passed_gate guards against,
so the gate goes back to requiring success from every job it needs and
windows-sdk and proxy-build-from-pip stop being part of it. They still run
and report on pushes to main and manual runs.