litellm/tests/integration/caching
joshua-berri 5792794691
feat(alerts): report observed Anthropic cache misses as advisories (#51973)
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache

Behind general_settings.prompt_cache_alerts, which is off by default, a
logger watches native Anthropic chat completions that send exactly one
explicit five-minute cache breakpoint. A cache write the provider reported
for that prefix proves it. Three later requests that each report zero
cache reads and writes, all started after the proof and within its five
minutes, save one alert for the deployment and configuration through the
existing alert repository, so a dismissed or resolved alert is never
reopened.

Only the provider's raw response counts. A missing or contradictory usage
is unknown, never zero. Streaming requests, failures and unknown usage end
the streak, and so does a caching request whose prefix can't be compared.
A read or a new write resets it. The prefix is a keyed hash, and the
configuration covers the model, endpoint and a keyed digest of the API key,
so nothing from the prompt or credentials is kept.

Streaks live in the proxy's shared Redis, one Lua script per update, and
requests are ordered by their own start and end so replays and overlaps
can't add misses. Without Redis nothing is tracked. A Redis or database
error never affects the request, and evidence lost to Redis ends its
scope's streak before anything else counts.

* fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration

Streamed attempts log an empty API key, so their evidence could not reach
the streak of the assessed configuration. Streams, failures, missing usage
and prefixes that can't be compared now reset every streak on their
deployment. Assessed traffic keeps its credential, model and endpoint
specific configuration.

After an alert is saved, its configuration stays quiet in Redis for a day,
so later misses don't write the same incident again. The quiet window is
only set once the save succeeds, and a failed save can trigger again.

Evidence lost to Redis now resets its deployment from a loop instead of
recursion, and a lost reset older than any proof is dropped. Asyncio
timeouts count as store failures on Python 3.10, and the module no longer
uses StrEnum. The proxy imports and registers the logger only when
prompt_cache_alerts is on, and removes it from the callback lists when
the lifespan ends.

* fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading

Registering prompt cache alerts had moved the asynccontextmanager decorator
from the proxy lifespan onto the new helper, which broke every proxy start.
The decorator is back on _proxy_lifespan, and a lifespan test now checks
that registration runs on entry and its cleanup on exit.

A config reload can leave the shared Redis getter empty for a moment.
Evidence seen then is remembered like evidence lost to a Redis error, so
its deployment is reset before anything else reaches Redis once it is back.

* test(alerts): let a failed completion raise its mapped error before asserting the deployment reset

* fix(alerts): describe prompt cache alerts as advisory and type their boundaries

Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view.

* fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof

A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry.

* fix(alerts): serialize each message's fields once in the prompt cache prefix

Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request.

* test(proxy): keep a leaked worker config out of the prompt cache lifespan test

An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs.

* test(alerts): clean each prompt cache deployment tag in its own comprehension

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-16 15:36:07 -08:00
..
response_cache_case.py fix(caching): keep reasoning, citations, and audio on streamed cache hits (#32096) 2026-10-29 10:08:12 -07:00
test_cache_max_messages.py test(integration): look up owned cache keys and clean up sweep models (#47255) 2026-10-18 10:27:53 +00:00
test_embedding_file_block_cache.py feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305) 2026-10-08 13:20:03 -07:00
test_evicted_client_closer_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00
test_in_memory_cache_heap_bound.py fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) 2026-10-14 21:10:09 -07:00
test_prompt_cache_alert_streaks_real_io.py feat(alerts): report observed Anthropic cache misses as advisories (#51973) 2026-11-16 15:36:07 -08:00
test_response_cache_chaos.py test(integration): give each owned proxy a private database and Redis by default (#47213) 2026-10-18 03:37:41 +00:00
test_response_cache_empty_output.py test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
test_response_cache_model_group_key.py fix(caching): read the cache key's model group from the router's metadata for the call (#48917) 2026-10-29 05:35:23 -07:00
test_response_cache_stream_replay.py fix(caching): keep reasoning, citations, and audio on streamed cache hits (#32096) 2026-10-29 10:08:12 -07:00
test_valkey_semantic_cache_real_io.py ci(unit): cut unit check time with cheaper per-test resets and fewer non-unit tests (#51864) 2026-11-14 15:41:30 -08:00