mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-17 09:34:48 +00:00
* feat(alerts): open a prompt cache alert when a proven Anthropic prefix stops reading and writing the cache Behind general_settings.prompt_cache_alerts, which is off by default, a logger watches native Anthropic chat completions that send exactly one explicit five-minute cache breakpoint. A cache write the provider reported for that prefix proves it. Three later requests that each report zero cache reads and writes, all started after the proof and within its five minutes, save one alert for the deployment and configuration through the existing alert repository, so a dismissed or resolved alert is never reopened. Only the provider's raw response counts. A missing or contradictory usage is unknown, never zero. Streaming requests, failures and unknown usage end the streak, and so does a caching request whose prefix can't be compared. A read or a new write resets it. The prefix is a keyed hash, and the configuration covers the model, endpoint and a keyed digest of the API key, so nothing from the prompt or credentials is kept. Streaks live in the proxy's shared Redis, one Lua script per update, and requests are ordered by their own start and end so replays and overlaps can't add misses. Without Redis nothing is tracked. A Redis or database error never affects the request, and evidence lost to Redis ends its scope's streak before anything else counts. * fix(alerts): reset the whole deployment for unassessable prompt cache evidence and quiet a saved configuration Streamed attempts log an empty API key, so their evidence could not reach the streak of the assessed configuration. Streams, failures, missing usage and prefixes that can't be compared now reset every streak on their deployment. Assessed traffic keeps its credential, model and endpoint specific configuration. After an alert is saved, its configuration stays quiet in Redis for a day, so later misses don't write the same incident again. The quiet window is only set once the save succeeds, and a failed save can trigger again. Evidence lost to Redis now resets its deployment from a loop instead of recursion, and a lost reset older than any proof is dropped. Asyncio timeouts count as store failures on Python 3.10, and the module no longer uses StrEnum. The proxy imports and registers the logger only when prompt_cache_alerts is on, and removes it from the callback lists when the lifespan ends. * fix(alerts): keep the proxy lifespan a context manager and reset deployments seen while Redis is reloading Registering prompt cache alerts had moved the asynccontextmanager decorator from the proxy lifespan onto the new helper, which broke every proxy start. The decorator is back on _proxy_lifespan, and a lifespan test now checks that registration runs on entry and its cleanup on exit. A config reload can leave the shared Redis getter empty for a moment. Evidence seen then is remembered like evidence lost to a Redis error, so its deployment is reset before anything else reaches Redis once it is back. * test(alerts): let a failed completion raise its mapped error before asserting the deployment reset * fix(alerts): describe prompt cache alerts as advisory and type their boundaries Only callbacks that reach the logger and Redis are observed, so delayed, lost or unlogged requests can fall between the misses. The docstring now says an alert does not guarantee consecutive misses or an outage, and that a hit delivered after the stored proof expired leaves the prefix unproven until a new write. Tests pin both behaviors. The logger no longer calls the partially typed CustomLogger constructor, whose two attributes every reader already defaults, and registration reads the typed general settings view. * fix(alerts): keep retained prompt cache misses alive as long as a refreshed proof A proof or hit that returned before already stored misses but arrived after them extended the proof key while the misses kept their older expiry, so they could vanish early and an observed streak would be lost. Both refresh branches now give the retained misses the proof's expiry. * fix(alerts): serialize each message's fields once in the prompt cache prefix Every block's identity carried its message's other fields, so a large field on a message with many blocks made the canonical prefix grow with fields times blocks. The message's fields are now one segment ahead of its blocks, keeping role, metadata, block order and fail-closed redaction in the prefix while its size stays linear in the request. * test(proxy): keep a leaked worker config out of the prompt cache lifespan test An earlier test leaves WORKER_CONFIG in the environment, and the lifespan then loads its config path. The registration test now clears it with the database URLs. * test(alerts): clean each prompt cache deployment tag in its own comprehension --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|---|---|---|
| .. | ||
| response_cache_case.py | ||
| test_cache_max_messages.py | ||
| test_embedding_file_block_cache.py | ||
| test_evicted_client_closer_real_io.py | ||
| test_in_memory_cache_heap_bound.py | ||
| test_prompt_cache_alert_streaks_real_io.py | ||
| test_response_cache_chaos.py | ||
| test_response_cache_empty_output.py | ||
| test_response_cache_model_group_key.py | ||
| test_response_cache_stream_replay.py | ||
| test_valkey_semantic_cache_real_io.py | ||