2 commits

Author	SHA1	Message	Date
devin-ai-integration[bot]	e136e91273	fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl (#46084) ... * fix(router): keep the prompt caching affinity pin alive as long as the prefix's cache_control ttl A prefix marked cache_control ttl 1h lost its pinned deployment after the fixed 300 s, so the next request went to another deployment and paid a cold cache. The pin TTL now follows the longest ephemeral breakpoint in the cacheable prefix, tools included: 3600 s for a 1h breakpoint and 300 s otherwise. * fix(router): give prompt cache pins their own tier and honour any declared ttl Pins now live in a DualCache of their own instead of the router's 200-slot in-memory tier, so an hour-long pin no longer evicts cooldowns and rate windows. The pin lifetime follows any <n>s|m|h cache_control ttl, floored at 300 s, not only the literal "1h". * fix(router): hash the pin ttl off the event loop and flush prompt cache pins with the router cache Compute the cache_control ttl scan through offload_token_count beside the prefix hash, clear the dedicated pin cache in Router.flush_cache, and point the order-fallback and priority-group pin tests at the pin cache they read * fix(router): pin 1h prefixes only on 1h-capable models, cap the pin at 1h, and give pins 10k in-memory slots * fix(router): match the 1h pin gate to what Bedrock forwards Bedrock deployments now decide the 1h pin with is_claude_4_5_on_bedrock, the same check that decides whether the 1h ttl reaches Bedrock, so an application inference profile ARN keeps a 5 minute pin whatever its base_model prices * test(router): give the anthropic_messages pin fixture its provider * fix(router): let each pin write set its own lifetime and require a positive 1h price In a group mixing 1h and 5m deployments the in-memory pin kept the lifetime of its first write, so a 5m deployment could hold an hour-long pin. Each write now clears the in-memory entry first, matching Redis. The Bedrock gate uses supports_bedrock_prompt_cache_ttl after #46058, and a zero 1h price no longer counts as 1h support. * test(router): integration cells for the one-hour prompt cache pin lifetime Cover every surface the pin lifetime touches: chat and /v1/messages through both SDKs, the responses bridge, response-cache hits, each provider family's 1h gate, malformed and duplicated ttl values, rewrites of one prefix, mixed and cross-group routing, concurrency, every Redis attach path, the in-memory slot pressure without Redis reads, a Redis outage and a killed worker * fix(caching): bound the in-memory expiry heap and count prompt cache pins in the memory debug routes * test(integration): cover prompt cache pins in the memory debug routes and the bounded expiration heap * fix(caching): serialize in-memory cache writes so a heap rebuild cannot lose another thread's expiry entry * fix(caching): keep the in-memory cache bounded when entries have no usable expiry * test(caching): cover the response cache bound when cached entries have no usable ttl --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-14 21:10:09 -07:00
devin-ai-integration[bot]	a76b59db9f	test(proxy): move middleware, spend_tracking, pass_through, common_utils and root proxy tests into tests/unit/proxy (#44015) ... Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-01 18:23:31 +00:00

Renamed from tests/test_litellm/proxy/common_utils/test_debug_utils.py (Browse further)