litellm/tests/unit/proxy/hooks
joshua-berri 076ded32be
feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066)
* feat(auto_router): report a session's whole spend on /auto_router/session

Claude Code sends its background calls straight to Haiku under the same session id, so those requests never
reached the auto-router session rollup and the status line under-reported the session. Every billed spend
row in a caller-supplied session now increments a key-scoped Redis counter whose one-hour expiry each write
resets, and the endpoint reports it as total_spend, with spend unchanged. The increment runs where the spend
row is accepted, so it counts each row exactly as often as the key's spend counters do. It reads as no total
without a shared Redis

* fix(auto_router): name the unsupported provider when a turn routes off Anthropic

A turn routed to a non-Anthropic tier with no native wire response, such as a Cloudflare DeepSeek model,
failed strict native validation and was logged as observation_unavailable, which reads like a broken
capture. It now reports unsupported_selected_provider. Turns with a native wire response keep their
existing reasons, and no estimate is added, so savings for these turns stay unknown

* docs(auto_router): explain when session savings stay unestimated

saved_spend on /auto_router/session now says a turn inside the longest prompt-cache TTL of the session's
first observed turn, or of its last incomplete observation, is not estimated, so a null there is expected
for that window

* fix(auto_router): say why a system-only cache prefix has no savings estimate

Claude Code puts its cache breakpoint on the system prompt, so the baseline prefix count sends Anthropic a
body with no messages, which it rejects, and every turn read as token_count_unavailable. A count that fails
only on system or tools prefixes now reports uncountable_system_or_tools_prefix. Gateways that accept the
empty body still count it. A live check against claude-opus-5-5 showed a differential count with a
placeholder message is off by -2 tokens for system-only and +68 for tools, so these turns stay unestimated
and the savings gap for Claude Code is not fixed here

* fix(auto_router): declare the session total's Redis key family and test its expiry

The session total read and wrote Redis without a service target, so its calls showed up as bare redis
spans. Both now run under session_spend_totals. The key no longer hashes the api key, which is already a
hashed token, and only digests the session id. A new test drives Redis expiry on a test clock to show an
active session keeps its total and an idle one restarts

* docs(auto_router): note which turns are still estimated inside the history window

Turns that took the identical baseline path, and turns whose whole cached prefix the session had already
written, are estimated even inside the first cache TTL window, so saved_spend now says so

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-11-10 17:07:11 -08:00
..
litellm_skills test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
__init__.py ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests (#42903) 2026-09-24 22:59:11 +00:00
test_async_post_call_streaming_iterator_hook.py fix(guardrails): run the end-of-stream post_call scan when the client disconnects mid-stream (#43839) 2026-10-05 01:44:43 -07:00
test_autorouter_baseline_cache.py feat(auto_router): report a session's whole spend and name why savings stay unestimated (#51066) 2026-11-10 17:07:11 -08:00
test_banned_keyword_list.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
test_batch_enqueued_tokens.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
test_batch_file_validation.py refactor(proxy): expose public names for private proxy helpers (#45170) 2026-10-07 18:59:27 -07:00
test_batch_rate_limiter.py feat(redis): serve the proxy's common Redis call with Rust stores (#50170) 2026-11-06 06:01:11 +00:00
test_dynamic_rate_limiter.py refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
test_dynamic_rate_limiter_v3.py feat(redis): serve the proxy's common Redis call with Rust stores (#50170) 2026-11-06 06:01:11 +00:00
test_image_generation_guardrails.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
test_key_management_event_hooks.py refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
test_max_budget_per_session_limiter.py fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473) 2026-11-07 17:50:06 +09:00
test_max_iterations_limiter.py refactor(typing): clear about 1,100 basedpyright Any errors across 303 files (#49986) 2026-11-06 13:31:15 -08:00
test_model_budget_shared_windows.py feat(proxy)!: model family and access group entries in model_max_budget (#47722) 2026-10-28 15:40:57 -07:00
test_model_max_budget_limiter.py feat: add OpenRouter-style -latest model aliases for first-party providers (#50315) 2026-11-06 18:22:38 +08:00
test_parallel_request_limiter.py fix(proxy): count native Responses WebSocket turn tokens toward key and tag TPM limits (#49257) 2026-10-31 02:53:27 -07:00
test_parallel_request_limiter_v3.py feat(rate-limit): enforce per-customer per-model tpm and rpm limits in the v3 limiter (#50857) 2026-11-10 09:22:18 -08:00
test_post_call_failure_hook_integration.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
test_post_call_response_headers_hook.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
test_post_call_streaming_hook_integration.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
test_post_call_success_hook_integration.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
test_prompt_cache_observer.py fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure (#45136) 2026-10-16 21:19:58 +00:00
test_prompt_injection_detection.py refactor(model-cost): move the cost map inputs, fetch and register_model overlay into Rust (#50501) 2026-11-10 02:37:35 +00:00
test_proxy_hooks_init.py feat(accounting): add opt-in SpendLogs ledger (#49856) 2026-11-04 17:49:15 -08:00
test_proxy_rate_limit_provider_field.py refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) 2026-10-29 17:26:33 +00:00
test_proxy_track_cost_callback.py fix(proxy): write SpendLogs failure rows for config-defined pass-through endpoints (#41831) 2026-11-09 17:05:10 -08:00
test_rate_limiter_toctou.py feat(redis): serve the proxy's common Redis call with Rust stores (#50170) 2026-11-06 06:01:11 +00:00
test_search_free_tier.py fix(proxy): scope the search free-tier counter to the request's provider (#47646) 2026-10-20 22:01:53 +00:00
test_send_invite_email.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00
test_sensitive_data_routing.py fix(redis): share one native circuit breaker between RedisCache and the Rust stores (#50473) 2026-11-07 17:50:06 +09:00
test_tpm_concurrent.py feat(redis): serve the proxy's common Redis call with Rust stores (#50170) 2026-11-06 06:01:11 +00:00
test_unit_test_max_model_budget_limiter.py fix(router): inspect image content without translating routed requests (#49371) 2026-10-31 10:44:24 -07:00
test_user_management_event_hooks.py test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy (#43998) 2026-10-01 10:11:45 -07:00