mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
71dfab7177
Auto-routed requests were indistinguishable from ordinary ones once logged: the spend log recorded the requested model group and the resolved deployment, but nothing about which tier was chosen or what chose it. That information existed only inside verbose_router_logger f-strings, so answering "why did my prompt land on the cheap model" required log access and a running proxy. The complexity, quality, and adaptive pre-routing strategies now return a typed StandardLoggingRoutingDecision on their PreRoutingHookResponse, and Router.async_pre_routing_hook records it once for every attempt. Those three previously side-channelled their own state through three different metadata keys; the decision now travels on the hook contract itself, so the bucket is resolved in one place, through get_or_create_metadata_bucket, which already owns the question of which dict holds proxy-internal metadata and replaces a non-dict value instead of skipping the write. Recording happens on every attempt rather than only on a successful route: a fallback from an auto-router group to a plain group re-enters the hook with the same request kwargs, and a decision left behind there would attribute the first router's tier to the deployment that actually served the retry. The log details drawer renders the result as a Routing card between Request Details and Metrics; the card is absent on rows that carry no decision, so ordinary and pre-upgrade rows are unchanged. Three defects surfaced while making the recorded cause truthful, each of which would have persisted a wrong answer. The complexity router hardcoded cause=complexity_scorer even when the LLM classifier decided, and its silent fallback to the heuristic on classifier failure meant a row could claim an LLM verdict the LLM never gave; the cause now reports the path that actually ran. The keyword that triggered a tier rule was discarded before logging, as was the escalation keyword. The 2-reasoning-marker override returned REASONING with a score far below the REASONING boundary and no marker saying so, which reads as a scoring bug to anyone comparing the two; it now emits a reasoning-override signal, and the card labels those rows as an override instead of claiming the score met a boundary. The LLM path no longer reports a synthetic score of 1.0, and heuristic decisions carry a snapshot of the tier boundaries that mapped the score, so a historical row stays interpretable after the boundaries change. Signals name a matched term only when the caller's own message contains it. Scoring still reads the system prompt, but a term matched solely there is reported as a count, since signals reach a spend row the caller can read and naming one would disclose a term from a prompt it cannot see. routing_decision is stripped from caller-supplied metadata at ingress, so a client cannot forge its own provenance. |
||
|---|---|---|
| .. | ||
| test_budget_reservation_redis_failure.py | ||
| test_cloudzero_endpoints.py | ||
| test_compression_savings.py | ||
| test_savings.py | ||
| test_spend_log_error_logger.py | ||
| test_spend_management_endpoints.py | ||
| test_spend_query_optimization.py | ||
| test_spend_tracking_utils.py | ||