mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-14 23:21:35 +00:00
test(e2e): record that the #25846 cache-token path is unsupported live
Probed real bedrock (commercial us-east-1) with a 14713-token cached prefix over streaming /v1/messages. Its raw message_delta already carries the cache breakdown alongside the uncached input_tokens, so usage stays self-consistent, prompt_tokens resolves to 14725 and the derived text_tokens to 12. Re-driving the same call with the fix reverted logged positive spend (0.00166463, 0.00165363) and zero negative rows, so _merge_message_start_cache_into_delta_usage is a no-op on this deployment. The bug needs a deployment whose message_delta omits the breakdown (GovCloud / LIT-2411), which a live commercial call cannot produce. Mark the path unsupported for live e2e rather than a gap nobody can close, keep the mock unit test as its guard, and write the evidence down so the probe is not repeated.
This commit is contained in:
parent
e2fb677f3c
commit
9fae5cfda7
2 changed files with 33 additions and 5 deletions
|
|
@ -30,7 +30,7 @@ proxy + SpendLogs rows. Status: `covered` / `partial` / `gap`.
|
|||
| Modality | Existing | Status | Live e2e |
|
||||
|----------|----------|--------|----------|
|
||||
| Non-negative spend (all providers, uncached traffic) | `test_anthropic_claude3_transformation.py`, `llm_cost_calc/utils.py` | partial | yes (`test_no_provider_logs_negative_spend`) |
|
||||
| Negative cost from bedrock cache-token split (#25846) | `test_anthropic_claude3_transformation.py` (mock) | gap (live) | no - needs bedrock streaming + cached prefix; the sweep above does not reach it |
|
||||
| Negative cost from bedrock cache-token split (#25846) | `test_anthropic_claude3_transformation.py` (mock) | unsupported (live) | no - not reproducible on commercial bedrock; see note below. Mock is the right guard |
|
||||
| Chat (non-stream) | `test_cost_calculator.py`, `local_testing/test_completion_cost.py` | covered | yes (`test_chat_completion_writes_nonzero_spend_row`) |
|
||||
| Chat (streaming) | `test_streaming_interrupt_spend_tracking.py` | partial | yes (`test_streaming_chat_completion_tracks_spend`) |
|
||||
| Embedding | `test_cost_calculator.py` (#29956) | partial | yes (`test_embedding_writes_nonzero_spend_row`) |
|
||||
|
|
@ -78,6 +78,29 @@ proxy + SpendLogs rows. Status: `covered` / `partial` / `gap`.
|
|||
| `test_spend_logs_v2_pagination_caps_pages_and_keeps_total` | `/spend/logs/v2` page cap, stable total on out-of-range page, zero total on no-match filter |
|
||||
| `test_spend_routes.py` (23) | no spend route 404s or 5xxs |
|
||||
|
||||
## Why the #25846 cache-token path is `unsupported` live, not a gap
|
||||
|
||||
Probed against real bedrock (`us.anthropic.claude-haiku-4-5`, commercial `us-east-1`) on
|
||||
2026-07-14 with a 14713-token cached prefix, streaming `/v1/messages`, cache write then
|
||||
cache read. Findings, so nobody re-runs this:
|
||||
|
||||
- `message_stop` carries **no** cache breakdown here, only `{input_tokens: 12, output_tokens: N}`.
|
||||
So the "cache fields land on message_stop" reading of the code comment does not hold on
|
||||
commercial bedrock.
|
||||
- The raw `message_delta` **already** carries the full breakdown
|
||||
(`input_tokens: 12, cache_read_input_tokens: 14713`), so it is self-consistent on its own.
|
||||
`prompt_tokens` becomes `12 + 14713 = 14725` and `text_tokens = 14725 - 14713 = 12`, never
|
||||
below zero.
|
||||
- Confirmed by reverting the fix (`patched_stream = completion_stream`) and re-driving the
|
||||
same cached call: spend stayed **positive** (0.00166463, 0.00165363; prompt_tokens 14725),
|
||||
zero negative rows. `_merge_message_start_cache_into_delta_usage` is a no-op on this
|
||||
deployment.
|
||||
|
||||
The bug needs a deployment whose `message_delta` omits the cache breakdown (GovCloud, per the
|
||||
transformation docstring / LIT-2411). A live commercial call cannot produce that payload, so
|
||||
this cell is `unsupported` for live e2e (excluded from the denominator) rather than a gap to
|
||||
chase, and the mock-based unit test is the correct guard. Revisit only if GovCloud is in scope.
|
||||
|
||||
## Design + timing
|
||||
|
||||
`proxy_batch_write_at` (~60s) means rows land late; every read polls to a deadline.
|
||||
|
|
|
|||
|
|
@ -472,10 +472,15 @@ def test_no_provider_logs_negative_spend(
|
|||
for the bedrock cache-token bug (#25846). That bug needs prompt_tokens to exclude
|
||||
cache tokens so `prompt_tokens - cache_hit` goes below zero; the traffic here is
|
||||
uncached, so `cache_hit == 0` and that subtraction can't go negative. Reverting
|
||||
#25846's clamp would leave this test green. Verified by mutation on 2026-07-14: the
|
||||
only way to turn it red was an injected fault unrelated to that bug, so the cell
|
||||
stays fail_before_fix=unproven. Proving it needs a test that drives the cache-token
|
||||
path itself (bedrock streaming with a cached prefix)."""
|
||||
#25846's clamp leaves this test green, so the cell is fail_before_fix=unproven.
|
||||
|
||||
Nor could that be fixed by adding a cached bedrock leg here: probing real bedrock
|
||||
(commercial us-east-1) on 2026-07-14 showed its raw message_delta already carries
|
||||
the cache breakdown, so usage stays self-consistent and spend stays positive even
|
||||
with the fix reverted. The bug needs a deployment whose message_delta omits it
|
||||
(GovCloud / LIT-2411), which a live commercial call cannot produce. That path is
|
||||
therefore unsupported live and guarded by the mock unit test instead; see
|
||||
SPEND_TRACKING_COVERAGE_MATRIX.md."""
|
||||
present = frozenset(entry.model_name for entry in client.gateway.model_info())
|
||||
chat = tuple(m for m in _CHAT_SWEEP if m.call in present)
|
||||
assert len(chat) >= 2, (
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue