test(e2e): record that the #25846 cache-token path is unsupported live

Probed real bedrock (commercial us-east-1) with a 14713-token cached prefix over
streaming /v1/messages. Its raw message_delta already carries the cache breakdown
alongside the uncached input_tokens, so usage stays self-consistent, prompt_tokens
resolves to 14725 and the derived text_tokens to 12. Re-driving the same call with
the fix reverted logged positive spend (0.00166463, 0.00165363) and zero negative
rows, so _merge_message_start_cache_into_delta_usage is a no-op on this deployment.

The bug needs a deployment whose message_delta omits the breakdown (GovCloud /
LIT-2411), which a live commercial call cannot produce. Mark the path unsupported
for live e2e rather than a gap nobody can close, keep the mock unit test as its
guard, and write the evidence down so the probe is not repeated.
This commit is contained in:
mubashir1osmani 2026-07-14 17:55:15 -07:00
parent e2fb677f3c
commit 9fae5cfda7
2 changed files with 33 additions and 5 deletions

View file

@ -30,7 +30,7 @@ proxy + SpendLogs rows. Status: `covered` / `partial` / `gap`.
| Modality | Existing | Status | Live e2e |
|----------|----------|--------|----------|
| Non-negative spend (all providers, uncached traffic) | `test_anthropic_claude3_transformation.py`, `llm_cost_calc/utils.py` | partial | yes (`test_no_provider_logs_negative_spend`) |
| Negative cost from bedrock cache-token split (#25846) | `test_anthropic_claude3_transformation.py` (mock) | gap (live) | no - needs bedrock streaming + cached prefix; the sweep above does not reach it |
| Negative cost from bedrock cache-token split (#25846) | `test_anthropic_claude3_transformation.py` (mock) | unsupported (live) | no - not reproducible on commercial bedrock; see note below. Mock is the right guard |
| Chat (non-stream) | `test_cost_calculator.py`, `local_testing/test_completion_cost.py` | covered | yes (`test_chat_completion_writes_nonzero_spend_row`) |
| Chat (streaming) | `test_streaming_interrupt_spend_tracking.py` | partial | yes (`test_streaming_chat_completion_tracks_spend`) |
| Embedding | `test_cost_calculator.py` (#29956) | partial | yes (`test_embedding_writes_nonzero_spend_row`) |
@ -78,6 +78,29 @@ proxy + SpendLogs rows. Status: `covered` / `partial` / `gap`.
| `test_spend_logs_v2_pagination_caps_pages_and_keeps_total` | `/spend/logs/v2` page cap, stable total on out-of-range page, zero total on no-match filter |
| `test_spend_routes.py` (23) | no spend route 404s or 5xxs |
## Why the #25846 cache-token path is `unsupported` live, not a gap
Probed against real bedrock (`us.anthropic.claude-haiku-4-5`, commercial `us-east-1`) on
2026-07-14 with a 14713-token cached prefix, streaming `/v1/messages`, cache write then
cache read. Findings, so nobody re-runs this:
- `message_stop` carries **no** cache breakdown here, only `{input_tokens: 12, output_tokens: N}`.
So the "cache fields land on message_stop" reading of the code comment does not hold on
commercial bedrock.
- The raw `message_delta` **already** carries the full breakdown
(`input_tokens: 12, cache_read_input_tokens: 14713`), so it is self-consistent on its own.
`prompt_tokens` becomes `12 + 14713 = 14725` and `text_tokens = 14725 - 14713 = 12`, never
below zero.
- Confirmed by reverting the fix (`patched_stream = completion_stream`) and re-driving the
same cached call: spend stayed **positive** (0.00166463, 0.00165363; prompt_tokens 14725),
zero negative rows. `_merge_message_start_cache_into_delta_usage` is a no-op on this
deployment.
The bug needs a deployment whose `message_delta` omits the cache breakdown (GovCloud, per the
transformation docstring / LIT-2411). A live commercial call cannot produce that payload, so
this cell is `unsupported` for live e2e (excluded from the denominator) rather than a gap to
chase, and the mock-based unit test is the correct guard. Revisit only if GovCloud is in scope.
## Design + timing
`proxy_batch_write_at` (~60s) means rows land late; every read polls to a deadline.

View file

@ -472,10 +472,15 @@ def test_no_provider_logs_negative_spend(
for the bedrock cache-token bug (#25846). That bug needs prompt_tokens to exclude
cache tokens so `prompt_tokens - cache_hit` goes below zero; the traffic here is
uncached, so `cache_hit == 0` and that subtraction can't go negative. Reverting
#25846's clamp would leave this test green. Verified by mutation on 2026-07-14: the
only way to turn it red was an injected fault unrelated to that bug, so the cell
stays fail_before_fix=unproven. Proving it needs a test that drives the cache-token
path itself (bedrock streaming with a cached prefix)."""
#25846's clamp leaves this test green, so the cell is fail_before_fix=unproven.
Nor could that be fixed by adding a cached bedrock leg here: probing real bedrock
(commercial us-east-1) on 2026-07-14 showed its raw message_delta already carries
the cache breakdown, so usage stays self-consistent and spend stays positive even
with the fix reverted. The bug needs a deployment whose message_delta omits it
(GovCloud / LIT-2411), which a live commercial call cannot produce. That path is
therefore unsupported live and guarded by the mock unit test instead; see
SPEND_TRACKING_COVERAGE_MATRIX.md."""
present = frozenset(entry.model_name for entry in client.gateway.model_info())
chat = tuple(m for m in _CHAT_SWEEP if m.call in present)
assert len(chat) >= 2, (