litellm/tests/e2e/quota_management/spend_tracking
Yuneng Jiang b95801172c
fix(e2e): only the negative fallback assertion needs every replica to agree
litellm-e2e-ui 68 failed the test this PR was meant to stabilise: "fallback
never took effect", streak 4 of a required 5, 60s timeout. Requiring a
consecutive streak of 200s after the fallback is set was wrong. It asserts that
the fallback path succeeds five times running, which is a reliability claim the
test never intended to make, and the path is inherently retry-ish because the
broken primary is attempted first on every call. One intermittent non-200
resets the streak, so a mostly-working fallback never converges.

The two directions are not symmetric:

  before the write  proving NO replica serves it   -> needs every replica
  after the write   proving the fallback serves it -> one success is the claim

So the control keeps a multi-sample window and the success assertion goes back
to polling for a first sighting, on the wider 60s budget rather than the
original 30s that expired on litellm-e2e-ui 63.

Also drops the two local rebinds Greptile flagged against the repo's
no-reassignment convention: the streak counter is gone with the helper it lived
in, and the cache-round loop is now a lazy generator consumed by next().
2026-08-26 18:57:38 -07:00
..
conftest.py test(e2e): spendlog cost for streaming /v1/messages via responses bridge (#33753) 2026-07-18 14:12:26 -07:00
cost_rows.py test(e2e): pin prompt-cache, service-tier, and cost-header billing 2026-08-20 01:50:01 -07:00
spend_e2e_client.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
SPEND_TRACKING_COVERAGE_MATRIX.md docs(tests/e2e): align docs with the hard-fail-on-dead-proxy contract and scope the no-unit-tests rule (#33755) 2026-07-17 12:56:10 -07:00
test_cache_cost_accounting_e2e.py test(e2e): request reasoning explicitly on the reasoning-cost assertions 2026-08-20 04:11:08 -07:00
test_cost_headers_e2e.py fix(e2e): only the negative fallback assertion needs every replica to agree 2026-08-26 18:57:38 -07:00
test_provider_edge_spend_e2e.py ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
test_service_tier_pricing_e2e.py test(e2e): request reasoning explicitly on the reasoning-cost assertions 2026-08-20 04:11:08 -07:00
test_spend_routes.py test(e2e): skip view-backed global spend probes pending LIT-5211 2026-08-04 18:40:13 -07:00
test_spend_tracking_e2e.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
test_team_daily_activity_e2e.py test(e2e): vendor API testing coverage (#34557) 2026-08-12 01:07:52 +00:00