The Anthropic and Together AI /v1/messages streaming tests required at
least two content_block_delta events. How many deltas a reply is split into
is the provider's choice, and Haiku answers a short count in one or two, so
the assertion failed on provider variance with no change in the proxy: four
of the day's full runs on the PR e2e gate went red on it on 2026-09-05.
The harness now stamps when each SSE event reached the client
(StreamingResponse.stream_event_arrivals, index-aligned with stream_events,
with the clock injectable so the reader has a unit test). Both tests ask for
a reply long enough to take seconds to generate and require the first
content delta to land at least STREAM_MIN_LEAD_SECONDS before message_stop.
A relayed stream shows a lead of about two seconds. A proxy that buffered
the response delivers every event in one burst and fails every time, which
a whole-response buffering relay in front of a live proxy confirmed. The
event-grammar assertions are unchanged.
Replay hands the proxy its recorded chunks back to back, so timing says
nothing there. The assertion is gated on provider_paces_stream() and replay
proves the grammar only, which tests/e2e/CLAUDE.md now says.
* test(e2e): retry provider-transient statuses at the transport with bounded backoff
The Anthropic passthrough cost test failed a full-suite run on a real 529
overloaded_error. Passthrough routes forward provider responses verbatim
and bypass the router's num_retries, so provider blips reach the harness
only on those paths. Following standard practice, the retry is scoped to
the dependency boundary instead of rerunning tests: only the enumerated
transient statuses (500/502/503/504/529, the set production SDKs retry by
default) are retried, with bounded exponential backoff and a printed line
per retry so flakiness stays visible in run logs.
429 is deliberately excluded: the quota suites assert the proxy's own
rate-limit and budget 429s, and a transport that absorbed them would break
those tests. Network errors and timeouts are not retried either, so a hang
surfaces as a hang. request_with_retry takes injected callables, and the
new harness tests pin the contract with protocol fakes, no monkeypatching
* test(e2e): narrow the transport retry to 529, the one status the proxy cannot emit
Greptile's review is right that status-only classification could absorb an
intermittently failing proxy: at the transport a 500/502/503/504 from the
proxy is indistinguishable from one it relayed, and the proxy is the system
under test. 529 is the only status litellm provably never originates
(Anthropic's overload signal, forwarded verbatim on passthrough) and the
only transient observed across the full-suite runs, so the set shrinks to
exactly that. The canary tests now also pin 500/502/503/504 as never
retried
