Commit graph

1 commit

Author SHA1 Message Date
Mateo Wang
53c9d48bd2
ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163)
Adds a scheduled GitHub Actions lane on top of the merged record/replay
transport. A Saturday cron records the `replayable` e2e tests against the
real providers and publishes the fixture bundle as a private
`e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull
that artifact by its pinned digest, verify the checksum before extracting,
and replay it with provider credentials set to bogus values, so a run that
ever reached a real provider fails instead of passing.

An egress sentinel pins the provider hostnames to a local sink for the whole
replay job and counts every connection that reaches them; the job asserts
that count is zero, so hermeticity is proven by measurement. A red Saturday
publishes no bundle, so the next weekday finds nothing fresh and fails loudly
rather than replaying a week-old recording, and the transport's seven-day
freshness gate hard-fails any bundle that has drifted too far. The lane also
runs on demand from the Actions tab with a record/replay `mode` input.

Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic
test now counts to twenty so its recorded response banks several content
deltas, matching the assertion that the stream arrives incrementally.
2026-08-24 23:49:03 -04:00