mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
53c9d48bd2
Adds a scheduled GitHub Actions lane on top of the merged record/replay transport. A Saturday cron records the `replayable` e2e tests against the real providers and publishes the fixture bundle as a private `e2e-fixtures-bundle` artifact with a SHA-256 sidecar. Weekday crons pull that artifact by its pinned digest, verify the checksum before extracting, and replay it with provider credentials set to bogus values, so a run that ever reached a real provider fails instead of passing. An egress sentinel pins the provider hostnames to a local sink for the whole replay job and counts every connection that reaches them; the job asserts that count is zero, so hermeticity is proven by measurement. A red Saturday publishes no bundle, so the next weekday finds nothing fresh and fails loudly rather than replaying a week-old recording, and the transport's seven-day freshness gate hard-fails any bundle that has drifted too far. The lane also runs on demand from the Actions tab with a record/replay `mode` input. Tests join the lane with `@pytest.mark.replayable`. The streaming Anthropic test now counts to twenty so its recorded response banks several content deltas, matching the assertion that the stream arrives incrementally.
11 lines
950 B
INI
11 lines
950 B
INI
[pytest]
|
|
# Config when any e2e suite under tests/e2e/ is run directly, e.g.
|
|
# uv run pytest tests/e2e/quota_management/spend_tracking/ -v
|
|
# The e2e marker is also registered in conftest.py for runs rooted elsewhere.
|
|
addopts = --strict-markers --strict-config --reruns 1 --only-rerun "kind='network'" --only-rerun "status_code=5[0-9][0-9]"
|
|
markers =
|
|
e2e: live test that requires a running proxy and real provider keys
|
|
replayable: edge-wired test whose provider traffic replays from a fixture bundle, so it makes zero provider calls in replay mode; the record/replay CI lane selects it with -m replayable
|
|
load: heavy throughput/load test; collected last so it never perturbs latency-sensitive suites
|
|
weekly: real-provider anomaly load test that spends real money; deselected unless E2E_WEEKLY_ANOMALY is set
|
|
managed_files: needs a proxy running with require_managed_files enabled; deselected unless E2E_MANAGED_FILES_STACK is set
|