Commit graph

6 commits

Author SHA1 Message Date
kerry
ffc16a4b0e test(e2e): restore the E2E_REDIS_CHAOS opt-in for the redis chaos test
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-11 22:46:31 +00:00
Kerry Lu
d4e083348c revert: drop the create-release.yml gating and E2E_REDIS_CHAOS opt-in
create-release.yml is back to calling the chaos test through no mechanism at
all; it never called it. Also drops the E2E_REDIS_CHAOS opt-in gate itself:
the redis_chaos marker still exists for -m selection and is still excluded
from the per-PR selector by path (tests/e2e/(ui|claude_code|load)/), but the
test no longer needs an env var to run once its file is targeted.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-11 15:16:00 -07:00
Kerry Lu
7d8f2c9ad3 ci(e2e): drop the weekly cron for the Redis chaos test
Now that create-release.yml gates stable and RC releases on this test directly,
the weekly schedule is redundant: every release gets a run against its own
commit instead of whatever happened to be on the default branch that Saturday.
workflow_dispatch stays for manual runs.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-11 15:00:57 -07:00
Kerry Lu
a8d0171800 ci: gate stable and RC releases on the Redis chaos load test
The chaos test only ran on a weekly cron, so a release could be cut from a
commit it had never covered. Making it callable lets create-release.yml run it
against the exact commit being tagged and refuse to tag if it fails.

Dev, nightly, alpha and beta tags skip the gate: they are cut far more often
than stable and RC tags, and the weekly schedule already covers the default
branch. Input validation moves into the gate job so a malformed tag or SHA
fails before spending a multi-minute chaos run.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-11 14:38:10 -07:00
Kerry Lu
66980bbb87 test(e2e): address Redis chaos PR review, add log-bytes budget
Pad the locust payload to tens of KB so per-request bookkeeping cost scales with
body size instead of hiding behind a 40-byte prompt. Turn on
use_redis_transaction_buffer in the chaos config and JSON_LOGS in the workflow so
the spend buffer, pod lock, and JSON-encoded breaker tracebacks are all part of
the measured chaos cost. Add a log-bytes-per-request budget alongside latency,
RSS, and CPU, reading the proxy's log file size at each phase split; its ceiling
is uncalibrated since no chaos run has measured it yet.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-11 12:40:25 -07:00
Kerry Lu
33ec56ed75 test(e2e): rewrite the Redis timeout test as a locust chaos load test
The sequential version sent one request at a time, so a Redis outage never
reached the concurrency where the failed-tracking alert body actually grows.
This drives the proxy with locust against one model group of three mock
deployments, two failing at order 1 and one serving at order 2, so every
request spends its retries on the failing pair and lands on the serving
deployment through the order-based fallback. Two phases, a healthy baseline
and a CLIENT PAUSE WRITE window, and every request must succeed in both.

Latency, RSS and CPU are reported as p50/p90/p99 per phase rather than
asserted on: RSS and CPU come from psutil on the proxy's process tree, since
a multi-worker proxy serves /metrics from the prometheus multiprocess
collector and that drops the process collector's series. Thresholds stay open
until weekly runs give real baselines.

Co-Authored-By: Claude Code <noreply@anthropic.com>
2026-09-10 21:55:23 -07:00
Renamed from .github/workflows/test-e2e-redis-timeout.yml (Browse further)