litellm/tests/e2e
Yuneng Jiang 30968fb721
test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows
These three pages carried no e2e coverage, so the manual QA checklist was the
only thing standing behind them.

Playground: sends a chat from the UI for both configured models, and for both
virtual-key sources (the logged-in session, and a key pasted into the panel).
This is the only spec that drives the dashboard's own LLM call path rather than
an admin CRUD endpoint.

Logs: a request the proxy actually served appears in the table, its drawer
expands to the real request and response bodies, both copy to the clipboard,
the Input card collapses, the JSON view exposes Request/Response, and the End
User filter narrows the table to one customer.

Usage: traffic billed to a virtual key reaches Top Virtual Keys, the card
toggles between table and chart, and the key opens its key-info panel.

Router settings: the existing spec proved the UI can record a fallback; the new
one proves the fallback is honoured, by pointing a model at an unreachable
upstream and asserting the reply comes back anyway. It asserts the un-fallen-back
call fails first, so a quietly-working primary cannot fake a pass.

Supporting changes:

- helpers/traffic.ts generates the traffic these pages render, rather than
  seeding rows no code produced. Its two wait helpers exist because the Logs
  and Usage pages read different stores: spend logs are flushed on a timer, and
  the Usage page reads a background rollup *and* fetches once on mount, so
  waiting on the DOM there can never converge.
- helpers/playground.ts holds the playground controls, now shared with the
  fallback spec. Everything is scoped to the visible copy of the config panel,
  which is rendered twice for the docked and collapsed layouts.
- run_e2e.sh gains E2E_KEEP_ALIVE=1, which brings the stack up and blocks so a
  spec can be re-run against it without paying for a UI rebuild each iteration.

Verified with the full suite on a fresh stack: 89 passed, 0 failed, 5 skipped.
2026-08-08 01:59:10 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control revert: "test(e2e): vendor API strategy coverage across endpoints (#34649)" 2026-08-04 19:00:34 -07:00
batches test(e2e): skip the batch rate-limiter spend-row test pending LIT-5027 (#35301) 2026-07-31 01:20:04 +00:00
claude_code refactor(e2e): fold claude_code HTTP probes onto shared Gateway methods (#33760) 2026-07-18 19:03:01 +00:00
coverage_registry revert: "test(e2e): vendor API strategy coverage across endpoints (#34649)" 2026-08-04 19:00:34 -07:00
guardrails test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
llm_translation test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
load test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820) 2026-08-04 14:12:25 -07:00
logging test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
management feat(proxy): add GET /management/v1/budgets (#35310) 2026-07-31 11:47:05 -07:00
mcp test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
other test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149) 2026-07-21 14:06:29 -07:00
quota_management Merge pull request #35875 from BerriAI/litellm_/inspiring-franklin-058a96 2026-08-04 19:14:48 -07:00
router test(e2e): add reliability suite covering fallback, timeout, and cache behavior (#34023) 2026-07-20 23:06:46 +00:00
ui test(e2e-ui): cover the Playground, Logs and Usage manual-QA flows 2026-08-08 01:59:10 -07:00
CLAUDE.md test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820) 2026-08-04 14:12:25 -07:00
conftest.py test(e2e): cover key max_budget blocks on personal, team, and team-member keys (#33895) 2026-07-22 17:48:18 -07:00
CONTRIBUTING.md fix(e2e): make spend-counter redis connection env-driven for non-cluster deployments (#35732) 2026-08-04 09:47:01 -07:00
e2e_config.py test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py revert: "test(e2e): vendor API strategy coverage across endpoints (#34649)" 2026-08-04 19:00:34 -07:00
junit_properties.py refactor(e2e): replace bespoke result reporter with standard JUnit report (#33758) 2026-07-17 20:53:22 +00:00
lifecycle.py test(e2e): cover key max_budget blocks on personal, team, and team-member keys (#33895) 2026-07-22 17:48:18 -07:00
models.py revert: "test(e2e): vendor API strategy coverage across endpoints (#34649)" 2026-08-04 19:00:34 -07:00
otel_client.py test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149) 2026-07-21 14:06:29 -07:00
proxy_client.py test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
pytest.ini test(e2e): add weekly session-anomaly load test against real providers 2026-07-21 14:54:02 -07:00
test_e2e_http.py test(e2e): retry provider-transient statuses at the transport with bounded backoff (#35824) 2026-08-04 14:57:42 -07:00
transport.py test(e2e): cover /v1/images/edits (#34476) 2026-07-25 10:38:07 -07:00