litellm/tests/e2e
yuneng-jiang 0cf236bebb
test(e2e/ui): cover creating, testing and deleting a guardrail (#39053)
* test(e2e/ui): cover creating, testing and deleting a guardrail

The Guardrails page had no browser coverage. The RC checklist covers it by
hand against a live Presidio, which is why it has always been skipped in CI.

These drive the LiteLLM content filter instead, which runs inside the proxy,
so the whole flow is exercised without a third-party moderation service. The
create test does not stop at the table row: it sends a prompt carrying the
keyword it just banned and asserts the gateway refuses it, then sends a clean
prompt through the same guardrail and asserts it is served.

* test(e2e/ui): delete the guardrails these tests create

Review caught the fixtures being left behind. Guardrails are database rows
that show up in the table and in the playground's list, so a run that leaves
them changes what the next run sees.

Also trims the comments that restated what the helpers already say.

* test(e2e/ui): fail the run when guardrail teardown does not delete

Review caught the afterEach discarding the DELETE response, so a failed
cleanup finished quietly and left the guardrail for the next run to trip on.

* test(e2e/ui): wait for a new guardrail to reach the request path

The wizard test drove one chat completion immediately after creating the
guardrail and required a 400. A trace from the deployed stack shows the
record is stored correctly (blocked_words, action BLOCK, block_on_violation)
and the call six seconds later is still served unguarded, so the first
request can land before the proxy picks the guardrail up.

Polls the same call to the same 400 instead, which keeps the assertion and
lets the refresh land. If it never blocks, this stays red, which is what we
want it to say.

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-01 12:21:45 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control fix(auth): resolve bare model names against wildcard deployments in model access groups (#37492) 2026-08-19 15:33:29 -07:00
batches test(e2e): pin require_managed_files enforcement behind a marker-gated stack phase 2026-08-24 10:59:13 -07:00
claude_code test(e2e): retry upstream-saturation failures in the claude CLI driver 2026-08-29 17:13:10 -07:00
coverage_registry test(e2e): drop coverage registry cell for the alias-grant test 2026-09-01 09:03:47 -07:00
gateway ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
guardrails test(e2e): add logging e2e coverage (s3_v2, gcs_bucket, team langfuse callback, datadog failure) (#38552) 2026-08-29 09:43:44 -07:00
llm_translation test(e2e): cover Together reasoning_effort=none, json_schema, and cache-read pricing 2026-08-28 12:49:09 -07:00
load test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820) 2026-08-04 14:12:25 -07:00
logging test(e2e): add logging e2e coverage (s3_v2, gcs_bucket, team langfuse callback, datadog failure) (#38552) 2026-08-29 09:43:44 -07:00
management test(e2e): drop the two mgmt registry cells no shared-proxy test can cover 2026-08-31 19:33:04 -07:00
mcp test(e2e): drop coverage registry cell for the alias-grant test 2026-09-01 09:03:47 -07:00
other test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149) 2026-07-21 14:06:29 -07:00
quota_management test(e2e): cover model access group budgets against a live proxy 2026-08-29 15:46:49 -07:00
router test(e2e): require spent completion tokens before accepting empty fallback content 2026-08-29 16:19:41 -07:00
ui test(e2e/ui): cover creating, testing and deleting a guardrail (#39053) 2026-09-01 12:21:45 -07:00
CLAUDE.md test(e2e): cover model access group budgets against a live proxy 2026-08-29 15:46:49 -07:00
conftest.py ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
CONTRIBUTING.md ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
e2e_config.py test(e2e): pin require_managed_files enforcement behind a marker-gated stack phase 2026-08-24 10:59:13 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
fixture_canonical.py test(e2e): key multipart uploads by structured part identity 2026-08-21 19:32:02 -07:00
fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
junit_properties.py refactor(e2e): replace bespoke result reporter with standard JUnit report (#33758) 2026-07-17 20:53:22 +00:00
lifecycle.py test(e2e): pin openai_passthrough routing, cost logging, and file list isolation 2026-08-20 03:03:35 -07:00
models.py test(e2e): cover alias MCP grant persisting verbatim on key generate 2026-09-01 08:10:41 -07:00
otel_client.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00
provider_edge.py fix(e2e): record a streamed chunk only after its downstream write lands 2026-08-24 13:26:58 -07:00
proxy_client.py test(e2e): drop docstrings on the cost map model and its client accessor 2026-08-25 18:51:40 -07:00
pytest.ini ci(e2e): record the e2e suite weekly and replay it on weekdays with zero egress (#38163) 2026-08-24 23:49:03 -04:00
test_e2e_http.py test(e2e): retry provider-transient statuses at the transport with bounded backoff (#35824) 2026-08-04 14:57:42 -07:00
test_fixture_bundle.py feat(e2e): record and replay streamed provider responses chunk-for-chunk 2026-08-24 12:51:44 -07:00
test_fixture_canonical.py test(e2e): pin query params and multipart form fields as replay match-key identity 2026-08-20 15:59:34 -04:00
test_fixture_mode.py feat(e2e): move record/replay to the provider edge (LIT-5745) 2026-08-19 18:39:15 -07:00
test_provider_edge.py fix(e2e): record a streamed chunk only after its downstream write lands 2026-08-24 13:26:58 -07:00
transport.py test(e2e): harden the suite against response-cache cross-talk, slow providers and single upstream blips (#37957) 2026-08-22 14:47:03 -07:00