litellm/tests/e2e
Yuneng Jiang 171660ab96
test(e2e): assert the guardrail enforces before asserting a team bypasses it
test_team_with_disable_flag_bypasses_global_guardrail created a default-on
content filter and asserted that a key on a team with disable_global_guardrails
got a successful response. It never checked the guardrail was enforcing, so any
reason it was inert (keyword never matching, default_on not applied, a partial
create) made the call succeed for the wrong reason and the test pass. A
completely broken disable_global_guardrails stayed green.

Prove enforcement on a key outside the opted-out team first, then assert the
bypass, and check the response is the model's own answer rather than the
content-blocked message by requiring the model was actually invoked. That
mirrors what the block_code_execution suite already does, and it holds up on a
thinking model where a low max_tokens budget can leave the text empty.

Verified against a live proxy on real gemini-2.5-flash: both tests pass, and
with the guardrail registered under a keyword absent from the prompt the old
shape still passed while the new one fails on the enforcement check.
2026-08-18 08:51:30 -07:00
..
a2a test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
access_control test(e2e): assert the model allow-list permits, not only denies 2026-08-13 13:37:31 -07:00
batches test(e2e): skip the batch rate-limiter spend-row test pending LIT-5027 (#35301) 2026-07-31 01:20:04 +00:00
claude_code test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
coverage_registry test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
guardrails test(e2e): assert the guardrail enforces before asserting a team bypasses it 2026-08-18 08:51:30 -07:00
llm_translation test(e2e): skip the bedrock web search cell the stack cannot provision 2026-08-15 16:09:02 -07:00
load test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820) 2026-08-04 14:12:25 -07:00
logging test(e2e): assert provider error shape instead of pinned prose 2026-08-15 16:08:52 -07:00
management test(e2e): add reproducers for passthrough and model budget gaps (#34657) 2026-08-11 18:15:34 -07:00
mcp test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
other test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149) 2026-07-21 14:06:29 -07:00
quota_management test(e2e): add reproducers for passthrough and model budget gaps (#34657) 2026-08-11 18:15:34 -07:00
router test(e2e): add reliability suite covering fallback, timeout, and cache behavior (#34023) 2026-07-20 23:06:46 +00:00
ui test(e2e/ui): assert the log drawer chevrons by their lucide classes 2026-08-15 17:50:40 -07:00
CLAUDE.md test(e2e): move load/perf testing out of the main suite and drop the vllm passthrough test (#35820) 2026-08-04 14:12:25 -07:00
conftest.py test(e2e): cover key max_budget blocks on personal, team, and team-member keys (#33895) 2026-07-22 17:48:18 -07:00
CONTRIBUTING.md build(lint): rename make pre-commit to make check with a working-tree fallback 2026-08-08 03:25:35 -07:00
e2e_config.py test(e2e): settle control-plane writes across every replica, not just one 2026-08-07 19:36:30 -07:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py test(e2e): vendor API testing coverage (#34557) 2026-08-12 01:07:52 +00:00
junit_properties.py refactor(e2e): replace bespoke result reporter with standard JUnit report (#33758) 2026-07-17 20:53:22 +00:00
lifecycle.py test(e2e): cover key max_budget blocks on personal, team, and team-member keys (#33895) 2026-07-22 17:48:18 -07:00
models.py test(e2e): replay a real tool-search assistant turn back to Bedrock Invoke (#36856) 2026-08-17 11:59:26 -07:00
otel_client.py test(e2e): add Other suite and Guardrails coverage incl. an MCP tool-call guardrail (#34149) 2026-07-21 14:06:29 -07:00
proxy_client.py chore(ci): sync main into internal staging 2026-08-08 10:42:07 -07:00
pytest.ini test(e2e): add weekly session-anomaly load test against real providers 2026-07-21 14:54:02 -07:00
test_e2e_http.py test(e2e): retry provider-transient statuses at the transport with bounded backoff (#35824) 2026-08-04 14:57:42 -07:00
transport.py test(e2e): bound the post-/model/new servable wait at 40s 2026-07-28 17:47:19 -07:00