litellm/test-quality-budget.json at 39a580aa91e5cc4d2677100d44546d2a6309568c - scriptoriumadmin/litellm - Forgejo: Beyond coding. We forge.

scriptoriumadmin/litellm

mirror of https://github.com/BerriAI/litellm.git synced 2026-08-28 05:25:59 +00:00

yuneng-jiang 39a580aa91

test(guardrails): stop five guardrail test files leaking env vars on failure (#37828 )

Onyx, prompt security, hiddenlayer, repelloai and deepkeep all write straight to
os.environ and unset again at the bottom of each test. None of the five has a
try/finally, so the moment a test fails it returns to the runner with the keys
still set and whatever runs next in that worker inherits them.

Raising inside test_onyx_guard_with_custom_timeout_from_kwargs on the current
files leaves ONYX_API_BASE and ONYX_API_KEY behind; doing the same in
test_hiddenlayer_config_saas leaves HIDDENLAYER_API_BASE. Both come back clean
after this.

89 raw writes and the hand-rolled deletes become monkeypatch calls. The
class-level setup_method and teardown_method pair in the onyx file, sweeping the
same three keys twice, becomes one autouse fixture. The sys.path.insert lines
and their now-unused imports go too, and litellm.set_verbose = True, which only
turned global debug logging on for whatever ran next, is dropped rather than
restored.

test_onyx_guard_config and test_prompt_security_guard_config asserted nothing at
all, so they could only fail by raising. Each now pins what init_guardrails_v2
produces: exactly one guardrail of the right class on litellm.callbacks,
carrying the configured name, default_on and hook. The zero-assert tests in the
other three are left alone; those are a judgement about each guardrail rather
than a mechanical sweep.

tests/test_litellm/proxy/guardrails passes at 2873.

2026-08-21 21:29:31 -07:00

23 lines

249 B

JSON

Raw Blame History

 {
   "TQ001": {
     "limit": 744
   },
   "TQ002": {
     "limit": 742
   },
   "TQ003": {
     "limit": 1068
   },
   "TQ004": {
     "limit": 469
   },
   "TQ005": {
     "limit": 2549
   },
   "TQ006": {
     "limit": 34
   },
   "TQ007": {
     "limit": 117
   }
 }