mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-05 02:41:56 +00:00
06a97c83bd
test_text_message_blocked_by_guardrail_no_ai_response classified the model's reply against a safe_markers keyword list to decide whether the guardrail had blocked the message. gpt-realtime words its refusal of the guardrail's "say exactly" voice prompt nondeterministically, so any new phrasing outside the list turned CI red on unrelated PRs; the list had already been extended in #28191, #28200 and #29477, and drifted again to "Sorry, I can't comply with that request" (11 of the 13 failed realtime_translation_testing runs since 2026-06-24, e.g. CircleCI job 2009316 on #32380). Record every frame the proxy sends to the backend through a RecordingBackendWebSocket wrapper and assert the invariant the product actually guarantees: the blocked phrase never reaches OpenAI, only the guardrail's own conversation.item.create and response.create are forwarded (the client's reflexive response.create is dropped), and the blocked phrase never appears in AI output. Replace the fixed 0.3s/3.0s sleeps with an event-driven wait for response.done; client frames are processed sequentially so no inter-message sleep is needed. Verified by mutation: disabling the response.create drop fails the response.create count assertion, and disabling the guardrail fails the guardrail_violation assertion. |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| base_realtime_tests.py | ||
| test_openai_realtime.py | ||
| test_openai_realtime_simple.py | ||
| test_realtime_guardrails_openai.py | ||
| test_xai_realtime.py | ||