Adds live coverage for three guardrails litellm implements itself, each driven
against a real chat completion and asserting on the permitted/denied tool or the
judge's verdict rather than a bare status code.
tool_permission is registered allow-list style (one allow rule,
default_action=deny, on_disallowed_action=block): an unlisted tool is rejected
pre-call with the denied tool named, and the listed tool survives the guardrail
and is called by the model. tool_policy is exercised through a key-scoped
blocked-tool override set with POST /v1/tool/policy: the blocked tool is
rejected pre-call while a sibling tool on the same key and the same guardrail
still reaches the model, so the block is attributable to the policy rather than
to the guardrail refusing all tool use. llm_as_a_judge scores the model's answer
against a French-only criterion; an English answer is rejected with 422 and the
failing verdict, and a French one comes back to the caller.
The judge test does not stop at the response. That guardrail fails open on any
internal error, and a fail-open returns an ordinary 200 that is identical to an
approval: same status, same body, and the applied-guardrails header still names
it. So both halves also read the guardrail's own run log at
/guardrails/usage/logs, where an approval is recorded as `passed`, an
intervention as `blocked`, and a fail-open as `flagged`. Without that leg the
accept half would pass just as happily against a build where adjudication never
ran.
Also corrects the llm_as_a_judge registry row. It asked for a pre_call block,
which the guardrail cannot do: it supports post_call only and the proxy rejects
registering it at pre_call outright, so the row could never go green as written.
Retargeted to post_call, which is what the guardrail actually enforces.
The judge runs on openai/gpt-4.1 rather than the suite's usual gpt-5.5 because
the guardrail hardcodes temperature=0 on its judge call, gpt-5.5 accepts only
the default temperature, and the resulting error is swallowed into a fail-open,
so gpt-5.5 can never adjudicate anything.
Covers guardrail.tool_permission.pre_call.blocks,
guardrail.tool_permission.pre_call.allows, guardrail.tool_policy.pre_call.blocks
and guardrail.llm_as_a_judge.post_call.blocks.
The guardrail params, the tool-policy override bodies and the chat helper live
in a suite-local module so the shared harness is untouched.
2fd7929dc3