Review follow-ups on the contract suite.
The restore case only checked the email, so a regression that left
<PHONE_NUMBER_2> in the answer passed. It now requires every masked value back
and rejects any numbered token by shape rather than by name.
Its poll returned as soon as an email placeholder appeared, which read as
accepting an unrestored response. The condition is now "the model echoed the
line in either form", and the docstring says why that is the right stopping
point: the restore runs per request, so a token that survived it is a verdict
and not something a retry fixes.
The unreachable case claimed to check that the failure does not name the
endpoint and never asserted it. It does now, and the unreachable base moved from
a loopback address to a host in the reserved .invalid domain, because the proxy
already rewrites IPs out of error messages: against 127.0.0.1 that assertion
could not fail even with the sanitizer removed. Verified by putting the raw
exception back into the analyzer error, which now turns the case red and
previously did not. A hostname is also what a real deployment looks like.
The registry entries used assertion names outside the documented guardrail
grammar. Rather than rename behaviors that are genuinely distinct, the grammar
now groups the names in use, including logs_masked_entities, which was already
in the registry and already outside it.
The exclusion pattern is back to two explicit filenames, so a future
self-contained presidio suite is not silently kept off the lane.
The presidio suites so far prove that masking happens. They do not prove that
the pieces we own still line up with what a real Presidio answers, and a mock
analyzer cannot show that: the mock resolves overlapping spans at analyze time,
returns recognition_metadata, and has no NER engine, so it agrees with our code
by construction.
Five cases against a live Presidio, each pinned to something LiteLLM does with
the response rather than to Presidio's own accuracy:
- the configured entity filter reaches the analyze request, so entity types a
customer did not configure survive untouched
- every detection is replaced in place by a placeholder naming its own type,
with the rest of the prompt byte for byte
- a PII entity configured BLOCK refuses the request, while clean text on the
same guardrail is still served
- output_parse_pii numbers the placeholders from the analyze spans and restores
the original values in the answer
- an analyzer the proxy cannot reach refuses the request rather than forwarding
raw PII to the model
None of them assert a confidence score, so Presidio adding a recognizer or
changing its scoring cannot turn them red. The UI spec's two score assertions
went the same way: it now pins that a score is rendered per entity, not which
numbers the analyzer chose.
The analyzer and anonymizer endpoints are an internal deployment, so they are
handled like a provider API key. presidio_env is the only place that reads
them, both suites route their failure messages through its scrubber so a
response body that echoes an endpoint cannot publish it into a CI log, and both
presidio files stay off the GitHub Actions lane, which is what keeps the secret
out of GHA entirely.
Also reframes the two echo prompts as transcription rather than "repeat this
back". Asked to repeat placeholders, a model may refuse and explain itself
instead, which left the merged pre_call assertion measuring the model's mood.
Mutation-tested against the live stack: dropping the entities field, ignoring
the anonymized text, never raising on a blocked entity, skipping the unmask,
and swallowing the analyzer connection error each turn their own case red, 5 of
5 killed.
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
A one-character value in the provider secret bundle was masked too, which
turned every 1 in the run log into ***, including the pass numbers and the
gateway addresses, so the only public diagnostics were unreadable
The changed-tests workflow overrode the suite's `--reruns 1` with `--reruns 0`, so a
transport blip failed a pass that pytest.ini already scopes to network errors and
5xx responses. Pass 2 of run 33692484803 also went red 15s after a model write with
"no healthy deployments": the barrier only polled /v1/models through nginx, which
proves one gateway converged, and the next request rolled the other. The stack now
exports LITELLM_PROXY_REPLICA_URLS, the barrier polls every replica with the full
budget before settling, and up.sh refuses to boot without DD_API_KEY, since the
gateway config enables the datadog callback on every run
Adds a required-check candidate that selects the tests/e2e test files a PR added or
modified, boots a stage-mirror stack on the runner (migrations, backend, two gateway
processes behind nginx, Postgres, Jaeger, TLS cluster Valkey), and runs those files
three times with retries off. The run job sits behind the e2e-changed GitHub
environment, so a reviewer approves each run before the OIDC token that reads the
provider keys from AWS Secrets Manager exists. Supersedes #34981
