test(guardrails): cover the LiteLLM/Presidio integration contract

The presidio suites so far prove that masking happens. They do not prove that
the pieces we own still line up with what a real Presidio answers, and a mock
analyzer cannot show that: the mock resolves overlapping spans at analyze time,
returns recognition_metadata, and has no NER engine, so it agrees with our code
by construction.

Five cases against a live Presidio, each pinned to something LiteLLM does with
the response rather than to Presidio's own accuracy:

- the configured entity filter reaches the analyze request, so entity types a
  customer did not configure survive untouched
- every detection is replaced in place by a placeholder naming its own type,
  with the rest of the prompt byte for byte
- a PII entity configured BLOCK refuses the request, while clean text on the
  same guardrail is still served
- output_parse_pii numbers the placeholders from the analyze spans and restores
  the original values in the answer
- an analyzer the proxy cannot reach refuses the request rather than forwarding
  raw PII to the model

None of them assert a confidence score, so Presidio adding a recognizer or
changing its scoring cannot turn them red. The UI spec's two score assertions
went the same way: it now pins that a score is rendered per entity, not which
numbers the analyzer chose.

The analyzer and anonymizer endpoints are an internal deployment, so they are
handled like a provider API key. presidio_env is the only place that reads
them, both suites route their failure messages through its scrubber so a
response body that echoes an endpoint cannot publish it into a CI log, and both
presidio files stay off the GitHub Actions lane, which is what keeps the secret
out of GHA entirely.

Also reframes the two echo prompts as transcription rather than "repeat this
back". Asked to repeat placeholders, a model may refuse and explain itself
instead, which left the merged pre_call assertion measuring the model's mood.

Mutation-tested against the live stack: dropping the entities field, ignoring
the anonymized text, never raising on a blocked entity, skipping the unmask,
and swallowing the analyzer connection error each turn their own case red, 5 of
5 killed.

.github/e2e-stack/select_tests.py

@@ -7,7 +7,7 @@ UNSUPPORTED: Final = re.compile(
     r"^tests/e2e/(ui|claude_code|load)/"
     r"|^tests/e2e/llm_translation/realtime/test_realtime_pipecat_audio_e2e\.py$"
     r"|^tests/e2e/batches/test_managed_files_enforcement_e2e\.py$"
-    r"|^tests/e2e/guardrails/test_presidio_masking_e2e\.py$"
+    r"|^tests/e2e/guardrails/test_presidio_.*_e2e\.py$"
 )
 HARNESS: Final = re.compile(
     r"^tests/e2e/[A-Za-z0-9_.-]+\.(py|ini)$"

tests/e2e/CONTRIBUTING.md

@@ -52,6 +52,8 @@ The suites run against a live proxy, so bring one up first by running the litell
 
 Some suites need extra services the bare proxy does not start. The `logging/` OTEL trace-completeness tests read spans back from a jaeger query API at `http://localhost:16686` (override with `E2E_OTEL_QUERY_URL`); run a `jaegertracing/all-in-one` and point `PHOENIX_COLLECTOR_HTTP_ENDPOINT` at its OTLP ingest. The `mcp/` suite needs the deterministic upstream MCP server in `mcp_tests/mcp_e2e_upstream_server.py` reachable by the proxy. The presidio guardrail tests need a running Presidio analyzer and anonymizer the proxy can reach, addressed by `PRESIDIO_ANALYZER_API_BASE` / `PRESIDIO_ANONYMIZER_API_BASE`
 
+Those two endpoints are treated as secrets, at the same handling as a provider API key: they name an internal deployment, so they are never checked in, never written into a config committed to this repo, and never printed. Supply them from the runner's environment only. `guardrails/presidio_env.py` is the single place that reads them, and every failure message in the presidio suites that interpolates a proxy response body passes it through that module's `scrub` first, so an endpoint cannot reach a CI log through a test failure. Two consequences worth knowing: run the proxy without `--detailed_debug` for these suites, because the guardrail logs the analyzer URL it is calling at debug level, and keep both presidio files on the `UNSUPPORTED` list in `.github/e2e-stack/select_tests.py`, which is what stops the GitHub Actions lane from needing the secret at all
+
 The Presidio spend-log audit test also requires prompt storage on the proxy, because its assertions inspect the detected entities and their scores. Add the following to the proxy config before starting it:
 
 ```yaml

tests/e2e/coverage_registry/guardrail.yaml

@@ -3,6 +3,11 @@
 - {id: guardrail.presidio.pre_call.masks, module: guardrail, tier: P0, hook_point: pre_call, assertions: [masks], exercised_on: [chat_completions, messages], source: "guardrail_hooks/presidio.py", rationale: "PII masking pre-call; data-leak blast radius"}
 - {id: guardrail.presidio.post_call.masks, module: guardrail, tier: P0, hook_point: post_call, assertions: [masks], exercised_on: [chat_completions, messages], source: "guardrail_hooks/presidio.py", rationale: "Mask PII in model output"}
 - {id: guardrail.presidio.logging_only.masks, module: guardrail, tier: P0, hook_point: logging_only, assertions: [masks], exercised_on: [chat_completions, messages], source: "guardrail_hooks/presidio.py", rationale: "Redact in logs without blocking"}
+- {id: guardrail.presidio.pre_call.masks_only_configured_entities, module: guardrail, tier: P0, hook_point: pre_call, assertions: [masks_only_configured_entities], exercised_on: [chat_completions], source: "guardrail_hooks/presidio.py", rationale: "pii_entities_config is put on the analyze request as the entities filter. If that stops reaching Presidio the guardrail masks every type Presidio recognizes, silently redacting text a customer configured it to leave alone"}
+- {id: guardrail.presidio.pre_call.masks_each_entity_in_place, module: guardrail, tier: P0, hook_point: pre_call, assertions: [masks_each_entity_in_place], exercised_on: [chat_completions], source: "guardrail_hooks/presidio.py", rationale: "Every detection Presidio returns must be replaced where it sits by a placeholder naming its own type. Spans applied at the wrong offsets still remove the PII while corrupting the surrounding prompt"}
+- {id: guardrail.presidio.pre_call.blocks, module: guardrail, tier: P0, hook_point: pre_call, assertions: [blocks], exercised_on: [chat_completions], source: "guardrail_hooks/presidio.py", rationale: "A PII entity configured BLOCK must refuse the request. The decision reads entity_type off the analyze response, so it is the one path that proves we parse the analysis and not just the anonymized text"}
+- {id: guardrail.presidio.pre_call.restores_masked_values, module: guardrail, tier: P0, hook_point: pre_call, assertions: [restores_masked_values], exercised_on: [chat_completions], source: "guardrail_hooks/presidio.py", rationale: "output_parse_pii numbers each placeholder from the analyze spans and restores the original value in the answer. A broken mapping either leaks the wrong value back or leaves a raw token in the customer response"}
+- {id: guardrail.presidio.pre_call.fails_closed_when_unreachable, module: guardrail, tier: P0, hook_point: pre_call, assertions: [fails_closed_when_unreachable], exercised_on: [chat_completions], source: "guardrail_hooks/presidio.py", rationale: "A Presidio the proxy cannot reach must refuse the request. Degrading to an unmasked call would send raw PII to the provider during an outage, which is the exact failure the guardrail exists to prevent"}
 - {id: guardrail.presidio.pre_call.logs_masked_entities, module: guardrail, tier: P0, hook_point: pre_call, assertions: [logs_masked_entities], exercised_on: [chat_completions], source: "guardrail_hooks/presidio.py", rationale: "A masking run must record itself on the spend log: the dashboard's guardrail panel renders the masked-entity counts and per-entity scores straight off metadata.guardrail_information, so a run that masks but records nothing leaves an operator unable to audit it"}
 - {id: guardrail.bedrock.pre_call.blocks, module: guardrail, tier: P0, hook_point: pre_call, assertions: [blocks], exercised_on: [chat_completions], source: "guardrail_hooks/bedrock_guardrails.py", rationale: "AWS content guardrail blocks harmful input"}
 - {id: guardrail.litellm_content_filter.pre_call.blocks, module: guardrail, tier: P0, hook_point: pre_call, assertions: [blocks], exercised_on: [chat_completions], source: "test_team_disable_global_guardrail_e2e.py", rationale: "Local content-filter default-on blocks banned keyword pre-call"}

tests/e2e/guardrails/presidio_env.py

@@ -0,0 +1,56 @@
+"""Where the Presidio analyzer/anonymizer endpoints come from, and how they stay
+out of test output.
+
+The endpoints are an internal deployment, so they are handled at the same
+secrecy as a provider API key: never checked in, supplied only through
+PRESIDIO_ANALYZER_API_BASE / PRESIDIO_ANONYMIZER_API_BASE in the runner's
+environment, and hard-failed when absent rather than skipped.
+
+`scrub` exists because a failing e2e test prints proxy response bodies into CI
+logs, which are read by more people than the endpoints are meant for. Every
+failure message in the presidio suites that interpolates a body goes through it,
+so a body that ever echoes an endpoint back cannot publish it.
+"""
+
+from __future__ import annotations
+
+import os
+from functools import cache, reduce
+from urllib.parse import urlparse
+
+import pytest
+
+ANALYZER_ENV_VAR = "PRESIDIO_ANALYZER_API_BASE"
+ANONYMIZER_ENV_VAR = "PRESIDIO_ANONYMIZER_API_BASE"
+
+REDACTED = "<presidio-endpoint>"
+
+
+@cache
+def presidio_bases() -> tuple[str, str]:
+    analyzer = os.environ.get(ANALYZER_ENV_VAR, "").strip()
+    anonymizer = os.environ.get(ANONYMIZER_ENV_VAR, "").strip()
+    if not analyzer or not anonymizer:
+        pytest.fail(
+            f"Presidio e2e requires {ANALYZER_ENV_VAR} and {ANONYMIZER_ENV_VAR} "
+            "(the running Presidio analyzer/anonymizer services); missing env is a hard failure, not a skip"
+        )
+    return analyzer, anonymizer
+
+
+def secret_fragments(urls: tuple[str, ...]) -> tuple[str, ...]:
+    """Every substring that would identify the deployment: each URL as given,
+    without its trailing slash, and its bare host, so a connection error naming
+    only the host is caught as well as a message quoting the whole URL. Longest
+    first, so a full URL is replaced whole instead of leaving its path behind."""
+    hosts = tuple(netloc for netloc in (urlparse(url).netloc for url in urls) if netloc)
+    return tuple(sorted({*urls, *(url.rstrip("/") for url in urls), *hosts}, key=len, reverse=True))
+
+
+def redact(text: str, fragments: tuple[str, ...]) -> str:
+    return reduce(lambda carried, fragment: carried.replace(fragment, REDACTED), fragments, text)
+
+
+def scrub(text: str) -> str:
+    """Replace any trace of the configured Presidio deployment in text bound for a log."""
+    return redact(text, secret_fragments(presidio_bases()))

tests/e2e/guardrails/test_presidio_contract_e2e.py

@@ -0,0 +1,359 @@
+"""Live e2e: the contract between LiteLLM and a real Presidio deployment.
+
+Whether Presidio finds an email address is Presidio's business, and this suite
+does not test it. What it tests is our half of the exchange: that the entity
+filter we were configured with is actually put on the /analyze request, that the
+spans and entity types Presidio answers with are turned into the right
+placeholders, blocks, and restored values, and that a Presidio we cannot reach
+stops the request instead of forwarding raw PII to the model.
+
+Every case is written to survive Presidio changing its mind about confidence
+scores or adding a recognizer: the assertions turn on which entity types were
+asked for and on what our code did with the answer, never on a score.
+
+Most cases drive POST /guardrails/apply_guardrail, which runs the guardrail over
+literal text and hands back what the model would have received. That puts the
+assertion on the masked text itself rather than on a model's willingness to echo
+it, and costs no provider spend. The two cases that must prove an effect on a
+real call, blocking and restoring the masked value on the way out, go through
+/chat/completions.
+
+The analyzer and anonymizer endpoints come from the environment and are treated
+as secrets; see presidio_env.
+"""
+
+from __future__ import annotations
+
+import time
+from collections.abc import Callable
+from typing import Final, Literal
+
+import pytest
+
+from e2e_config import MASTER_KEY, POLL_INTERVAL, POLL_TIMEOUT, unique_marker
+from e2e_http import Result, Success, UnknownApiError
+from guardrails_client import (
+    ApplyGuardrailResponse,
+    GuardrailMode,
+    GuardrailsClient,
+    PiiAction,
+    PiiEntity,
+    PresidioParamsBody,
+)
+from lifecycle import ResourceManager
+from models import ChatResponse
+from presidio_env import presidio_bases, scrub
+
+pytestmark = pytest.mark.e2e
+
+MODEL = "gemini-2.5-flash"
+
+RAW_EMAIL = "dana.reyes@example.com"
+RAW_PHONE = "+1 415-555-0134"
+RAW_CARD = "4111 1111 1111 1111"
+PII_SENTENCE = f"Reach Dana at {RAW_EMAIL} or on {RAW_PHONE} about card {RAW_CARD} today."
+
+UNREACHABLE_BASE = "http://127.0.0.1:9/"
+
+MAX_ECHO_TOKENS = 128
+
+
+def _register(
+    client: GuardrailsClient,
+    resources: ResourceManager,
+    *,
+    name: str,
+    entities: dict[PiiEntity, PiiAction],
+    mode: GuardrailMode = "pre_call",
+    filter_scope: Literal["input", "output", "both"] = "input",
+    output_parse_pii: bool = False,
+    analyzer_base: str | None = None,
+    anonymizer_base: str | None = None,
+) -> None:
+    configured_analyzer, configured_anonymizer = presidio_bases()
+    guardrail_id: Final = client.register(
+        name,
+        PresidioParamsBody(
+            mode=mode,
+            default_on=False,
+            presidio_analyzer_api_base=analyzer_base or configured_analyzer,
+            presidio_anonymizer_api_base=anonymizer_base or configured_anonymizer,
+            presidio_filter_scope=filter_scope,
+            output_parse_pii=output_parse_pii,
+            pii_entities_config=entities,
+        ),
+    )
+    resources.defer(lambda: client.delete_guardrail(guardrail_id))
+
+
+def _apply(client: GuardrailsClient, name: str, text: str) -> str:
+    """Run the guardrail over literal text and return what the model would see.
+
+    /guardrails/apply_guardrail is a management route, so it is called with the
+    master key like the rest of the apply surface. A replica that has not
+    reloaded its config yet answers 404, which is in-flight propagation and
+    worth retrying; every other refusal is a verdict and fails immediately
+    rather than being waited out to the deadline.
+    """
+    deadline: Final = time.monotonic() + POLL_TIMEOUT
+    last: Result[ApplyGuardrailResponse] = client.apply_guardrail(MASTER_KEY, name=name, text=text)
+    while not isinstance(last, Success):
+        if not isinstance(last, UnknownApiError) or last.status_code != 404:
+            pytest.fail(f"apply_guardrail refused to run {name!r}: {scrub(str(last))}")
+        if time.monotonic() >= deadline:
+            pytest.fail(f"apply_guardrail never ran {name!r} within {POLL_TIMEOUT}s: {scrub(str(last))}")
+        time.sleep(POLL_INTERVAL)
+        last = client.apply_guardrail(MASTER_KEY, name=name, text=text)
+    return last.data.response_text
+
+
+def _poll_until_refused(call: Callable[[], Result[ChatResponse]]) -> Result[ChatResponse]:
+    """Retry a call a guardrail should refuse until it does, returning the last
+    result. A call served right after the create ran on a worker that has not
+    picked the guardrail up yet, which is propagation, not a guardrail that
+    failed to refuse; one that is still served at the deadline is the failure."""
+    deadline: Final = time.monotonic() + POLL_TIMEOUT
+    last: Result[ChatResponse] = call()
+    while isinstance(last, Success) and time.monotonic() < deadline:
+        time.sleep(POLL_INTERVAL)
+        last = call()
+    return last
+
+
+def _first_content(response: ChatResponse) -> str:
+    if not response.choices:
+        return ""
+    message: Final = response.choices[0].message
+    return (message.content if message else None) or ""
+
+
+class TestPresidioEntityFilterContract:
+    @pytest.mark.covers(
+        "guardrail.presidio.pre_call.masks_only_configured_entities",
+        exercised_on=["chat_completions"],
+    )
+    def test_only_the_configured_entity_types_are_masked(
+        self, client: GuardrailsClient, resources: ResourceManager
+    ) -> None:
+        """The configured entity filter must reach Presidio, not stay in our own
+        bookkeeping.
+
+        A guardrail configured for EMAIL_ADDRESS alone puts `entities` on the
+        analyze request, so the phone number and card in the same sentence come
+        back untouched. Stop sending that field and Presidio answers with
+        everything it recognizes, which masks all three and fails here.
+        """
+        name: Final = f"e2e-presidio-filter-{unique_marker()}"
+        _register(client, resources, name=name, entities={"EMAIL_ADDRESS": "MASK"})
+
+        masked: Final = _apply(client, name, PII_SENTENCE)
+
+        assert "<EMAIL_ADDRESS>" in masked, f"the configured entity must be masked, got {scrub(masked)!r}"
+        assert RAW_EMAIL not in masked, f"the raw address must not survive masking, got {scrub(masked)!r}"
+        assert RAW_PHONE in masked, (
+            "PHONE_NUMBER was not configured, so it must reach the model untouched; the guardrail "
+            f"masked more than it was asked to: {scrub(masked)!r}"
+        )
+        assert RAW_CARD in masked, (
+            "CREDIT_CARD was not configured, so it must reach the model untouched; the guardrail "
+            f"masked more than it was asked to: {scrub(masked)!r}"
+        )
+
+    @pytest.mark.covers(
+        "guardrail.presidio.pre_call.masks_each_entity_in_place",
+        exercised_on=["chat_completions"],
+    )
+    def test_every_detection_is_replaced_in_place_by_its_own_type_placeholder(
+        self, client: GuardrailsClient, resources: ResourceManager
+    ) -> None:
+        """Presidio answers with one span per detection and we replace each one
+        where it sits.
+
+        Pinning the whole sentence is what makes this more than "something was
+        redacted": spans applied at the wrong offsets would still remove the PII
+        while eating the words around it, and a placeholder built from the wrong
+        field would name the wrong type.
+        """
+        name: Final = f"e2e-presidio-placeholders-{unique_marker()}"
+        _register(
+            client,
+            resources,
+            name=name,
+            entities={"EMAIL_ADDRESS": "MASK", "PHONE_NUMBER": "MASK", "CREDIT_CARD": "MASK"},
+        )
+
+        masked: Final = _apply(client, name, PII_SENTENCE)
+
+        assert masked == "Reach Dana at <EMAIL_ADDRESS> or on <PHONE_NUMBER> about card <CREDIT_CARD> today.", (
+            "each detection must be replaced by a placeholder naming its own entity type, in place, "
+            f"leaving the rest of the sentence byte for byte; got {scrub(masked)!r}"
+        )
+
+
+class TestPresidioBlockContract:
+    @pytest.mark.covers(
+        "guardrail.presidio.pre_call.blocks",
+        exercised_on=["chat_completions"],
+    )
+    def test_a_block_entity_refuses_the_request_while_clean_text_still_passes(
+        self, client: GuardrailsClient, resources: ResourceManager, scoped_key: str
+    ) -> None:
+        """BLOCK is decided by matching Presidio's entity_type against our config,
+        so it proves we read the analyze response and not only its masked text.
+
+        The clean prompt is the liveness half. Without it a guardrail that
+        refused every request, or an analyzer erroring on everything, would look
+        exactly like one that blocks the right thing.
+        """
+        name: Final = f"e2e-presidio-block-{unique_marker()}"
+        _register(client, resources, name=name, entities={"CREDIT_CARD": "BLOCK"})
+
+        marker: Final = unique_marker()
+        blocked: Final = _poll_until_refused(
+            lambda: client.chat(scoped_key, MODEL, f"{marker} Charge card {RAW_CARD}.", guardrails=[name])
+        )
+        assert not isinstance(blocked, Success), (
+            f"a request carrying a CREDIT_CARD must be refused, but it was served: {scrub(str(blocked))}"
+        )
+        assert "CREDIT_CARD" in scrub(str(blocked)), (
+            "the refusal must name the entity that caused it, so an operator can tell it from an "
+            f"unrelated 4xx; got {scrub(str(blocked))}"
+        )
+
+        allowed: Final = client.chat(
+            scoped_key,
+            MODEL,
+            f"{marker} Say the word hello and nothing else.",
+            guardrails=[name],
+            max_tokens=32,
+        )
+        assert isinstance(allowed, Success), (
+            "text carrying no blocked entity must still be served by the same guardrail; it refused "
+            f"everything, so the block above proves nothing: {scrub(str(allowed))}"
+        )
+
+
+class TestPresidioOutputParseContract:
+    @pytest.mark.covers(
+        "guardrail.presidio.pre_call.restores_masked_values",
+        exercised_on=["chat_completions"],
+    )
+    def test_numbered_tokens_mask_the_prompt_and_are_restored_in_the_answer(
+        self, client: GuardrailsClient, resources: ResourceManager, scoped_key: str
+    ) -> None:
+        """With output_parse_pii each placeholder is numbered per detection and
+        the caller gets the original values back.
+
+        Both halves are ours and both are built from Presidio's spans: the
+        numbering comes from sorting the analyze results left to right, and the
+        restore puts back the text those offsets pointed at. The apply_guardrail
+        call pins what the model receives, the chat call pins what the caller
+        receives, and the caller only sees the real address if the
+        token-to-value mapping survived the round trip.
+        """
+        name: Final = f"e2e-presidio-parse-{unique_marker()}"
+        _register(
+            client,
+            resources,
+            name=name,
+            entities={"EMAIL_ADDRESS": "MASK", "PHONE_NUMBER": "MASK"},
+            output_parse_pii=True,
+        )
+
+        to_the_model: Final = _apply(client, name, f"Dana is {RAW_EMAIL} on {RAW_PHONE}")
+        assert to_the_model == "Dana is <EMAIL_ADDRESS_1> on <PHONE_NUMBER_2>", (
+            "output_parse_pii numbers each placeholder in the order its entity appears, which is what "
+            f"makes the restore reversible; got {scrub(to_the_model)!r}"
+        )
+
+        restored: Final = _poll_until_restored(client, scoped_key, name, _echo_prompt(unique_marker()))
+        assert RAW_EMAIL in restored, (
+            "the caller must get the real address back, not the placeholder the model saw; a token left "
+            f"in the answer means the mapping was lost between the two hooks: {scrub(restored)!r}"
+        )
+        assert "EMAIL_ADDRESS_1" not in restored, (
+            f"no numbered token may survive into the caller's response: {scrub(restored)!r}"
+        )
+
+
+def _echo_prompt(marker: str) -> str:
+    """A transcription framing rather than "repeat this back".
+
+    The model is handed the numbered placeholders. Asked to repeat them, a model
+    may instead explain that it will not echo someone's contact details, which
+    leaves the restore nothing to act on and turns a product assertion into a
+    test of the model's mood. Framed as transcription, the line comes back
+    verbatim.
+    """
+    return (
+        f"{marker} You are a text transcription tool. Output the input text character for character, "
+        f"with no commentary and no explanation. Input: Dana is {RAW_EMAIL} on {RAW_PHONE}"
+    )
+
+
+def _poll_until_restored(client: GuardrailsClient, key: str, name: str, prompt: str) -> str:
+    """Retry until the guardrail has attached and the model has echoed the line.
+
+    The applied-guardrails header is the liveness gate: without it a response
+    carrying the raw address would be indistinguishable from one the guardrail
+    never touched, and the restore assertion would pass vacuously.
+    """
+    deadline: Final = time.monotonic() + POLL_TIMEOUT
+    last = "<no successful response yet>"  # rebind-ok: last-observation accumulator for the failure message
+    while True:
+        outcome = client.chat_raw(key, MODEL, prompt, guardrails=[name], max_tokens=MAX_ECHO_TOKENS)
+        if outcome.ok and name in outcome.headers.get("x-litellm-applied-guardrails", ""):
+            last = _first_content(ChatResponse.model_validate_json(outcome.body))
+            if RAW_EMAIL in last or "EMAIL_ADDRESS" in last:
+                return last
+        else:
+            last = f"<guardrail not applied: HTTP {outcome.status_code}>"
+        if time.monotonic() >= deadline:
+            pytest.fail(
+                f"the guarded call never came back with the model's echo within {POLL_TIMEOUT}s; "
+                f"last observation: {scrub(last)[:300]!r}"
+            )
+        time.sleep(POLL_INTERVAL)
+
+
+class TestPresidioUnreachableContract:
+    @pytest.mark.covers(
+        "guardrail.presidio.pre_call.fails_closed_when_unreachable",
+        exercised_on=["chat_completions"],
+    )
+    def test_an_unreachable_analyzer_refuses_the_request_instead_of_forwarding_raw_pii(
+        self, client: GuardrailsClient, resources: ResourceManager, scoped_key: str
+    ) -> None:
+        """A Presidio outage must not degrade into "no masking today".
+
+        The guardrail is pointed at a closed local port so the analyzer call
+        cannot connect. With PII entities configured the guardrail fails closed
+        and the prompt never reaches the model with the address still in it. The
+        error goes straight back to the caller, so it is also asserted to carry
+        neither the raw address nor the endpoint it failed to reach.
+        """
+        name: Final = f"e2e-presidio-unreachable-{unique_marker()}"
+        _register(
+            client,
+            resources,
+            name=name,
+            entities={"EMAIL_ADDRESS": "MASK"},
+            analyzer_base=UNREACHABLE_BASE,
+            anonymizer_base=UNREACHABLE_BASE,
+        )
+
+        marker: Final = unique_marker()
+        outcome: Final = _poll_until_refused(
+            lambda: client.chat(scoped_key, MODEL, f"{marker} Repeat: {RAW_EMAIL}", guardrails=[name])
+        )
+
+        assert not isinstance(outcome, Success), (
+            "with the analyzer unreachable the guardrail cannot know whether the prompt holds PII, so "
+            f"the request must be refused rather than forwarded unmasked: {scrub(str(outcome))}"
+        )
+        body: Final = scrub(str(outcome))
+        assert RAW_EMAIL not in body, f"the failure must not echo the prompt's PII back to the caller: {body}"
+        assert "Presidio" in body, (
+            f"the refusal must say the guardrail could not run, or an operator cannot tell an outage "
+            f"from a model error: {body}"
+        )

tests/e2e/guardrails/test_presidio_env.py

@@ -0,0 +1,60 @@
+"""Harness coverage for the Presidio endpoint scrubber.
+
+`scrub` is what keeps an internal endpoint out of a CI log when a presidio test
+fails, and nothing downstream would notice if it quietly stopped replacing
+anything, so its two pure halves are covered directly. No proxy, no e2e marker.
+"""
+
+from __future__ import annotations
+
+from presidio_env import REDACTED, redact, secret_fragments
+
+ANALYZER = "https://analyzer.internal.example/"
+ANONYMIZER = "https://anonymizer.internal.example/"
+BASES = (ANALYZER, ANONYMIZER)
+
+
+class TestSecretFragments:
+    def test_each_endpoint_contributes_its_url_slashless_form_and_bare_host(self) -> None:
+        fragments = secret_fragments(BASES)
+
+        assert set(fragments) == {
+            ANALYZER,
+            ANALYZER.rstrip("/"),
+            "analyzer.internal.example",
+            ANONYMIZER,
+            ANONYMIZER.rstrip("/"),
+            "anonymizer.internal.example",
+        }
+
+    def test_fragments_are_longest_first(self) -> None:
+        """A host is a substring of its own URL, so replacing the short form first
+        would leave the scheme and path of the long form behind in the log."""
+        lengths = [len(fragment) for fragment in secret_fragments(BASES)]
+
+        assert lengths == sorted(lengths, reverse=True)
+
+
+class TestRedact:
+    def test_a_full_url_is_replaced_whole(self) -> None:
+        redacted = redact(f"Making request to: {ANALYZER}analyze", secret_fragments(BASES))
+
+        assert redacted == f"Making request to: {REDACTED}analyze"
+        assert "internal.example" not in redacted
+
+    def test_a_bare_host_is_replaced_too(self) -> None:
+        """aiohttp names only the host when it cannot connect, so a connection
+        error is the shape most likely to carry the endpoint into a log."""
+        redacted = redact("Cannot connect to host anonymizer.internal.example:443", secret_fragments(BASES))
+
+        assert redacted == f"Cannot connect to host {REDACTED}:443"
+
+    def test_every_occurrence_goes_not_just_the_first(self) -> None:
+        redacted = redact(f"{ANALYZER} then {ANALYZER}", secret_fragments(BASES))
+
+        assert redacted == f"{REDACTED} then {REDACTED}"
+
+    def test_text_carrying_no_endpoint_is_returned_unchanged(self) -> None:
+        assert redact("Presidio PII analysis failed: ClientConnectorError", secret_fragments(BASES)) == (
+            "Presidio PII analysis failed: ClientConnectorError"
+        )

tests/e2e/guardrails/test_presidio_masking_e2e.py

@@ -14,8 +14,8 @@ configuration: the post_call prompt asks for a value derived from the raw email
 address itself, so the answer proves the model saw the raw address while the
 address in the same response comes back as <EMAIL_ADDRESS>.
 
-The analyzer/anonymizer endpoints come from PRESIDIO_ANALYZER_API_BASE /
-PRESIDIO_ANONYMIZER_API_BASE; missing env is a hard failure, never a skip.
+The analyzer/anonymizer endpoints come from the environment and are treated as
+secrets; see presidio_env. Missing env is a hard failure, never a skip.
 Each guardrail registers with an explicit presidio_filter_scope so only the
 configured hook's callback exists (the default "both" registers input masking
 AND a post_call output masker), and is deleted on teardown.
@@ -29,7 +29,6 @@ this suite deliberately requires the detected-entity details to remain visible.
 
 from __future__ import annotations
 
-import os
 import time
 from collections.abc import Callable
 from typing import Final, Literal
@@ -48,6 +47,7 @@ from models import (
     GuardrailRunRecord,
     SpendLogRow,
 )
+from presidio_env import presidio_bases, scrub
 
 pytestmark = pytest.mark.e2e
 
@@ -70,17 +70,6 @@ MASKED_PHONE_TOKEN = "PHONE_NUMBER"
 FAKE_PHONE = "+1 415-555-0134"
 
 
-def _presidio_bases() -> tuple[str, str]:
-    analyzer = os.environ.get("PRESIDIO_ANALYZER_API_BASE", "").strip()
-    anonymizer = os.environ.get("PRESIDIO_ANONYMIZER_API_BASE", "").strip()
-    if not analyzer or not anonymizer:
-        pytest.fail(
-            "Presidio e2e requires PRESIDIO_ANALYZER_API_BASE and PRESIDIO_ANONYMIZER_API_BASE "
-            "(the running Presidio analyzer/anonymizer services); missing env is a hard failure, not a skip"
-        )
-    return analyzer, anonymizer
-
-
 def _register_presidio(
     client: GuardrailsClient,
     resources: ResourceManager,
@@ -90,7 +79,7 @@ def _register_presidio(
     filter_scope: Literal["input", "output", "both"] = "input",
     entities: dict[PiiEntity, PiiAction] | None = None,
 ) -> None:
-    analyzer, anonymizer = _presidio_bases()
+    analyzer, anonymizer = presidio_bases()
     guardrail_id = client.register(
         name,
         PresidioParamsBody(
@@ -110,9 +99,17 @@ def _fake_email() -> str:
 
 
 def _pii_prompt(marker: str, email: str) -> str:
+    """A transcription framing rather than "repeat this back".
+
+    The model is handed the masked placeholders. Asked to repeat those, a model
+    may instead explain that it will not echo someone's contact details, which
+    leaves the assertion testing the model's mood rather than the guardrail.
+    Framed as transcription, the line comes back verbatim.
+    """
     return (
-        f"{marker} Repeat this sentence back to me exactly, word for word: "
-        f"My email address is {email} and my phone number is {FAKE_PHONE}."
+        f"{marker} You are a text transcription tool. Output the input text character for character, "
+        f"with no commentary and no explanation. "
+        f"Input: My email address is {email} and my phone number is {FAKE_PHONE}."
     )
 
 
@@ -160,7 +157,7 @@ def _assert_eventually_masked[R: BaseModel](
                     )
                     return
             case _:
-                last = f"<non-Success result: {result}>"
+                last = scrub(f"<non-Success result: {result}>")
         if time.monotonic() >= deadline:
             pytest.fail(
                 f"presidio pre_call guardrail never masked the PII within "
@@ -281,7 +278,7 @@ class TestPresidioPostCallMasking:
                         )
                         return
                 case _:
-                    last = f"<non-Success result: {result}>"
+                    last = scrub(f"<non-Success result: {result}>")
             if time.monotonic() >= deadline:
                 pytest.fail(
                     f"presidio post_call guardrail never masked the model's output within "

tests/e2e/ui/tests/guardrails/presidioUserStory.spec.ts

@@ -146,8 +146,9 @@ test.describe("Presidio PII guardrail, end to end from the dashboard", () => {
     await expect(onlyVisible(drawer.getByText("Detected Entities (2)"))).toBeVisible({ timeout: 10_000 });
     await expect(onlyVisible(drawer.getByText("EMAIL_ADDRESS", { exact: true }))).toBeVisible({ timeout: 10_000 });
     await expect(onlyVisible(drawer.getByText("PHONE_NUMBER", { exact: true }))).toBeVisible({ timeout: 10_000 });
-    await expect(onlyVisible(drawer.getByText("Score: 1.00", { exact: true }))).toBeVisible({ timeout: 10_000 });
-    await expect(onlyVisible(drawer.getByText("Score: 0.75", { exact: true }))).toBeVisible({ timeout: 10_000 });
+    await expect(drawer.getByText(/^Score: \d\.\d{2}$/).filter({ visible: true })).toHaveCount(2, {
+      timeout: 10_000,
+    });
 
     await expect(drawer.getByText(RAW_EMAIL)).toHaveCount(0);
     await expect(drawer.getByText(RAW_PHONE)).toHaveCount(0);